search

wfu-dmds / teaching-r-study

3 stars 2 forks source link

Switch to single upload of event data and use a future promise #14

Closed nstrayer closed 3 years ago

nstrayer commented 3 years ago

By avoiding the checking of a file and downloading of existing files we save a lot of slowness caused by waiting for external server responses. The difference here is later you will need to go through and process the many smaller files but that's a one-time-thing.

The future promise will also take the slow upload onto another worker to avoid slowing down your app.

Make sure you have the latest version of promises to get promises::future_promise().

LucyMcGowan commented 3 years ago

@nstrayer when I run this I get:

Warning: UNRELIABLE VALUE: Future (‘<none>’) unexpectedly generated random numbers without specifying argument 'seed'. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify 'seed=TRUE'. This ensures that proper, parallel-safe random numbers are produced via the L'Ecuyer-CMRG method. To disable this check, use 'seed=NULL', or set option 'future.rng.onMisuse' to "ignore".
createTcpServer: address already in use
Unhandled promise error: Failed to create server
Unhandled promise error: Failed to create server

a whole bunch of times (it seems each time the event recorder is run). I know literally nothing about future or promises is there a good resource for me to go to to troubleshoot something like this?

Also, on.exit() seems to remove the file before it can be uploaded? I get an error that the file does not exist unless I comment that out, in which case I get the above errors.

(HUGE thank you again for looking at this for me!)