search

OuhscBbmc / REDCapR

R utilities for interacting with REDCap
https://ouhscbbmc.github.io/REDCapR
Other
114 stars 46 forks source link

Read EAV -oneshot #138

Closed wibeasley closed 7 years ago

wibeasley commented 7 years ago

Read EAV (instead of wide/flat) from the REDCap API, and convert it to a flat (non-EAV) dataset. This will be more work for R on the client than before. But it should be faster overall, because the REDCap webserver doesn't have to concatenate each cell in the EAV to as long of a string.

The advantages will be even greater when the REDCap project is 'sparse', in the sense that many variables aren't assigned to every event. The string concatenation in PHP still has to add a comma (and thus the memory-performance hit) in the wide format. But in the EAV format, those columns/rows/cells are simply passed over.

Even if it took the client the same amount of time, it's still better, because the work is distributed to the clients, instead of clogging up the webserver on long 10k+ reads.

(Thanks to discussion and guidance from @taylorr4, @123andy, & @aarenson for a 1+ years)

wibeasley commented 7 years ago

The playground for the oneshot is done. I'm moving on to experimenting w/ batching & EAV (#145).