search

vz-risk / verisr

R package for working with VERIS data
21 stars 11 forks source link

Parallelize json2veris #5

Open gdbassett opened 8 years ago

gdbassett commented 8 years ago

The json corpus is getting very large. It would be really nice if json2veris ran across multiple cores.

gdbassett commented 3 years ago

attempted replacing the main 'for' loop with furrr::map(. Because the futures don't share a single veris data.table object in memory, the updates in the futures don't get written back to the original object.

This could likely be fixed by splitting the veris data table across the futures, returning each subset of the object from the future, and joining them.

Alternately the veris object could be stored in shared, writable memory since the futures each only write a separate part, however that might prevent splitting the task out over futures that lack shared memory.

either way it's going to be a bit more work.