Closed pipliggins closed 2 months ago
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 97.70%. Comparing base (
627c07e
) to head (e809c1e
). Report is 3 commits behind head on main.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
While I think there might be more speed increases by re-writing some of the dictionary creation, I'm going to see how slow it gets when creating the 'full' dengue mapping files, in comparison to the time taken to read in the data files, before worrying about it too much more.
Different approach to increasing ingestion speed.
This PR uses
joblib
to parallelise the ingestion across resources. The current two time sinks are:apply
going on.Without parallelisation, current time is:
With parallelisation one, the speed up is:
(readout edited for easy comparison)