search

cmu-delphi / epiprocess

Tools for basic signal processing in epidemiology
https://cmu-delphi.github.io/epiprocess/
Other
13 stars 9 forks source link

slide profiling compared to various other backends #392

Open dsweber2 opened 9 months ago

dsweber2 commented 9 months ago

In exploration tooling, https://github.com/cmu-delphi/exploration-tooling/blob/bbcef76a822c36f78d32669ab7b0461dea7a2302/R/data_transforms.R#L38-L47, the forecasters that use epi_slide to do averaging/sd take ~5-10x as long as those that don't. It is unclear if this is specific to epi_slide or would hold for any method of doing this. Towards that end, we should do some profiling of epi_slide to make sure it isn't significantly slower than comparable methods, specifically for mean and standard deviation calculations.

nmdefries commented 8 months ago

Partially addressed in https://github.com/cmu-delphi/epiprocess/pull/397. Adding a specialized rolling mean fn in https://github.com/cmu-delphi/epiprocess/pull/400. We discussed and decided that a specialized rolling sd fn was unnecessary.

nmdefries commented 8 months ago

Profile of 7dav on dataset with 3 geos and 400k obs per geo, commit 7ae9e879

profile

nmdefries commented 8 months ago

With the current implementation, we're unlikely to make epi_slide much faster. A fair amount of the runtime is slider setup and the actual computation (maybe 1/2-3/4 of total time), plus (lots of) low-level functions that are hard to replace.