search

ClimateImpactLab / dodola

Containerized application for running individual tasks in a larger, orchestrated CMIP6 bias-adjustment and downscaling workflow.
https://climateimpactlab.github.io/dodola/
Apache License 2.0
14 stars 7 forks source link

Unit tests are very very slow #207

Open brews opened 2 years ago

brews commented 2 years ago

Unit tests run in CI have become very very slow. The can take about 15 minutes to complete.

I suspect dodola.services.validate() using dask.delayed is the culprit. Dask was used here as a hack to work around memory problems (see #126). The drawback to this hack has been that simple unit tests on Datasets covering a long time range take much more time to run.

If this is the case I see two options:

  1. Re-write validation to work around the original memory problems in #126 while also making unit tests complete faster. i.e. replace the hack solution with a "real" solution.
  2. Find and refactor the one or two long-running unit tests so that they test the same behavior, but run faster.