Ouranosinc / PAVICS-e2e-workflow-tests

Test user-level workflow.
Apache License 2.0
0 stars 2 forks source link

Need to unpin dask, distributed in Jupyter env, climex.ipynb intermittent slow #100

Closed tlvu closed 1 year ago

tlvu commented 2 years ago

Pinned since https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/pull/101.

dask and distributed was causing PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb to run excessively slow: Finch did a similar dask and distributed pinning in https://github.com/bird-house/finch/pull/226

Normal: 1 minute

11:56:50  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
11:57:29  ....s.                                                                   [ 87%]
11:57:35  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]

Slow: 34 minutes

19:49:39  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
20:23:52  FxXXsX                                                                   [ 87%]
20:24:00  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]

cf_xarray caused climex.ipynb to be intermitently slow:

See many builds in https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/pull/101 for intermittent slow climex.ipynb.

Normal: 1.5 minutes

11:09:36  pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 13%]
11:10:51  pavics-sdi-master/docs/source/notebooks/climex.ipynb ..........          [ 18%]

Slow: 4.5 minutes

19:35:49  pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 13%]
19:40:13  pavics-sdi-master/docs/source/notebooks/climex.ipynb ..........          [ 18%]

climex.ipynb intermittent slow seems to already present since image pavics/workflow-tests:220121 (before this pinning) but much less frequent but here is one of the rare build that shows it: job-PAVICS-e2e-workflow-tests-current-production-version-1-consoleText.txt a whopping 11 minutes !

12:02:35  pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 13%]
12:13:15  pavics-sdi-master/docs/source/notebooks/climex.ipynb ..........          [ 18%]

Update:

tlvu commented 2 years ago

FYI @huard @tlogan2000

tlvu commented 1 year ago

Should be fixed by https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/pull/106