search

coiled / examples

Examples using Dask and Coiled
14 stars 3 forks source link

Pin packages in conda environment #14

Open crusaderky opened 1 year ago

crusaderky commented 1 year ago

At PyConDE 2022, we spent several hours with the TV screen showing a hanged cluster and we had to debug the issue in realtime, while embarassingly standing at the stall. The problem was caused by a regression in the very latest dask version, released two days before, which was immediately picked up by the examples because the packages in the conda environment weren't pinned.

Everything should be pinned to prevent this kind of surprises. Upgrades should be a conscious act that includes retesting.

jrbourbeau commented 1 year ago

Thanks @crusaderky. I'm curious what the regression was

crusaderky commented 1 year ago

Shuffle hang badly. It was the time when dask was full of deadlocks.