Package sync taking forever with building Zict

[mrocklin]

I noticed that my TPCH builds stopped running. They were stuck on package sync. I created a normal cluster to see what was going on.

In [1]: import coiled
cluste
In [2]: cluster = coiled.Cluster()
╭────────────────────────────────────────── Package Sync ──────────────────────────────────────────╮
│ Fetching latest package priorities                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Scanning 331 conda packages                             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Scanning 244 python packages                            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Running pip check                                       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Validating environment                                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Creating wheel for /Users/mrocklin/workspace/benchmarks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Creating wheel for coiled                               ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:02 │
│ Creating wheel for dask                                 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:04 │
│ Creating wheel for dask-expr                            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:03 │
│ Creating wheel for distributed                          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:04 │
│ Creating wheel for zict                                 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:02:07 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

cc @dan-blanchard for context, zict is a tiny pure-python package.

Any thoughts on what might be going on? Anything I can help with here with regards to information?

[dan-blanchard]

That's very weird. Could you point me to which scan on our list of scans for you represents this environment? https://cloud.coiled.io/admin/declarative/scanresultenvironmentmodel/?q=mrocklin

[mrocklin]

Hrm, that 403s to me. When I log into the admin dashboard separately it works, but I'm having difficulty finding that table.

[dan-blanchard]

Once you're logged into the admin dashboard the link should work, but if you search for "scan" on the main Django admin page, the only result will be a link to that table.

[mrocklin]

I can reproduce this with the following steps taken from this repository

mamba env create -n tpch -f ci/environment.yml
conda activate tpch
mamba env update -f ci/environment-git-tip.yml
mamba env update -f ci/environment-test.yml

ipython

import coiled
cluster = coiled.Cluster()

I apologize that this is far from minimal.

[mrocklin]

I'll also mention that this may be a genuine issue with zict. I don't know. @fjetter you may want to be aware.

I am resolving this personally with the following:

pip uninstall zict
pip install zict

[dan-blanchard]

@mrocklin are you still experiencing this? I cannot reproduce this even with the example you gave.

In [1]: import coiled

In [2]: cluster = coiled.Cluster()
╭────────────────────────────────────────── Package Sync ──────────────────────────────────────────╮
│ Fetching latest package priorities                                ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Scanning 331 conda packages                                       ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Scanning 241 python packages                                      ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Running pip check                                                 ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Validating environment                                            ━━━━━━━━━━━━━━━━━━━━━━ 0:00:01 │
│ Creating wheel for /Users/danielblanchard/repos/coiled/benchmarks ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Creating wheel for dask                                           ━━━━━━━━━━━━━━━━━━━━━━ 0:00:19 │
│ Creating wheel for dask-expr                                      ━━━━━━━━━━━━━━━━━━━━━━ 0:00:09 │
│ Creating wheel for distributed                                    ━━━━━━━━━━━━━━━━━━━━━━ 0:00:16 │
│ Creating wheel for zict                                           ━━━━━━━━━━━━━━━━━━━━━━ 0:00:08 │
│ Uploading coiled_local_benchmarks                                 ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Uploading dask                                                    ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Uploading dask-expr                                               ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Uploading distributed                                             ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Uploading zict                                                    ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
│ Requesting package sync build                                     ━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯