Problem with outdated/mismatched versions of datetime (for pandas/vaex)

[espg]

Running into issues with the worker load-set where some software versions are out of sync. Here's the error message:

   raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (python-dateutil 2.7.5 (/srv/conda/envs/notebook/lib/python3.9/site-packages), Requirement.parse('python-dateutil>=2.8.1'), {'pandas'})

this is calling function:

def get_thing(s3file, cells):
    df = vaex.open(s3file)
    # Get to our query resolution
    df['midx11'] = df.midx // 10**7
    df['basin2'] = df.midx11.isin(cells)
    df.select(df.basin2 == True)
    tmp = df.to_arrow_table(selection=True, parallel=True)
    return tmp

Here's the traceback:

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Input In [18], in <cell line: 1>()
----> 1 tmp.result()

File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/client.py:233, in Future.result(self, timeout)
    231 if self.status == "error":
    232     typ, exc, tb = result
--> 233     raise exc.with_traceback(tb)
    234 elif self.status == "cancelled":
    235     raise result

Input In [14], in get_thing()
      1 def get_thing(s3file, cells):
----> 2     df = vaex.open(s3file)
      3     # Get to our query resolution
      4     df['midx11'] = df.midx // 10**7

File /srv/conda/envs/notebook/lib/python3.9/site-packages/vaex/__init__.py:232, in open()
    230     ds = vaex.dataset.open(path_output, fs_options=fs_options, fs=fs, **kwargs)
    231 else:
--> 232     ds = vaex.dataset.open(path, fs_options=fs_options, fs=fs, **kwargs)
    233 df = vaex.from_dataset(ds)
    234 if df is None:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/vaex/dataset.py:85, in open()
     83 failures = "\n".join([f'\n-----{who}-----\n:' + vaex.utils.format_exception_trace(e) for e, who in failures])
     84 if failures:
---> 85     raise IOError(f'Cannot open {path}, failures: {failures}.')
     86 else:
     87     raise IOError(f'Cannot open {path} nobody knows how to read it.')

OSError: Cannot open s3://geostacks/icesat_2/shards=2111134/cycle=4/atl06.parquet, failures: 
-----arrow = vaex.arrow.opener:ArrowOpener-----
:Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/vaex/dataset.py", line 63, in open
    opener = entry.load()
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2464, in load
    self.require(*args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2487, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pkg_resources/__init__.py", line 782, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (python-dateutil 2.7.5 (/srv/conda/envs/notebook/lib/python3.9/site-packages), Requirement.parse('python-dateutil>=2.8.1'), {'pandas'})

Tried fixing it with custom calls to install on the worker nodes (using conda, using pip upgrade, and using pip uninstall + install) but doesn't work since there's still datetime version that I can't purge.

@consideRatio can you update the base python install to address this and fix? I don't think I'm able to fix at runtime...it needs an image update =/

[consideRatio]

@espg sorry for the slow turnaround, will do!

Is this a version mismatch between juputerhub user pod and the schedulers/workers? I'll rebuild the image and ensure the user pods and scheduler/worker pods will get the same image no matter what. I'd only expect you to see this kind of error if there was some pod aged ~1 month or so, but this is really surprising.

If this problem occur still, it think the only remaining explanation is a version mismatch associated with:

• a temporary change to the user environment after server startup not made in the workers, or the other way around
• a persisted change in your user environment, for example by creating a new conda environment (doesn't seem to be the case from the logs above, as they reference /srv/conda/envs/notebook which is associated with the image, any change there won't be persisted).

[consideRatio]

@espg the image is rebuilt and updated for use. For reference, the dateutil.__version__ resolved to 2.8.2 for me in this fresh image.

[consideRatio]

I recall this as resolved, closing this issue for now.