search

civisanalytics / civis-python

Civis API Python Client
BSD 3-Clause "New" or "Revised" License
34 stars 26 forks source link

[CIVIS-1508] DEP upgrade joblib dep #429

Closed dleatherman-civis closed 3 years ago

dleatherman-civis commented 3 years ago

Features

This PR upgrades the joblib dependency to allow versions 1.1.x which provides several enhancements and bug fixes around pickling which impacted previous runs.

Closes #423.

dleatherman-civis commented 3 years ago

Thank you for the contribution! I have a high-level question:

  • Assuming joblib follows semantic versioning, the jump from joblib version 0.* to 1.* could mean breaking changes from joblib. If there are any, would these changes affect civis-python in any way?

Also, it looks like you might have started this work before some of the most recent PRs got merged into master. Some of the no-longer-valid changes show up in your PR here and should be dropped.

I've taken the liberty to update this PR's title to link this work to the relevant JIRA ticket, and added Closes #423 to the PR description so that GitHub will close the issue once this PR is merged.

The breaking change is identified here. with joblib <1.x, pickling joblib.Memory objects was backwards compatible. Newer versions of numpy changed how numpy.dtypes was pickled which caused objects that should be the same to create different hashes. Essentially, if you're loading in joblib.Memory pickle objects, they are no longer compatible across versions (hence the major version bump).