Upgrade Pangeo Image - Githubissues

NASA-IMPACT / veda-jupyterhub

VEDA JupyterHub technical planning and documentation

1 stars 1 forks source link

Upgrade Pangeo Image #40

Closed wildintellect closed 2 months ago

wildintellect commented 3 months ago

Epic

Description

I know we don't have the documentation yet on #5 , and haven't established a regular cadence yet for updates, but the latest Pangeo brings about some big changes:

Lonboard included (we have another ticket somewhere for this)
Numpy 2.0
Geopandas 1.0

There is a question about the jump to Numpy 2/Geopandas 1 because it could be a major change for some existing users. So we probably need a secondary ticket that covers:

How users know/pick which image version they are using
Allowing users to select older image versions (not sure how to keep this sane)

Acceptance Criteria:

[x] Upgrade the Pangeo image to at least a version that contains Lonboard, could be the last version before Numpy 2.0 if we think that's an issue.

Checklist:

[ ] Epic Link
[x] Detailed description
[x] Assignee

batpad commented 3 months ago

@wildintellect - there's a PR up ^ that upgrades to Pangeo 24.06.02 - I'm fairly certain that's a version with lonboard but without the Numpy 2.0 upgrade.

Do you think we should create a separate issue for the upgrade that involves moving to Numpy 2.0? I'm not fully sure what we need to consider there.

Allowing users to select older image versions (not sure how to keep this sane)

Currently, we could do this via adding a new profile option pinned to an older version - with the new "Fancy Profiles" interface, this might get a little bit better interface-wise for users. Happy to do whatever is best for users here with the version upgrades. Let's test and push through https://github.com/2i2c-org/infrastructure/pull/4312, and then happy to discuss how we can best support users to transition to the latest Pangeo version that has more breaking package updates.

wildintellect commented 3 months ago

@jsignell any thoughts on safety of numpy/pandas upgrades to existing EIS work? @freitagb who should our main User contact be on GHG and other VEDA instances before we bump image versions on them?

jsignell commented 3 months ago

there's a PR up ^ that upgrades to Pangeo 24.06.02 - I'm fairly certain that's a version with lonboard but without the Numpy 2.0 upgrade.

I just double checked as well and I agree with that assessment. It's all laid out in https://github.com/pangeo-data/pangeo-docker-images/blob/2024.06.02/pangeo-notebook/conda-lock.yml

any thoughts on safety of numpy/pandas upgrades to existing EIS work?

I'd be happy to run the notebooks once there is a new vedahub image to test it with. I am a little wary about there being changes, but they might be for the best. For instance https://github.com/Earth-Information-System/fireatlas/issues/61 turned out to be a fiona vs pyogrio issue, so updating geopandas (which uses pyogrio by default) might improve things.

batpad commented 3 months ago

The upgrade to Pangeo 2024.06.02 has been deployed. Thank you so much @jsignell for the checks.

I'd be happy to run the notebooks once there is a new vedahub image to test it with.

I'll create a branch to upgrade to the latest Pangeo version and drop an image tag here to test with.