Adapt to the new xclim-testdata structure

bird-house / birdhouse-deploy

Scripts and configurations to deploy the various birds and servers required for a full-fledged production platform

https://birdhouse-deploy.readthedocs.io/en/latest/

Apache License 2.0

4 stars 6 forks source link

Adapt to the new xclim-testdata structure #467

Closed Zeitsperre closed 3 months ago

Zeitsperre commented 3 months ago

Overview

This PR updates the cloning of the xclim-testdata repo to reflect structural changes.

Changes

Non-breaking changes

Adjusts the location of the xclim-testdata data folder

Related Issue / Discussion

CI Operations

birdhouse_daccs_configs_branch: master birdhouse_skip_ci: false

Zeitsperre commented 3 months ago

@tlvu We are currently grabbing main here, but the testdata is tagged according to version (which is based on the date). If we want, we could base this on the tags for less breakage potential.

The last stable tag was v2023.12.14.

tlvu commented 3 months ago

@tlvu We are currently grabbing main here, but the testdata is tagged according to version (which is based on the date). If we want, we could base this on the tags for less breakage potential.

The last stable tag was v2023.12.14.

@Zeitsperre

FYI, if you want to specify the tag v2023.12.14 you can do it here (replace origin/main with v2023.12.14) https://github.com/bird-house/birdhouse-deploy/blob/86ffba01f30187bbde85f3f12d656be7740e7785/birdhouse/deployment/deploy-data-xclim-testdata-to-thredds.yml#L4

So are we planning to only hardcode the tag just as a temporary transition measure or are we planning to go this route forward for production deployment?

Zeitsperre commented 3 months ago

Once my PR at xclim-testdata is approved, I was going to tag a new commit. I don't see why we wouldn't want to use the tags going forward, though. xclim-testdata doesn't move very often, but this could prevent breakage for users.

tlvu commented 3 months ago

Once my PR at xclim-testdata is approved, I was going to tag a new commit.

Oh I didn't realized that you will want to synchronize the merge of this PR with your PR so the tagging is not really required.

I don't see why we wouldn't want to use the tags going forward, though. xclim-testdata doesn't move very often, but this could prevent breakage for users.

Hardcoded tags means:

need to update here each time for new tag
any quick change (fix existing data, add new data) will also need a tag to trigger the deploy
changing tag is slower to autodeploy (nightly) instead of the content (every 30 mins)

Since breaking change is not often, I think this is the first time, is it worth it? It's up to you.

tlvu commented 3 months ago

Since breaking change is not often, I think this is the first time, is it worth it? It's up to you.

Sorry to clarify, we can go the temporary route just for this time, so you do not need to worry about synchronizing merge.

My concerns were about always hardcoding going forward.

Zeitsperre commented 3 months ago

@tlvu I follow your logic. It makes sense here. Given that the structure shouldn't change after this, as well as the fact that we uniquely glob NetCDF files, I don't think we need to use tags here.

Will finalize changes on the xclim PR, merge xclim-testdata, then finally merge the remaining PRs. I don't anticipate needing to make any other changes here after this.

tlvu commented 3 months ago

Oh by the way, you'll need to update the CHANGES.md at the root of the repo as well. Just quickly refer to your xclim-testdata PR for more info.