osi_saf_450_430-a_rg025

[lzampier]

First try at a recipe for the OSI-SAF Global Sea Ice Concentration Climate Data Record (1990-2021 Daily) Release 3. The entire dataset is reprocessed (regrinding and merging the north and south hemispheres) and stored on the NCAR supercomputer. This makes it more approachable for model validation purposes compared to polar projections. Globus is used to create a URL for the netcdf files.

[lzampier]

It is not clear to me why the checks failed. It would be great if an expert could have a look! Thanks in advance for the help :)

[cisaacstern]

Hi @lzampier thanks for this contribution!

I failing check is just a formatting hook. For future reference, you should be able to fix these issues yourself as described in https://github.com/pangeo-forge/staged-recipes/pull/230#issuecomment-1352443908.

But I should be able to auto-fix them for you here, which I'll do now.

[cisaacstern]

pre-commit.ci autofix

[lzampier]

Thank you, @cisaacstern!

[cisaacstern]

@lzampier I'm currently in the process of cleaning up a bug in our cloud service. I'll check back here to run a test of your recipe once it's fixed.

[lzampier]

Take your time, @cisaacstern. I am not in a hurry of having the dataset on the cloud as I have my local copy.

[cisaacstern]

@lzampier:

1. I just merged master into this PR branch, so before you work on it further in the future, you'll need to pull/fetch from your remote (lzampier/staged-recipes).
2. I think the bug is fixed in the backend service so I'll try running a test of this recipe now.

[cisaacstern]

/run OSI-SAF-450-430-a_rg025

[pangeo-forge[bot]]

The test failed, but I'm sure we can find out why!

Pangeo Forge maintainers are working diligently to provide public logs for contributors. That feature is not quite ready yet, however, so please reach out on this thread to a maintainer, and they'll help you diagnose the problem.

[cisaacstern]

This failure is because the backend service is still not fixed. It's not a problem with the recipe contribution. Working on fixing the backend now and will re-run the test once it's working.

[lzampier]

That makes sense. Thanks for your help.

[lzampier]

@cisaacstern, is there any update as to when the data could be available in the cloud? I am asking because I need to make a decision about copying data from an HPC system to another, but this can be done easily, and therefore there is no hurry on my side. Thank you for your help.

[cisaacstern]

@lzampier thanks for checking in. I'm still in the process of testing our latest deployment of the cloud service for the edge cases which caused it to break recently, but it is now working for many cases, so I'll re-try the test above, and we'll see what happens.

[cisaacstern]

/run OSI-SAF-450-430-a_rg025

[cisaacstern]

This test failed to deploy.

@lzampier, re: timeline, I will take a look at this now. If this can't be fixed by Friday of this week, I'll let you know here.

[cisaacstern]

@lzampier, here's an end-of-week update as promised. As noted in https://github.com/pangeo-forge/mar_greenland-feedstock/issues/3#issuecomment-1405453212, the backend service's capability to submit jobs is currently still down. Timelines for fixing these type of bugs are notoriously hard to estimate, but as noted in the linked comment, I expect it will be at least another week before everything is running as normal. Thank you for your patience as we work through this.

[lzampier]

Thank you for the update, and best of luck in debugging this issue!

[lzampier]

Hi @cisaacstern. Is there any update on the status of the bug fix? Thank you for your help.

[cisaacstern]

@lzampier thanks for checking in. Regrettably we're still not in a position to deploy this (or any) current recipe to the cloud service, as the search for the solution to the initial bug as opened up into some broader conversations among the core team as to how to best manage this infrastructure for the long term. Some of that thinking is captured in https://github.com/pangeo-forge/pangeo-forge-orchestrator/issues/233, and there are a couple PR's mentioned in that thread in which I've started this restructuring work.

Your experience of the downtime delaying your PR here is in part a product of the initial iteration of this cloud service being somewhat over-built in terms of its aims and scope, and we have through challenges like this, come to see the necessity of simplifying the service so that it can both continue to operate more easily in the coming year, and also have a greater chance of longevity beyond our current funding horizon (which brings us to about Spring 2024).

This is a somewhat long way of thanking you again for your patience in engaging with our platform at such an early stage. I do feel confident that we will get this recipe built for you in the coming weeks or (to be honest, because I don't want to misrepresent our capability) possibly months timeframe. At this point the scope of our reworking of the platform has become more open-ended, however, so I can't offer a more specific prediction at this time.

Please feel free to keep checking in here, and again, thanks for your patience and engagement.

[lzampier]

Hi @cisaacstern. Thank you for the update, and sorry for my slow reply. I understand the situation, and I wish you the best of luck in moving forward with this cool project.

I am considering how to best proceed with my dataset, and I was wondering whether an interim solution could be found. In particular, would it be possible to make the locally generated cloud object, which in my case could be created on the NCAR server, available through a different cloud service than pangeo forge cloud? For example, is it an option to make it visible through Globus? It would be great to know if someone in the project has experience with this.

Thanks again!