Closed jbusecke closed 2 years ago
:tada: New recipe runs created for the following recipes at sha 55337f3db0ac54e429a3f33fb1935da4a239e2b6
:
CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.dummyCMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.dummy
: https://pangeo-forge.org/dashboard/recipe-run/818CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.dummy
: https://pangeo-forge.org/dashboard/recipe-run/819:tada: New recipe runs created for the following recipes at sha bf72d5fc47eb8be36651d1b009f5fc1cb007c7dd
:
CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.dummy
: https://pangeo-forge.org/dashboard/recipe-run/820CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.dummy
: https://pangeo-forge.org/dashboard/recipe-run/821CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.dummy
: https://pangeo-forge.org/dashboard/recipe-run/822/run recipe-test recipe_run_id=820
:sparkles: A test of your recipe CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.dummy
is now running on Pangeo Forge Cloud!
I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)
In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/820
:partying_face: Hooray! The test execution of your recipe CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.dummy
succeeded.
Here is a static representation of the dataset built by this recipe:
You can also open this dataset by running the following Python code
import fsspec
import xarray as xr
dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-820/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.dummy.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds
in this (or your Python interpreter of choice).
Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.
Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g,
"time"
or equivalent) to be2
.
- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?
@CommonClimate could you check the output, and see if these look ok to you? Ill run the others now.
/run recipe-test recipe_run_id=821
/run recipe-test recipe_run_id=822
:sparkles: A test of your recipe CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.dummy
is now running on Pangeo Forge Cloud!
I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)
In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/821
:sparkles: A test of your recipe CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.dummy
is now running on Pangeo Forge Cloud!
I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)
In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/822
Pangeo Forge Cloud told me that our test of your recipe CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.dummy
failed. But don't worry, I'm sure we can fix this!
To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/821
If you haven't yet tried pruning and running your recipe locally, I suggest trying that now.
Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps!
Pangeo Forge Cloud told me that our test of your recipe CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.dummy
failed. But don't worry, I'm sure we can fix this!
To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/822
If you haven't yet tried pruning and running your recipe locally, I suggest trying that now.
Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps!
The logs say:
2022-07-01T14:45:33.649718+00:00 (INFO)
Submitted for execution: Job prefect-job-fad0a939
2022-07-01T14:57:08.484788+00:00 (INFO)
Rescheduled by a Lazarus process. This is attempt 1.
2022-07-01T14:57:11.170213+00:00 (INFO)
Submitted for execution: Job prefect-job-73b21ee5
2022-07-01T15:09:07.186314+00:00 (INFO)
Rescheduled by a Lazarus process. This is attempt 2.
2022-07-01T15:09:10.845308+00:00 (INFO)
Submitted for execution: Job prefect-job-9768abae
2022-07-01T15:21:16.937544+00:00 (INFO)
Rescheduled by a Lazarus process. This is attempt 3.
2022-07-01T15:21:18.624253+00:00 (INFO)
Submitted for execution: Job prefect-job-6c752e03
2022-07-01T15:32:22.22285+00:00 (ERROR)
A Lazarus process attempted to reschedule this run 3 times without success. Marking as failed.
I am not at all sure what this means (maybe @cisaacstern knows more). will retry one of the jobs.
/run recipe-test recipe_run_id=822
:sparkles: A test of your recipe CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.dummy
is now running on Pangeo Forge Cloud!
I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)
In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/822
Pangeo Forge Cloud told me that our test of your recipe CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.dummy
failed. But don't worry, I'm sure we can fix this!
To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/822
If you haven't yet tried pruning and running your recipe locally, I suggest trying that now.
Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps!
Same issue with the Lazarus process. I am going to wait for some input from @cisaacstern side here.
Thanks for the ping Julius. I admittedly don't know what this is off the top of my head but I'll look into it.
Thanks Charles.
The code snippet above works like a charm! I'm able to load and plot the data without a hitch, in just a couple of seconds:
@cisaacstern should we try to rerun the recipe for the two failed runs or is that something that would need more time to fix?
:tada: New recipe runs created for the following recipes at sha 4d6ca67355547dbc7a46636cad53474609ae7600
:
CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.dummy
: https://pangeo-forge.org/dashboard/recipe-run/890CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.dummy
: https://pangeo-forge.org/dashboard/recipe-run/891Seems like the 2k simulation is (possibly temporarily) unavailable. I will try to run one of the others one more time.
/run recipe-test recipe_run_id=891
:sparkles: A test of your recipe CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.dummy
is now running on Pangeo Forge Cloud!
I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)
In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/891
From the logs it seems like the 'Lazarus' thing might have just been a temporary issue.
/run recipe-test recipe_run_id=890
:partying_face: Hooray! The test execution of your recipe CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.dummy
succeeded.
Here is a static representation of the dataset built by this recipe:
You can also open this dataset by running the following Python code
import fsspec
import xarray as xr
dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-891/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.dummy.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True)
ds
in this (or your Python interpreter of choice).
Please copy-and-paste the list below into a new comment on this thread, and check the boxes off as you've reviewed them.
Note: This test execution is limited to two increments in the concatenation dimension, so you should expect the length of that dimension (e.g,
"time"
or equivalent) to be2
.
- [ ] Are the dimension lengths correct?
- [ ] Are all of the expected variables present?
- [ ] Does plotting the data produce a plot that looks like your dataset?
- [ ] Can you run a simple computation/reduction on the data and produce a plausible result?
Apologies for the delayed response here. Please let me know if/how I can be of assistance at this point.
adding @jordanplanders to this conversation, as she will be taking this over for a bit
Well @cisaacstern the question remains the same: what do we have have to do at this point? Go through the checklist and complete the review? Sorry we are new to this process, so we need a bit of hand-holding.
Yes, someone with knowledge of the data opening it and answering the checklist provided in https://github.com/pangeo-forge/cmip6-feedstock/pull/16#issuecomment-1178001348 would be helpful. Beyond that, I will defer to @jbusecke who has been following this PR much more closely than me.
:tada: New recipe runs created for the following recipes at sha 6df9f298fbde1f0af3b35ac03852b68de5f91eea
:
CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507
: https://pangeo-forge.org/dashboard/recipe-run/903CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r2i1p1f1.Amon.pr.gn.v20190507
: https://pangeo-forge.org/dashboard/recipe-run/904CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r3i1p1f1.Amon.pr.gn.v20190508
: https://pangeo-forge.org/dashboard/recipe-run/905CMIP6.DAMIP.CAS.FGOALS-g3.hist-aer.r1i1p1f1.Amon.pr.gn.v20200411
: https://pangeo-forge.org/dashboard/recipe-run/906CMIP6.DAMIP.CAS.FGOALS-g3.hist-aer.r2i1p1f1.Amon.pr.gn.v20200411
: https://pangeo-forge.org/dashboard/recipe-run/907CMIP6.DAMIP.CAS.FGOALS-g3.hist-aer.r3i1p1f1.Amon.pr.gn.v20200411
: https://pangeo-forge.org/dashboard/recipe-run/908CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/909CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/910CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r11i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/911CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r11i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/912CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r12i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/913CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r12i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/914CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r13i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/915CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r14i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/916CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r14i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/917CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r15i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/918CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r15i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/919CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r1i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/920CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r2i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/921CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r2i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/922CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r3i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/923CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r3i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/924CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r4i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/925CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r5i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/926CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r5i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/927CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r6i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/928CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r6i1p2f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/929CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r9i1p1f1.Amon.pr.gn.v20190429
: https://pangeo-forge.org/dashboard/recipe-run/930CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r10i1p1f2.Amon.pr.gr.v20190308
: https://pangeo-forge.org/dashboard/recipe-run/931CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r1i1p1f2.Amon.pr.gr.v20190308
: https://pangeo-forge.org/dashboard/recipe-run/932CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r2i1p1f2.Amon.pr.gr.v20190308
: https://pangeo-forge.org/dashboard/recipe-run/933CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r3i1p1f2.Amon.pr.gr.v20190308
: https://pangeo-forge.org/dashboard/recipe-run/934CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r4i1p1f2.Amon.pr.gr.v20190308
: https://pangeo-forge.org/dashboard/recipe-run/935CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r5i1p1f2.Amon.pr.gr.v20190308
: https://pangeo-forge.org/dashboard/recipe-run/936CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r6i1p1f2.Amon.pr.gr.v20190308
: https://pangeo-forge.org/dashboard/recipe-run/937CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r7i1p1f2.Amon.pr.gr.v20190308
: https://pangeo-forge.org/dashboard/recipe-run/938CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r8i1p1f2.Amon.pr.gr.v20190308
: https://pangeo-forge.org/dashboard/recipe-run/939CMIP6.DAMIP.CNRM-CERFACS.CNRM-CM6-1.hist-aer.r9i1p1f2.Amon.pr.gr.v20190308
: https://pangeo-forge.org/dashboard/recipe-run/940CMIP6.DAMIP.CSIRO-ARCCSS.ACCESS-CM2.hist-aer.r1i1p1f1.Amon.pr.gn.v20201120
: https://pangeo-forge.org/dashboard/recipe-run/941CMIP6.DAMIP.CSIRO-ARCCSS.ACCESS-CM2.hist-aer.r2i1p1f1.Amon.pr.gn.v20201120
: https://pangeo-forge.org/dashboard/recipe-run/942CMIP6.DAMIP.CSIRO-ARCCSS.ACCESS-CM2.hist-aer.r3i1p1f1.Amon.pr.gn.v20201120
: https://pangeo-forge.org/dashboard/recipe-run/943CMIP6.DAMIP.CSIRO.ACCESS-ESM1-5.hist-aer.r1i1p1f1.Amon.pr.gn.v20200615
: https://pangeo-forge.org/dashboard/recipe-run/944CMIP6.DAMIP.CSIRO.ACCESS-ESM1-5.hist-aer.r2i1p1f1.Amon.pr.gn.v20200615
: https://pangeo-forge.org/dashboard/recipe-run/945CMIP6.DAMIP.CSIRO.ACCESS-ESM1-5.hist-aer.r3i1p1f1.Amon.pr.gn.v20200615
: https://pangeo-forge.org/dashboard/recipe-run/946CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-aer.r10i1p1f1.Amon.pr.gr.v20180914
: https://pangeo-forge.org/dashboard/recipe-run/947CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-aer.r1i1p1f1.Amon.pr.gr.v20180914
: https://pangeo-forge.org/dashboard/recipe-run/948CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-aer.r2i1p1f1.Amon.pr.gr.v20180914
: https://pangeo-forge.org/dashboard/recipe-run/949CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-aer.r3i1p1f1.Amon.pr.gr.v20180914
: https://pangeo-forge.org/dashboard/recipe-run/950CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-aer.r4i1p1f1.Amon.pr.gr.v20180914
: https://pangeo-forge.org/dashboard/recipe-run/951CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-aer.r5i1p1f1.Amon.pr.gr.v20180914
: https://pangeo-forge.org/dashboard/recipe-run/952CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-aer.r6i1p1f1.Amon.pr.gr.v20180914
: https://pangeo-forge.org/dashboard/recipe-run/953CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-aer.r7i1p1f1.Amon.pr.gr.v20180914
: https://pangeo-forge.org/dashboard/recipe-run/954CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-aer.r8i1p1f1.Amon.pr.gr.v20180914
: https://pangeo-forge.org/dashboard/recipe-run/955CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-aer.r9i1p1f1.Amon.pr.gr.v20180914
: https://pangeo-forge.org/dashboard/recipe-run/956CMIP6.DAMIP.MIROC.MIROC6.hist-aer.r4i1p1f1.Amon.pr.gn.v20201228
: https://pangeo-forge.org/dashboard/recipe-run/957CMIP6.DAMIP.MIROC.MIROC6.hist-aer.r5i1p1f1.Amon.pr.gn.v20201228
: https://pangeo-forge.org/dashboard/recipe-run/958CMIP6.DAMIP.MIROC.MIROC6.hist-aer.r7i1p1f1.Amon.pr.gn.v20201228
: https://pangeo-forge.org/dashboard/recipe-run/959CMIP6.DAMIP.MOHC.HadGEM3-GC31-LL.hist-aer.r1i1p1f3.Amon.pr.gn.v20190814
: https://pangeo-forge.org/dashboard/recipe-run/960CMIP6.DAMIP.MOHC.HadGEM3-GC31-LL.hist-aer.r2i1p1f3.Amon.pr.gn.v20190815
: https://pangeo-forge.org/dashboard/recipe-run/961CMIP6.DAMIP.MOHC.HadGEM3-GC31-LL.hist-aer.r3i1p1f3.Amon.pr.gn.v20190814
: https://pangeo-forge.org/dashboard/recipe-run/962CMIP6.DAMIP.MOHC.HadGEM3-GC31-LL.hist-aer.r4i1p1f3.Amon.pr.gn.v20190814
: https://pangeo-forge.org/dashboard/recipe-run/963CMIP6.DAMIP.MOHC.HadGEM3-GC31-LL.hist-aer.r5i1p1f3.Amon.pr.gn.v20211123
: https://pangeo-forge.org/dashboard/recipe-run/964CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r1i1p1f1.Amon.pr.gn.v20180821
: https://pangeo-forge.org/dashboard/recipe-run/965CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r1i1p1f2.Amon.pr.gn.v20191226
: https://pangeo-forge.org/dashboard/recipe-run/966CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r1i1p3f1.Amon.pr.gn.v20191226
: https://pangeo-forge.org/dashboard/recipe-run/967CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r2i1p1f1.Amon.pr.gn.v20180821
: https://pangeo-forge.org/dashboard/recipe-run/968CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r2i1p1f2.Amon.pr.gn.v20191226
: https://pangeo-forge.org/dashboard/recipe-run/969CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r2i1p3f1.Amon.pr.gn.v20191226
: https://pangeo-forge.org/dashboard/recipe-run/970CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r3i1p1f1.Amon.pr.gn.v20180822
: https://pangeo-forge.org/dashboard/recipe-run/971CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r3i1p1f2.Amon.pr.gn.v20191226
: https://pangeo-forge.org/dashboard/recipe-run/972CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r3i1p3f1.Amon.pr.gn.v20191226
: https://pangeo-forge.org/dashboard/recipe-run/973CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r4i1p1f1.Amon.pr.gn.v20180823
: https://pangeo-forge.org/dashboard/recipe-run/974CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r4i1p1f2.Amon.pr.gn.v20191226
: https://pangeo-forge.org/dashboard/recipe-run/975CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r4i1p3f1.Amon.pr.gn.v20191226
: https://pangeo-forge.org/dashboard/recipe-run/976CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r5i1p1f1.Amon.pr.gn.v20180823
: https://pangeo-forge.org/dashboard/recipe-run/977CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r5i1p1f2.Amon.pr.gn.v20191226
: https://pangeo-forge.org/dashboard/recipe-run/978CMIP6.DAMIP.NASA-GISS.GISS-E2-1-G.hist-aer.r5i1p3f1.Amon.pr.gn.v20191226
: https://pangeo-forge.org/dashboard/recipe-run/979CMIP6.DAMIP.NCC.NorESM2-LM.hist-aer.r1i1p1f1.Amon.pr.gn.v20190920
: https://pangeo-forge.org/dashboard/recipe-run/980CMIP6.DAMIP.NCC.NorESM2-LM.hist-aer.r2i1p1f1.Amon.pr.gn.v20190920
: https://pangeo-forge.org/dashboard/recipe-run/981CMIP6.DAMIP.NCC.NorESM2-LM.hist-aer.r3i1p1f1.Amon.pr.gn.v20190920
: https://pangeo-forge.org/dashboard/recipe-run/982CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701
: https://pangeo-forge.org/dashboard/recipe-run/983CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714
: https://pangeo-forge.org/dashboard/recipe-run/984/run recipe-test recipe_run_id=984
:sparkles: A test of your recipe CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714
is now running on Pangeo Forge Cloud!
I'll notify you with a comment on this thread when this test is complete. (This could be a little while...)
In the meantime, you can follow the logs for this recipe run at https://pangeo-forge.org/dashboard/recipe-run/984
Pangeo Forge Cloud told me that our test of your recipe CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714
failed. But don't worry, I'm sure we can fix this!
To see what error caused the failure, please review the logs at https://pangeo-forge.org/dashboard/recipe-run/984
If you haven't yet tried pruning and running your recipe locally, I suggest trying that now.
Please report back on the results of your local testing in a new comment below, and a Pangeo Forge maintainer will help you with next steps!
Ughhh this does not seem a great day for ESGF. I am seeing a bunch of 503s and the one datasets that generates a recipe then fails with another server error:
FileNotFoundError(url) from exc FileNotFoundError: http://esgf3.dkrz.de/thredds/fileServer/cmip6/PMIP/MPI-M/MPI-ESM1-2-LR/past2k/r1i1p1f1/Amon/tas/gn/v20210714/tas_Amon_MPI-ESM1-2-LR_past2k_r1i1p1f1_gn_700101-702012.nc
Ill try to push to this some time later and see if we have more luck?
@jbusecke I took the gn.dummy version for a spin and it worked fine, except it only returned the first two chunks. Running:
dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-984/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds_tmp = xr.open_zarr(mapper, consolidated=True, decode_times=False)
ds_tmp
returned:
FileNotFoundError: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge-test/prod/recipe-run-984/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr/.zmetadata
It's probably user error (pardon), but maybe you can give me a nudge in the right direction?
First of all, hi Jordan! Its been a while since we were working on problem-sets for oceanography? Very good to see you here!
It's probably user error (pardon), but maybe you can give me a nudge in the right direction?
I actually think this is not on you. I think this particular run failed (for reasons I need to investigate). I just tried to open it and I think there is only a folder with nothing in it.
I am looking into this with @cisaacstern right now, and we will report back!
Hi!! So it has!
I think I'm still pretty fuzzy about the backend of pangeo-forge, but I grabbed code from the cmip6-feedstock recipe.py and tried it on CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701
and it identified all parts of the dataset correctly (when I compared it to the web ui), but when I tried it on CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714
, it only returned 10 of 93 (Gist here). It's not obvious to me why that would cause it to fail per se, but I can imagine there might be a check that fails if the dataset end date doesn't correspond with the last date of the last block?
I wrote a little bit of code to pull together a url list for past2k, and expected it to break for the GFDL data (which is to say: I expected there was something weird about past2k that was fouling things up), but it didn't (I changed the stem url string appropriately).
Maybe something in all of this will help root out the issue...
:tada: New recipe runs created for the following recipes at sha 3da3eed2b4ab78a3a3b766587ce90e0b7fbe62f1
:
CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714
: https://pangeo-forge.org/dashboard/recipe-run/1000Ah that is helpful, thanks.
This bit is interesting and warrants some more digging:
it only returned 10 of 93
I assume that is file urls in this case? If there is a discrepancy between the web search and the API based search, that might have to be raised upstream with the ESGF folks.
I will merge this one now, and see if the issue you describe persists. Ill investigate a bit further next week.
Overall @cisaacstern and I have discussed to split out the PMIP datasets into a separate feedstock (https://github.com/pangeo-forge/staged-recipes/pull/162) to keep things a bit more organized. None of that should for now make a difference with this issue, but it will enable us to treat your request more in isolation. Once we have that feedstock merged we can move dissussions over there.
I think I found the bug, actually. I previously did not set the limit on the request parameters (and I believe it defaults to 10! results or files in this case). I just changed it to 500 here and locally get the correct number of 93 files! Lets see if this fixes that issue.
So I was able to do this:
import fsspec
import xarray as xr
dataset_public_url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr'
mapper = fsspec.get_mapper(dataset_public_url)
ds = xr.open_zarr(mapper, consolidated=True, decode_times=False)
ds
and get
Does that look about right to you?
Yes! Thank you!
Looks dandy, thank you. I support moving this to a different recipe if it simplifies your workflow - just let us know where to follow this discussion.
We are still working out some kinks. I will definitely notify you here once the feedstock is generated. Thanks for the patience!
This PR superseeds #14, and is already rebased on #15. For previous discussions please refer to https://github.com/pangeo-forge/cmip6-feedstock/pull/14
It passed locally for me, lets see if we can actually build the dataset.