Republication failure because of old version of the dataset.

[pabretonniere]

Hello,

I have a recurrent issue when republishing a dataset.

I don't know if it is related to the publication itself, or the previous unpublish.

We published a dataset in our BSC ESGF node in last May (2021/05/11), and the process went well. Then to add a missing year in a variable, I unpublished it, and I saw the dataset being unpublished from the ESGF as expected, both from the search and from our thredds, without any error message.

Then I corrected the dataset, redid the drs, mapfiles and publish but when getting to the publish, I have this error:

INFO       2021-06-09 12:15:53,019 Requesting to publish PID for dataset "CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn" (version 20210608) and its files at "esgf.bsc.es" (handle hdl:21.14100/3b089b7d-fd66-3a59-a286-2272be8e8116).
INFO       2021-06-09 12:15:53,038 Publishing: CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn
WARNING    2021-06-09 12:15:54,430 Publication failed for dataset CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn with message: Status=400, Invalid THREDDS catalog at uri: http://esgf.bsc.es/thredds/catalog/esgcet/237/CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210511.xml#CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210511 error: ----Catalog Validation
**Fatal:  InvCatalogFactory.readXML failed
 Exception= java.io.FileNotFoundException http://esgf.bsc.es/thredds/catalog/esgcet/237/CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210511.xml#CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210511
 fatalMessages=
 errMessages=
 warnMessages=

It seems the publisher is still looking for the previous version, even if doesn't exist anymore anywhere (the file thredds/catalog/esgcet/237/CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210511.xml does not exist anymore).

I see that the new version is present in THREDDS (https://esgf.bsc.es/thredds/catalog/esgcet/249/CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608.html) but not in the index node, probably because of the previous error.

In the past we unpublished and republished without any problem, but this is happening for the 3 last datasets we worked on.

Would @lukaszlacinski or @soay (if related to the PIDS, which I doubt) have an idea about what could be happening?

We are publishing from our data node, with LIU as index node and esg-publisher (esgcet) version 3.7.3.

Thanks a lot.

Tagging @aearamos who faced the same issue working with me on these datasets.

[sashakames]

@pabretonniere another option for you is to start using the pre-release v5 of the publisher. This doesn't use THREDDS catalogs at all. https://esg-publisher.readthedocs.io/en/gen-five-pkg/ We are planning a major update soon (still a pre-release) but that doesn't affect the basic usage. The configuration is new, as this is a major release change, although you should be able to migrate your old settings.

[pabretonniere]

Thanks @sashakames ! I'll give it a try with the new version. Is it fully compatible with the previous one? I mean, can I use only the publish command with mapfiles generated with my previous conda env (esgmapfile (from esgprep v2.9.4 2018-10-12))?

[sashakames]

Correct, same mapfiles as before. the esg.ini format is not compatible (you still use the old format with esgmapfile), but the new file format is much simpler and so has a shorter file. Also, should be no surprise, the new version is Python3.8 so you create a new conda env separate from the Python2 publishing tools.

[pabretonniere]

Right. Thanks a lot!

[pabretonniere]

Update on this: we tried to update the publisher but it was not successful. Debugging this with @pchengi it seems to come from the fact we had an old version of the node. We are now upgrading the node with Ansible but still have an issue (https://github.com/ESGF/esgf-ansible/issues/153).

In the case we can't update the node, do you see any possible solution with the current publisher/node?

Thanks

[pabretonniere]

Hi @sashakames

After our discussion in the last CDNOT meeting, here are the details of the run with the latest publisher (esgcetv5 beta):

[root@esgf pbretonn]# conda activate esgf-pub-esgcetv5
[root@esgf pbretonn]# esgpublish --verbose --ini ini --map /esg/mapfiles/aemet_ssp5-ScenarioMIP-r3-republish/CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608.map

I attach the log and strace if it helps. It looks like a permission problem (User: https://esg-dn1.nsc.liu.se/esgf-idp/openid/kserradell_liu is not authorized to publish/unpublish resource: CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608.zostoga_Omon_EC-Earth3_ssp585_r3i1p1f1_gn_209501-209512.nc) but we checked them with Prashanth and they looked OK.

Thank you.

publisher.log

strace.log

[sashakames]

@pchengi2 @pchengi @pabretonniere I hope you all can take another look at the permissions issue. Looking through the trace the only issue I see is that it detected that the dataset version that is being published appears to match the same version that was previously published. If this publication is intended to update a previous version, the recommended practice is to create a new version. But that shouldn't prevent the publication on the index. v20210608

The spot in the logs shows that there already exists a dataset with the same id: "response":{"numFound":1,"start":0,"maxScore":1.0,"docs":[ { "id":"CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608|esgf.bsc.es", "version":"20210608", "score":1.0}] },

[pabretonniere]

Thank you for the answer. This version might come from a previous intent of publication, but the data corresponding to v20210608 is the one we want to publish. But it doesn't appear on the index node anyway, so something else in the workflow must have failed...

[sashakames]

the dataset record appears for me (full record): https://esg-dn1.nsc.liu.se/esg-search/search/?distrib=false&format=application%2Fsolr%2Bjson&data_node=esgf.bsc.es&master_id=CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn However no File records appear, so there must have been a problem at some point. Nonetheless the publisher should replace it in this case.

I'd recommend a different access control string as I've found that the rules application to be at times unpredictable and restrict things I expect to pass.

[pabretonniere]

What do you mean by "I'd recommend a different access control string"?

Would you recommend trying to unpublish the current v20210608 that didn't make it properly to the index node and republish it again with a new version?

[sashakames]

You should not have to unpublish in order to republish in this case. The access control issue is on the index server side. Presumably if there is a problem with access control, you wouldn't be able to unpublish either because of the same rule.

Also, (1) if you haven't change the data for v20210608 then should be no need to create a new version here. (2) we can try to invoke the publication API directly with the xml data in event there is some unforeseen problem with using Python requests as the https client. This seems very unlikely to me however.

[pabretonniere]

@pchengi2 @pchengi, can we discuss about the first point that Sasha is mentioning? What would you recommend?

@sashakames : meanwhile, I would be keen on trying your second option. Could you please give more details (or point me to some documentation) to do it?

Thank you both!

[sashakames]

Sure: https://esgf.github.io/esg-search/REST_Publishing_Services.html#push-operations You should be able to use the XML data in the log. copy/paste from ... Start with the record that has type=File

Apparently the "dataset" passes as I see this now in the log you provided: <?xml version="1.0" encoding="UTF-8"?>Published record: CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608|esgf.bsc.es Done. Cleaning up.

And that explains why the record might appear when I searched but there are no files associated with it. But it raises an issue that the publisher should halt when it encounters the first error of this type.

[pabretonniere]

Thank you, I will check this and let you know!

[pchengi]

Sorry about the radio silence. My wife delivered a child three weeks ahead of schedule, so I couldn't inform about my limited availability. I'm expected to be working one day a week after tomorrow. I'll go through this trail and see what I can do.

Regards Prashanth

Den ons 29 sep. 2021 kl 08:43 skrev pabretonniere @.***

:

Thank you, I will check this and let you know!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESGF/esg-publisher/issues/180#issuecomment-929883293, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3P3I3ORENVBMPFFC5A4WDUEKYR5ANCNFSM46MCFT5Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[sashakames]

@pabretonniere I found a bug in the publisher that impacts the "File" "id" field and that might be responsible for why the publications are failing to return files. I'll try to get a release out but might mean tomorrow. I'll keep you posted.

[pabretonniere]

Thank you @sashakames ! Looking forward to testing the new release!

[pabretonniere]

Hi,

After trying the new version of the publisher (5.1.0b6), I get a different error, solr related it seems.

Log attached.

Note that I had to comment the lines 25 and 26 in esgcet/cmip6.py because it threw me an error.

 if self.replica:
      self.skip_prepare= argdict["skip-prepare"]

Thanks

[sashakames]

Thanks for the update and pointing out the error. Sorry you needed to comment that line. I'm surprised, you didn't set replica = true. That will be fixed in the next release.

I'm not sure why the LiU index is reporting a 500 error. if this happens repeatedly we may need to troubleshoot with Prashanth. For now though I see what's happening. The workaround is to clean up: delete the record that was partially published.

You may need additional arguments for esgunpublish but here's an example:

esgunpublish --delete --dset-id "CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608|esgf.bsc.es"

Then check if it was properly deleted:

curl 'https://esg-dn1.nsc.liu.se/esg-search/search/?latest=true&distrib=false&format=application%2Fsolr%2Bjson&data_node=esgf.bsc.es&master_id=CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn&fields=version,id'

If so you can start the publication run again.

[pabretonniere]

Excellent, thanks!

Now I could unpublish the data and the new publish works fine until getting a permission error from LIU. I will see this with Prashanth.

(esgf-pub-v5.1.0-b6) [root@esgf esgcet-5]# esgpublish --verbose --ini ini --map /esg/mapfiles/aemet_ssp5-ScenarioMIP-r3-republish/CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608.map

(...)
  <field name="id">CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608.zostoga_Omon_EC-Earth3_ssp585_r3i1p1f1_gn_202101-202112.nc|esgf.bsc.es</field>
  <field name="short_description">EC-Earth3 output prepared for CMIP6</field>
  <field name="replica">True</field>
  <field name="latest">true</field>
  <field name="type">File</field>
  <field name="project">CMIP6</field>
  <field name="version">20210608</field>
  <field name="dataset_id_template_">%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s</field>
  <field name="directory_format_template_">%(root)s/%(mip_era)s/%(activity_drs)s/%(institution_id)s/%(source_id)s/%(experiment_id)s/%(member_id)s/%(table_id)s/%(variable_id)s/%(grid_label)s/%(version)s</field>
  <field name="variable_long_name">Global Average Thermosteric Sea Level Change</field>
  <field name="cf_standard_name">global_average_thermosteric_sea_level_change</field>
  <field name="variable_units">m</field>
  <field name="variable">zostoga</field>
  <field name="dataset_id">CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608|esgf.bsc.es</field>
  <field name="tracking_id">hdl:21.14100/be932f17-c3a8-4a6f-8c90-566a1412f2c5</field>
  <field name="size">40379</field>
  <field name="timestamp">2021-05-07T08:47:30Z</field>
  <field name="checksum">f62abafee844a1bd6f73e60972b5b297e8171c256d582f1d1259db7e23f5ec58</field>
  <field name="checksum_type">SHA256</field>
  <field name="url">https://esgf.bsc.es/thredds/fileServer/esgf_data/CMIP6/ScenarioMIP/EC-Earth-Consortium/EC-Earth3/ssp585/r3i1p1f1/Omon/zostoga/gn/v20210608/zostoga_Omon_EC-Earth3_ssp585_r3i1p1f1_gn_202101-202112.nc|application/netcdf|HTTPServer</field>
  <field name="url">https://esgf.bsc.es/thredds/dodsC/esgf_data/CMIP6/ScenarioMIP/EC-Earth-Consortium/EC-Earth3/ssp585/r3i1p1f1/Omon/zostoga/gn/v20210608/zostoga_Omon_EC-Earth3_ssp585_r3i1p1f1_gn_202101-202112.nc.html|application/opendap-html|OPENDAP</field>
  <field name="citation_url">http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608.json</field>
  <field name="pid">hdl:21.14100/3b089b7d-fd66-3a59-a286-2272be8e8116</field>
</doc>

/data/home/pbretonn/conda/envs/esgf-pub-v5.1.0-b6/lib/python3.8/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'esg-dn1.nsc.liu.se'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(

2021-10-26 18:15:06 INFO     <?xml version="1.0" encoding="UTF-8"?><response status="error"><message>User: https://esg-dn1.nsc.liu.se/esgf-idp/openid/kserradell_liu is not authorized to publish/unpublish resource: CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608.zostoga_Omon_EC-Earth3_ssp585_r3i1p1f1_gn_202101-202112.nc|esgf.bsc.es</message></response>
2021-10-26 18:15:06 INFO     <?xml version="1.0" encoding="UTF-8"?><response status="error"><message>User: https://esg-dn1.nsc.liu.se/esgf-idp/openid/kserradell_liu is not authorized to publish/unpublish resource: CMIP6.ScenarioMIP.EC-Earth-Consortium.EC-Earth3.ssp585.r3i1p1f1.Omon.zostoga.gn.v20210608.zostoga_Omon_EC-Earth3_ssp585_r3i1p1f1_gn_202101-202112.nc|esgf.bsc.es</message></response>
2021-10-26 18:15:06 ERROR    code = 401
2021-10-26 18:15:06 ERROR    code = 401

[pabretonniere]

Hi @pchengi

When you can, could you have a look at the error I'm having please? It looks like a permission issue on your side.

If you prefer to communicate through the ESGF slack or by email.

Thank you very much

[aearamos]

Hi,

Following @pabretonniere's error here at BSC, I wanted do add the steps I did to try to publish one variable of one of our experiments:

1. esgdrs upgrade --project cmip6 /esarchive/$exp/cmorfiles/$mip --symlink --rescan
2. esgmapfile --project cmip6 /data/${exp}-${mip}-${member} --outdir /data/mapfiles/${exp}-${mip}-${member}
3. esgpublish --verbose --ini a3o2-Omon-tos.ini --map CMIP6.CMIP.EC-Earth-Consortium.EC-Earth3-CC.historical.r8i1p1f1.Omon.tos.gn.v20210702.map --verify &> log_a3o2_Omon_tos.log

I first ran the esgdrs command to create the links, then the esgmapfile command to generate the mapfiles, and then I loaded the last version of the publisher to try to publish one of the variables. I'm attaching the log file from that last run.

log_a3o2_Omon_tos.log

[sashakames]

@aearamos I took a look at the log, and you were getting "success" messages, so presumably the publication had worked. Unfortunately the index node at LiU is offline at the moment (Prashanth is addressing a security concern) so we can't verify the publication.

[pabretonniere]

Thanks. Actually we were surprised not to see errors in the logs. But we tried this command already a few weeks ago and at this time, I think the LIU node was up and running (we could see other of our data published there).

But we can come back in a few days when the index node is back to life.

[sashakames]

I checked again (in event there's already a replica) and I found that the LLNL replica shard picked it up. See on our new UI (pre-beta test)

https://aims2.llnl.gov/metagrid/search?project=CMIP6&data=%7B%22activeFacets%22%3A%7B%22source_id%22%3A%5B%22EC-Earth3-CC%22%5D%2C%22experiment_id%22%3A%5B%22historical%22%5D%2C%22variant_label%22%3A%5B%22r8i1p1f1%22%5D%7D%7D

(if the server is down try again in a few minutes as we may be updating it at some point...)

[pabretonniere]

Indeed. It seems to be findable. And the data that @aearamos mentioned and that you found is completely new (it doesn't come from an errata-unpublish-correct-republish) but we never saw it appear on the "classic search" through the portal. So I understand this indicates a LIU/index node issue?

[sashakames]

Correct, maybe check back next week with P on the status of their node. If publishing works now, that's great news, but clearly you need the index node online to go further.

[pabretonniere]

Hi @pchengi

The LIU index node seems to be back but the issue we discussed above with Sasha is still present. Can you have a look please?

Thanks!

[pabretonniere]

Hello @sashakames

I've been able to publish the new data (all successful and no error in the logs, as you can see in an example log attached) and can indeed see it in your new UI:

However, I can't download the files, I get the following message when accessing them through opendap:

Error {
    code = 500;
    message = "java.net.MalformedURLException: /data/CMIP6/DCPP/EC-Earth-Consortium/EC-Earth3/dcppB-forecast/s2021-r1i4p1f1/Amon/pr/gr/v20211222/pr_Amon_EC-Earth3_dcppB-forecast_s2021-r1i4p1f1_gr_202111-202210.dods must start with dods: or http: or file:";
};

Here is the full command I ran:

esgpublish --verbose --ini /data/home/pbretonn/esgcet-5/a3w5-dcppB-forecast-r1i4p1f1.ini --map /data/mapfiles/a3w5-dcppB-forecast-r1i4p1f1/

The corresponding log, ini and an example of mapfiles used are attached in the esgf-files.zip

Do you have an idea of what could be wrong? The URL seems to be right and has the same syntax and directory tree as what we had successfully published before.

Thanks a lot,

[sashakames]

Hi @pabretonniere I took a look, but first wanted to see if I could download the file via http. I'm getting a 404, see

ames4$ wget http://esgf.bsc.es/thredds/fileServer/esg_dataroot/CMIP6/DCPP/EC-Earth-Consortium/EC-Earth3/dcppB-forecast/s2021-r1i4p1f1/Amon/pr/gr/v20211222/pr_Amon_EC-Earth3_dcppB-forecast_s2021-r1i4p1f1_gr_202111-202210.nc
--2022-01-13 07:04:07--  http://esgf.bsc.es/thredds/fileServer/esg_dataroot/CMIP6/DCPP/EC-Earth-Consortium/EC-Earth3/dcppB-forecast/s2021-r1i4p1f1/Amon/pr/gr/v20211222/pr_Amon_EC-Earth3_dcppB-forecast_s2021-r1i4p1f1_gr_202111-202210.nc
Resolving esgf.bsc.es (esgf.bsc.es)... 84.88.53.197
Connecting to esgf.bsc.es (esgf.bsc.es)|84.88.53.197|:80... connected.
HTTP request sent, awaiting response... 404 404
2022-01-13 07:04:08 ERROR 404: 404.

You may want to check your data mount.
If that works, it looks like the OpenDAP format appends the .html extension as a legacy from when we were exposing the THREDDS OpenDAP form, but in too many cases there have been issues with openDAP so the form is of questionable use. In the next version of the publisher, we'll change those to just use .nc and that can be handed off to a opendap client.

[pabretonniere]

Thank you @sashakames

There might indeed be 2 issues. I was investigating the mounting point and I don't get:

My data is located in /data/a3w5-dcppB-forecast-r1i4p1f1/CMIP6/DCPP/EC-Earth-Consortium/EC-Earth3/dcppB-forecast/s2021-r1i4p1f1/Amon/pr/gr/v20211222/pr_Amon_EC-Earth3_dcppB-forecast_s2021-r1i4p1f1_gr_202111-202210.nc

In the esg.ini (attached in the previous comment) called in the esgpublish, I specified the following:

data_roots = {"/data/a3w5-dcppB-forecast-r1i4p1f1": "esg_dataroot"}

whereas in /esg/config/esgcet/esg.ini, I have

thredds_dataset_roots =
        esg_dataroot | /data

So I guess there is a mismatch between the 2 where one is looking for the data in /data/a3w5-dcppB-forecast-r1i4p1f1/CMIP6 while the other one is looking in /data/CMIP6.

[sashakames]

this worked: wget http://esgf.bsc.es/thredds/fileServer/esg_dataroot/a3w5-dcppB-forecast-r1i4p1f1/CMIP6/DCPP/EC-Earth-Consortium/EC-Earth3/dcppB-forecast/s2021-r1i4p1f1/Amon/pr/gr/v20211222/pr_Amon_EC-Earth3_dcppB-forecast_s2021-r1i4p1f1_gr_202111-202210.nc

[sashakames]

I think I have a solution for you. data_roots = {"/data/a3w5-dcppB-forecast-r1i4p1f1": "esg_dataroot"} performs a replacement. So I understand now why the a3w5-... component was missing.

data_roots = {"/data/a3w5-dcppB-forecast-r1i4p1f1": "esg_dataroot/a3w5-dcppB-forecast-r1i4p1f1"} is what you need for now.

I think there might be an issue with "submounts" The common case is that the "project root", eg "CMIP6" is to be found right after the data root. but that might not fit well for all arrangements. If the old publisher supported "/data" -> "esg_dataroot" then that should be the case here as well. I'll need to ensure the logic works or correct if not, as this was updated recently in a PR and I tested various cases but this feature is admittedly lacking a comprehensive specification of valid inputs and outputs. A release update is in the works.

[pabretonniere]

This makes sense. My data has always been in /data/$expid-$exp-$member/CMIP6/... (published with the old and the new publisher)

Now with the new publisher, I was using an esg.ini template with

data_roots = {"/data/$expid-$exp-$member": "esg_dataroot"} where I was substituting the $expid-$exp-$member by its value for each dataset (I've published 10 members and 2 experiments with the new publisher).

With your new release, I will use data_roots = {"/data/$expid-$exp-$member": "esg_dataroot/$expid-$exp-$member"}.

Thanks again!

[sashakames]

Sorry that having a separate {"/data/$expid-$exp-$member": "esg_dataroot/$expid-$exp-$member"} for each tuple sounds inconvenient. So we will get the map {"/data" | "esg_dataroot" } to work soon (or it might now but not sure if you want to be the first tester, as I can't promise I'll look at that until next week given my schedule and commitments.

[sashakames]

Hi @pabretonniere I was able to test the publisher and it should work fine with the data_roots = {"/data" : "esg_dataroot" } setting. Sorry I couldn't confirm that earlier. So no need to specify each tuple under your /data.
This works with v5.1.0b7 which is the latest release (new release forthcoming in next day or so)

[pabretonniere]

Hi,

After talking with @pchengi he suggested me to try (with the old publisher) he suggested me to split the command I was using for publishing in 2:

from:

esgpublish --project cmip6 --map /data/mapfiles/${expid} --service fileservice --thredds --publish

to

esgpublish --project cmip6 --map /data/mapfiles/${expid} --service fileservice --thredds

followed by:

esgpublish --project cmip6 --map /data/mapfiles/${expid} --service fileservice --publish

I did it for an old experiment that we could publish some time ago, the ones that got me the failure I reported initially here and one completely new experiment.

With this strategy, I get some errors that are that I didn't see while merging both keywords as you can see in the logs attached and Prashanth thought our initial issue might come from there.

It's strange because the files are fine and they could be published (=seeable) on https://aims2.llnl.gov/metagrid but we thought it might be worth reporting it.

logs-a3o4.zip

[sashakames]

Hi @pabretonniere We had a hardware issue with the aims2.llnl.gov host. We are working on restoring metagrid. That said, I'm unable to support issues with the old publisher. I'd hope you could continue with the v5.1.x-beta versions (I recommend keeping up to date as we push out bug fixes). If you go that route (rather than reverting to the "old" [v3.x] versions) I'll make every effort to support.

[pabretonniere]

With the new publisher, it works, I can publish the data, both for the DCPP data that I mentioned a few weeks ago and for the one I published this afternoon. I did a try right now and I didn't get any error message.

I can't double check the data can be seen through your new search engine when aims2.llnl.gov is back, but as before, I can't see it in the official search https://esg-dn1.nsc.liu.se/search/cmip6-liu/.

[sashakames]

https://aims2.llnl.gov/metagrid/search is back on line. What are the search criteria? I'm wondering why it doesn't appear on the liu CMIP6 site...

[pabretonniere]

The search was for example https://aims2.llnl.gov/metagrid/search?project=CMIP6&data=%7B%22activeFacets%22%3A%7B%22activity_id%22%3A%5B%22CMIP%22%5D%2C%22data_node%22%3A%5B%22esgf.bsc.es%22%5D%2C%22source_id%22%3A%5B%22EC-Earth3-CC%22%5D%2C%22experiment_id%22%3A%5B%22historical%22%5D%2C%22variant_label%22%3A%5B%22r9i1p1f1%22%5D%7D%7D and it indeed worked, I can see and download the file.

So we are left with the initial issue of why it is not appearing on the LIU and other CMIP6 websites.

@pchengi2 suggested that it might come from the error of the THREDDS that appeared with the old publisher but I understand that it is not the case as the file seems to be correctly published and seeable through your metagrid.

Would it be an option to try to publish this file (or a new one if needed) on another index node to see if the error comes from there?

[sashakames]

@pabretonniere @pchengi I figured out why the published datasets aren't appearing. I noticed this in the metadata:

        "replica":true,

So the publisher configuration likely has replica=true Is is not necessary to republish if you prefer but you would need to use the esg-search/ws REST API to correct the metadata records. I can assist to some extent.

[pabretonniere]

It wooooooorks!!!! Thank you so much @sashakames !!! I could publish a new dataset deactivating the replica and I can see it on the LIU node!

I'll have a look at how to modify the metadata records for the old data and if I can't make it work, I'll talk to you on Slack.

I still don't get why it stopped working at some point as I don't think I ever touched this, but at this point, it doesn't really matter any more, as long as the issue is fixed!

Thanks again!

[pabretonniere]

Just one minor additional thing, I see the OpenDAP URL given by the metagrid search is not working:

https://aims2.llnl.gov/metagrid/search?project=CMIP6&data=%7B%22activeFacets%22%3A%7B%22activity_id%22%3A%5B%22CMIP%22%5D%2C%22data_node%22%3A%5B%22esgf.bsc.es%22%5D%2C%22source_id%22%3A%5B%22EC-Earth3-CC%22%5D%2C%22experiment_id%22%3A%5B%22historical%22%5D%2C%22variant_label%22%3A%5B%22r9i1p1f1%22%5D%7D%7D

gives

https://esgf.bsc.es/thredds/dodsC/esg_dataroot/a3o4-r9-noreplica/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-CC/historical/r9i1p1f1/Amon/uas/gr/v20220302/uas_Amon_EC-Earth3-CC_historical_r9i1p1f1_gr_185001-185012.dods

It is missing the .nc.

https://esgf.bsc.es/thredds/dodsC/esg_dataroot/a3o4-r9-noreplica/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-CC/historical/r9i1p1f1/Amon/uas/gr/v20220302/uas_Amon_EC-Earth3-CC_historical_r9i1p1f1_gr_185001-185012.nc.dods works though. Same for the thredds search in the official CMIP website.

But the http download works, so that's good.

[sashakames]

On the OpenDAP urls: This should be corrected if using the most recent publisher version (v5.1.0b8). I recall there was an issue and we had the template wrong .dods, or .html extension . As this error proliferated we'll try to get it corrected in the Metagrid copy url feature if feasible.

But best practice to upgrade the publisher. An updated release is close so you may prefer to delay for a day or so.

[tiantdk]

Hi I have similar failure. 1) I have one dataset $expid, its mapfiles was made from its original path /data/$user/$expid. I published the mapfiles info to local PostgreSQL database with esgpublish --no-create-cim --project cmip6 --map /esg/mapfiles/$expid --service fileservice.

2) Then I moved $expid to /data/cmip6/$expid, because all other cmip6 experiments are published there with a common esg.ini "thredds_dataset_roots = cmip6 | /data/cmip6"

3) I deleted the previous mapfiles, and made new mapfiles from the path /data/cmip6/$expid. I failed to publish these new mapfiles info to local PostgreSQL database. with error "... INFO 2023-03-16 14:01:14,772 New dataset version = 20201125 Traceback (most recent call last): File "/usr/local/conda/envs/esgf-pub/bin/esgpublish", line 783, in main(sys.argv[1:]) File "/usr/local/conda/envs/esgf-pub/bin/esgpublish", line 677, in main handlerExtraArgs=handler_extra_args, commitEvery=commitEvery, validate_standard_name=validate_standard_name) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet/publish/utility.py", line 857, in iterateOverDatasets pid_connector=pid_connector, test_publication=test_publication, commitEvery=commitEvery, context) File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/esgcet/publish/extract.py", line 278, in extractFromDataset raise ESGPublishError("Error in creating dataset version, did you already publish this version to the database**?") esgcet.exceptions.ESGPublishError: Error in creating dataset version, did you already publish this version to the database?"

How can I clean up the previous info on the local PostgreSQL database only for $expid, without affecting other cmip6 data publication? Can I use this " esgunpublish --project cmip6 --map /esg/mapfiles/$expid --database-delete --delete"

[sashakames]

@tiantdk Is this still an issue for you? if so, I recommend you upgrade your publisher. Once done you should be able to publish new versions without relying r PostgreSQL database issues. Please see https://esg-publisher.readthedocs.io/en/latest/ for more information.

[tiantdk]

@sashakames I have solved the problem by using the above mentioned command and the current version is 3.7.3. Many thanks for your suggestion! I will consider an update.