search

cthoyt / zenodo-client

A tool for automated uploading and version management of scientific data to Zenodo
MIT License
25 stars 5 forks source link

No publication date found #39

Closed daviddavo closed 2 months ago

daviddavo commented 3 months ago

I'm trying to update some dataset nightly, but it fails every night and, when I go to Zenodo, it turns out that the publication date field is empty

image

I tried to replicate the error in Zenodo Sandbox, but I couldn't...

In case it helps, this is the tracelog from when it fails uploading:

Traceback (most recent call last):
Uploading to zenodo
  File "/opt/hostedtoolcache/Python/3.12.3/x64/bin/dao-utils-upload-dw", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/dao_analyzer/cache_scripts/utils/uploadDataWarehouse.py", line 137, in main
    archiveToZenodo(tmpdir, args.zenodo_max_retries)
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/dao_analyzer/cache_scripts/utils/uploadDataWarehouse.py", line 82, in archiveToZenodo
    raise e
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/dao_analyzer/cache_scripts/utils/uploadDataWarehouse.py", line 75, in archiveToZenodo
    uploadToZenodo([zpath / 'archive.zip'])
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/dao_analyzer/cache_scripts/utils/uploadDataWarehouse.py", line 64, in uploadToZenodo
    raise e
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/dao_analyzer/cache_scripts/utils/uploadDataWarehouse.py", line 55, in uploadToZenodo
    z.update(ZENODO_DEPOSITION_ID, paths)
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/zenodo_client/api.py", line 2[31](https://github.com/Grasia/dao-scripts/actions/runs/9217342059/job/25359211826#step:6:32), in update
    new_deposition_id, new_deposition_data = self._update_submitted_deposition_metadata(deposition_id, res)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/zenodo_client/api.py", line 279, in _update_submitted_deposition_metadata
    res.raise_for_status()
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://zenodo.org/api/deposit/depositions/11271085?access_token=***

And this is the code I used:

https://github.com/Grasia/dao-scripts/blob/2e2013b5a502de9d7a0a9e1ee169459869b4af68/src/utils/uploadDataWarehouse.py#L47-L64

daviddavo commented 3 months ago

I realized that there's no version either...

daviddavo commented 3 months ago

I enabled debug logging for requests, ran everything again, and this is what it prints:

2024-06-04T06:41:50.0425886Z send: b'PUT /api/deposit/depositions/11468336?access_token=*** HTTP/1.1\r\nHost: zenodo.org\r\nUser-Agent: python-requests/2.32.3\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 1910\r\nContent-Type: application/json\r\n\r\n'
2024-06-04T06:41:50.1948944Z DEBUG:urllib3.connectionpool:https://zenodo.org:443 "PUT /api/deposit/depositions/11468336?access_token=*** HTTP/1.1" 400 91
2024-06-04T06:41:50.2291053Z Traceback (most recent call last):
2024-06-04T06:41:50.2304800Z send: b'{"metadata": {"title": "DAO Analyzer dataset", "description": "<p><a href=\\"https://dao-analyzer.science/about\\">DAO-Analyzer</a>&#39;s dataset.</p>\\n\\n<p>Explore data from Decentralized Autonomous Organizations deployed on the DAOstack, DAOhaus and Aragon platforms.</p>", "access_right": "open", "creators": [{"name": "Arroyo, Javier", "affiliation": "Complutense University of Madrid", "orcid": "0000-0001-6127-7538"}, {"name": "Dav\\u00f3, David", "affiliation": "Complutense University of Madrid", "orcid": "0000-0003-1744-8314"}, {"name": "Faqir-Rhazoui, Youssef", "affiliation": "Complutense University of Madrid", "orcid": "0000-0001-7833-1086"}], "keywords": ["online communities", "decentralized autonomous organizations", "blockchain", "ethereum"], "grants": [{"id": "05r0vyz12::RTI2018\\u2010096820\\u2010A\\u2010I00"}, {"id": "10.13039/501100000780::759207"}, {"id": "05r0vyz12::PID2021-127956OB-I00"}], "license": "cc-zero", "imprint_publisher": "Zenodo", "communities": [{"identifier": "eu"}], "notes": "<p>DAO-Analyzer is created under the umbrella of multiple research projects:</p>\\n<ul>\\n<li>Chain Community, funded by the Spanish Ministry of Science and Innovation (<a href=\\"https://produccioncientifica.ucm.es/proyectos/48103/detalle\\" rel=\\"nofollow\\">RTI2018\\u2010096820\\u2010A\\u2010I00</a>) and led by Javier Arroyo and Samer Hassan</li>\\n<li>P2P Models, funded by the European Research Council (ERC-2017-STG 625 grant no.: 75920), led by Samer Hassan.</li>\\n<li>DAOapplications, funded by the Spanish Ministry of Science and Innovation (<a href=\\"https://produccioncientifica.ucm.es/proyectos/551171/detalle\\" rel=\\"nofollow\\">PID2021-127956OB-I00</a>) and led by Javier Arroyo, Samer Hassan and maria Cruz Valiente</li>\\n</ul>", "upload_type": "dataset", "prereserve_doi": {"doi": "10.5281/zenodo.11468336", "recid": 11468336}, "version": "2024-06-04", "publication_date": "2024-06-04"}}'
2024-06-04T06:41:50.2314046Z reply: 'HTTP/1.1 400 BAD REQUEST\r\n'
2024-06-04T06:41:50.2314589Z header: server: nginx
2024-06-04T06:41:50.2321286Z   File "/opt/hostedtoolcache/Python/3.12.3/x64/bin/dao-utils-upload-dw", line 8, in <module>
2024-06-04T06:41:50.2322039Z     sys.exit(main())
2024-06-04T06:41:50.2322504Z              ^^^^^^
2024-06-04T06:41:50.2323614Z   File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/dao_analyzer/cache_scripts/utils/uploadDataWarehouse.py", line 155, in main
2024-06-04T06:41:50.2324633Z     archiveToZenodo(tmpdir, args.zenodo_max_retries)
2024-06-04T06:41:50.2325944Z   File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/dao_analyzer/cache_scripts/utils/uploadDataWarehouse.py", line 83, in archiveToZenodo
2024-06-04T06:41:50.2326955Z     raise e
2024-06-04T06:41:50.2328104Z   File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/dao_analyzer/cache_scripts/utils/uploadDataWarehouse.py", line 76, in archiveToZenodo
2024-06-04T06:41:50.2329193Z     uploadToZenodo([zpath / 'archive.zip'])
2024-06-04T06:41:50.2330353Z   File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/dao_analyzer/cache_scripts/utils/uploadDataWarehouse.py", line 65, in uploadToZenodo
2024-06-04T06:41:50.2331400Z     raise e
2024-06-04T06:41:50.2332408Z   File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/dao_analyzer/cache_scripts/utils/uploadDataWarehouse.py", line 56, in uploadToZenodo
2024-06-04T06:41:50.2333408Z     z.update(ZENODO_DEPOSITION_ID, paths)
2024-06-04T06:41:50.2334623Z   File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/zenodo_client/api.py", line 231, in update
2024-06-04T06:41:50.2335725Z     new_deposition_id, new_deposition_data = self._update_submitted_deposition_metadata(deposition_id, res)
2024-06-04T06:41:50.2336584Z                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-04T06:41:50.2337736Z   File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/zenodo_client/api.py", line 279, in _update_submitted_deposition_metadata
2024-06-04T06:41:50.2338669Z     res.raise_for_status()
2024-06-04T06:41:50.2339661Z   File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
2024-06-04T06:41:50.2340558Z     raise HTTPError(http_error_msg, response=self)
2024-06-04T06:41:50.2342017Z requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://zenodo.org/api/deposit/depositions/11468336?access_token=***

It certainly seems that version and publication_date are there without a problem... I'll contact Zenodo support because the problem seems to be elsewhere

daviddavo commented 3 months ago

The problem seems to be with the grant 05r0vyz12::RTI2018\\u2010096820\\u2010A\\u2010I00, which uses invalid characters

daviddavo commented 2 months ago

The problem was not with the invalid characters, it was with the accepted funders DOI prefixes. See https://developers.zenodo.org/#representation and go to grants

They will eventually change to a new API that supports this