dataCite reserve error on title

tloubrieu-jpl commented 2 years ago

🐛 Describe the bug

📜 To Reproduce

Steps to reproduce the behavior:

Run the command:

 pds-doi-cmd reserve -s loubrieu@jpl.nasa.gov -i ~/Downloads/DOI_reserve_template.1bebacc3.xlsx --node eng

With following file DOI_reserve_template.1bebacc3.xlsx

See error:

Traceback (most recent call last):
File "/Users/loubrieu/PycharmProjects/pds-doi-service/src/pds_doi_service/core/outputs/web_client.py", line 89, in _submit_content
response.raise_for_status()
File "/Users/loubrieu/PycharmProjects/pds-doi-service/venv/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.test.datacite.org/dois

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/loubrieu/PycharmProjects/pds-doi-service/src/pds_doi_service/core/actions/reserve.py", line 253, in run
output_doi, o_doi_label = self._web_client.submit_content(
File "/Users/loubrieu/PycharmProjects/pds-doi-service/src/pds_doi_service/core/outputs/datacite/datacite_web_client.py", line 81, in submit_content
response_text = super()._submit_content(
File "/Users/loubrieu/PycharmProjects/pds-doi-service/src/pds_doi_service/core/outputs/web_client.py", line 95, in _submit_content
raise WebRequestException(
pds_doi_service.core.entities.exceptions.WebRequestException: DOI submission request to DataCite service failed, reason: 404 Client Error: Not Found for url: https://api.test.datacite.org/dois
Details: ('{"errors":[{"status":"404","title":"The resource you are looking for '
'doesn\'t exist."}]}')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/loubrieu/PycharmProjects/pds-doi-service/venv/bin/pds-doi-cmd", line 33, in <module>
sys.exit(load_entry_point('pds-doi-service', 'console_scripts', 'pds-doi-cmd')())
File "/Users/loubrieu/PycharmProjects/pds-doi-service/src/pds_doi_service/core/cmd/pds_doi_cmd.py", line 42, in main
output = action.run(**kwargs)
File "/Users/loubrieu/PycharmProjects/pds-doi-service/src/pds_doi_service/core/actions/reserve.py", line 284, in run
raise CriticalDOIException(err)
pds_doi_service.core.entities.exceptions.CriticalDOIException: DOI submission request to DataCite service failed, reason: 404 Client Error: Not Found for url: https://api.test.datacite.org/dois
Details: ('{"errors":[{"status":"404","title":"The resource you are looking for '
'doesn\'t exist."}]}')

🕵️ Expected behavior

The reserve action should work, it used to work...

📚 Version of Software Used

Latest snapshot version

🩺 Test Data / Additional context

🏞Screenshots

🖥 System Info

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

🦄 Related requirements

⚙️ Engineering Details

alexdunnjpl commented 2 years ago

@jordanpadams ~do I need to register as a direct member of DataCite to use the test api? Seems like so - have fired off a request.~

Nevermind, just saw the placeholders in the ini. Will get those creds myself.

alexdunnjpl commented 2 years ago

@jordanpadams are the creds you sent me actively in use by anyone to the point where we'd get more reports if they'd failed?

Attempting to log in to https://doi.datacite.org/sign-in with those creds results in no UI activity, and in the background a recurring loop of GETS returning HTTP404 errors and doing their best impression of a DOS attack...

I'll follow up with DataCite, but as unlikely as it seems it may be an issue with the creds and/or the service itself.

Object { isAdapterError: true, stack: "r@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5979:19\nr@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5983:54\nhandleResponse@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:6049:17\nhandleResponse@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:11092:171\nr@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:3003:9\nhandleResponse@https://doi.datacite.org/assets/bracco-2c7a94ac4e0596e6dc06e72fccf87eb7.js:118:158\nr@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:3003:9\nu@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:6018:14\najax/</<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:6056:24\najax/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:6056:35\nw@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5013:12\n_@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5011:128\nt.invoke@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4545:163\nt.flush@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4537:74\nt.flush@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4551:207\nr._end@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4614:9\ne/this._boundAutorunEnd@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4565:574\npromise callback*i/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4521:28\no@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:3445:66\nr._scheduleAutorun@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4628:3\nr._ensureInstance@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4626:86\nr.schedule@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4586:13\n@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:2760:140\ny@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5009:225\nm@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5007:63\ne/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5034:29\nd/</e.onload@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5732:78\nEventHandlerNonNull*d/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5732:57\ne/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5034:6\ne@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5034:89\nd@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5732:36\nh/this.text@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5743:62\ne.determineBodyPromise@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5953:65\najax/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:6052:119\nw@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5013:12\n_@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5011:128\nt.invoke@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4545:163\nt.flush@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4537:74\nt.flush@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4551:207\nr._end@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4614:9\ne/this._boundAutorunEnd@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4565:574\npromise callback*i/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4521:28\no@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:3445:66\nr._scheduleAutorun@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4628:3\nr._ensureInstance@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4626:86\nr.schedule@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4586:13\n@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:2760:140\ny@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5009:225\nh@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5006:35\nm@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5009:43\ne/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5034:29\n_/</s.onload@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5775:2\nn@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:22173:52\nEventHandlerNonNull*c@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:14590:140\ne.prototype._wrapXHR/je/je<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:22214:75\ne.prototype._wrapXHR/je@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:22214:10\nh/</<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:16389:86\n_/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5775:590\ne/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5034:6\ne@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5034:89\n_@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5767:152\n_fetchRequest@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:6064:13\najax@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:6052:15\nfindRecord@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:6026:13\n_fetchRecord/u<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:6758:127\nw@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5013:12\nE/<@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:5020:28\nt.invoke@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4545:163\nt.flush@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4537:74\nt.flush@https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js:4551:207\n", description: undefined, fileName: "https://doi.datacite.org/assets/vendor-909e03b7cb2c6ae708acac8c67e263c6.js", lineNumber: 5979, message: "Ember Data Request GET https://api.datacite.org/providers/nasapds returned a 404\nPayload (application/json; charset=utf-8)\n[object Object]", name: "Error", number: undefined, code: undefined, errors: (1) […] }

alexdunnjpl commented 2 years ago

New credentials provided by Jordan are confirmed valid through the datacite web ui.

A get request to https://api.test.datacite.org/dois returns a list of DOIs (as expected) An empty POST request returns HTTP400 (as expected) A non-compliant POST request returns HTTP422 (as expected) The following minimal example POST request returns HTTP404, so the API seems to be clearly bugged in that it's returning the wrong error code, at a minimum.

{
  "data": {
    "type": "dois",
    "attributes": {

    }
  }
}

@jordanpadams am I good to follow this up with DataCite, or is there something else you'd suggest before I bother them?

jordanpadams commented 2 years ago

@alexdunnjpl hmm, it definitely appears there is a return code error there, but I don't think that is the center of the issue. have you tried see if there is maybe an issue with the spreadsheet inputs it is trying to query?

it almost looks like from the error there may have been a newline introduced somewhere that DataCite doesn't like?

in other words, I would maybe try to see if we can perform any reserve step with good data (maybe start with the default XLS from here and work backwards), and then backtrack from there.

alexdunnjpl commented 2 years ago

@jordanpadams I'm currently unable to perform a (draft) reserve action with even the minimal example from the docs (see "Auto-generated DOI's"), which is even more fundamental a test than a minimal XLS, no?

Given that fact, and:

the endpoint stopped working suddenly (for @tloubrieu-jpl)
the returned error doesn't seem reasonable (should be 400 or similar for a bad request - there's no sane reason I can think of to return 404 for a POST to an endpoint which definitely exists and definitely accepts POSTs)

it doesn't seem unreasonable to think the issue might be on DataCite's end.

Is there anyone else you know of using the reservation function on a regular basis?

I can try with a default XLS, but it seems moot if the endpoint isn't even responding to a null-data reservation when the docs indicate it should.

jordanpadams commented 2 years ago

@alexdunnjpl yes. @rsjoyner . I will ping him offline via email and CC you.

jordanpadams commented 2 years ago

@alexdunnjpl Ron is out until next week, so let's ping DataCite and send them the info

alexdunnjpl commented 2 years ago

@tloubrieu-jpl (not that you'll get to this anytime soon) what credentials and prefix were you using when attempting this reservation?

Note to self - check answer for mismatch between credentials/prefix and use of test API/system in ticket instructions.

alexdunnjpl commented 2 years ago

@jordanpadams I've attempted reservation using the provided template, but run into the same issue.

The only lead I have currently (besides asking DataCite to take on a collaborative role to help me troubleshoot the requests) is to wait for @rsjoyner to get his access to the DataCite test system and attempt reservation, and proceed from there.

If he is unable to perform reservations, it seems like there has to be an issue with either our test-system account or the system itself.

I'll send another email to DataCite and cc you.

jordanpadams commented 2 years ago

@alexdunnjpl can we confirm the test service we are running is using the correct credentials? you should have access to the AWS EC2 instance.

@c-suh can you point Alex to where the service is running?

alexdunnjpl commented 2 years ago

Failure to reserve using curl call occurs due to presence of shell expansion character in credentials

Separately, failure to reserve using doi-service occurs due to value in ./src/pds_doi_service/core/util/conf.ini.default being erroneously loaded/used in preference to ./pds_doi_service.ini

@jordanpadams I'll investigate and fix, just wanted to let you know I'm back on track for this issue and should have it sorted today.

alexdunnjpl commented 2 years ago

This issue should be fixed, but we won't know for sure until @tloubrieu-jpl is back from leave to confirm for certain.

@jordanpadams recommending this be closed via the PR - if I'm wrong and it rears its head again it's not like we won't know immediately.

jordanpadams commented 2 years ago

@alexdunnjpl let's talk about next steps. we can't rely on @tloubrieu-jpl to get this out. he will not be back for several more months.

NASA-PDS / doi-service