search

nomad-coe / nomad

NOMAD lets you manage and share your materials science data in a way that makes it truly useful to you, your group, and the community.
https://nomad-lab.eu
Apache License 2.0
64 stars 14 forks source link

"Publish upload to central NOMAD" does not work. #17

Closed ondracka closed 2 years ago

ondracka commented 3 years ago

I'm not sure if this is actually supposed to work or not but I got a report form user that it is not working right now and I can confirm it. If this is broken, maybe we can get rid of the button in the uploads page in the meantime to not confuse users?

ERROR    nomad.processing     2021-05-14T10:04:22 Could not upload to central NOMAD
  - nomad.commit: 
  - nomad.deployment: standard
  - nomad.processing.proc: Upload
  - nomad.processing.process: publish_from_oasis
  - nomad.processing.process_status: RUNNING
  - nomad.processing.status_code: 401
  - nomad.processing.task: cleanup
  - nomad.processing.tasks_status: SUCCESS
  - nomad.processing.upload_name: None
  - nomad.processing.user: 66743202-925f-4fe6-80f0-27f0eb4e17dc
  - nomad.processing.user_name: Ondračka Pavel
  - nomad.release: devel
  - nomad.service: unknown nomad service
  - nomad.upload_id: M2paJhHaRlKGx7DQaQHf2g
  - nomad.version: 0.10.4
ondracka commented 3 years ago

Or maybe this is broken because of some settings in our Oasis?

markus1978 commented 3 years ago

The Oasis is HTTP putting the upload to the central nomad (similar to what you would do as a regular user when uploading with curl). This error is thrown, if the central nomad replies with something that is not 200 to this HTTP request.

I see an Oasis upload from May 14th with this upload id. The request was denied with 401, which suggests a config problem. There are a few things that need to be set to make this work.

With this we should try this again.

Of course, we should add this to our documentation.

ondracka commented 3 years ago

I added deployment_id but can't retest the upload now due to: https://matsci.org/t/uploads-page-keeps-redownloading-the-uploads-info/36617

markus1978 commented 3 years ago

You can reset the processing state of an upload from the CLI:

nomad admin uploads reset -- <upload-id>
ondracka commented 3 years ago

I uploaded and published a new test upload locally to have a clean testcase and the publish to central NOMAD is still not working. Could you check on the server what is the issue this time (upload_id: NWBHESk6TzmbMbhPq1uXJA)?

markus1978 commented 3 years ago

It is the same error code, probably the same thing. I can see the changed deployment_id though. I'll have to investigate some more. See also my email.

markus1978 commented 3 years ago

There was indeed a bug. It was not caught by our tests, because the mocked test authentication is slightly different from the actual authentication. A fix is merged into v0.10.4.

ondracka commented 3 years ago

This still doesn't work with latest v0.10.4: 

ERROR    nomad.processing     2021-05-27T15:25:12 Could not upload to central NOMAD
  - nomad.commit: 
  - nomad.deployment: standard
  - nomad.processing.proc: Upload
  - nomad.processing.process: publish_from_oasis
  - nomad.processing.process_status: RUNNING
  - nomad.processing.status_code: 502
  - nomad.processing.task: cleanup
  - nomad.processing.tasks_status: SUCCESS
  - nomad.processing.upload_name: OUTCAR.zip
  - nomad.processing.user: 66743202-925f-4fe6-80f0-27f0eb4e17dc
  - nomad.processing.user_name: Ondračka Pavel
  - nomad.release: devel
  - nomad.service: unknown nomad service
  - nomad.upload_id: NWBHESk6TzmbMbhPq1uXJA
  - nomad.version: 0.10.4
ondracka commented 3 years ago

Actually my bad, I did not read the new docs properly. So using the:

meta:
    release: 'oasis'
    deployment_id: "oasis.mch.rwth-aachen.de"
    maintainer_email: 'pavel.ondracka@........cz'

I'm back at 401 status code.

However thinking about it more, it was not clear from the answer, if the bug was on the central NOMAD side (if it needs new release there) or if using latest v0.10.4 should be enough to make it work?

markus1978 commented 3 years ago

Yes yes, I am sorry. This whole NOMAD to NOMAD thing got me confused. I wasn't thinking this through. Your guess is spot on, the fix is necessary on our side. The central NOMAD is not authorising the oasis admin correctly. I'll have a release Monday the latest.

ondracka commented 3 years ago

Thanks, but there is no rush :-)

ondracka commented 3 years ago

This is now working, the only minor thing I see is that the filename of the upload is lost during transfer to the central NOMAD. In other words, it is uploaded using streaming, but the &name=<name> is not set to the name from the Oasis.

markus1978 commented 2 years ago

I close this. The upcoming nomad v1 will have a reimplemented mechanism and the last comment should not apply anymore.