search

AtlasOfLivingAustralia / extended-data-model

0 stars 0 forks source link

Event download - EMR reported 'Status: False' #105

Closed qifeng-bai closed 1 year ago

qifeng-bai commented 1 year ago

Error from EMR:

Starting DOI service
Checking file available: /tmp/pipelines-export/f5a67abd-2a04-4187-b52e-f1711207d3c4/dr18560.zip
Checking CollectoryService. File available /tmp/pipelines-export/f5a67abd-2a04-4187-b52e-f1711207d3c4/dr18560.zip
Bad response from DOI server
{'status': False}
Finished

This error also indicates something wrong in doi service, but after a code review, we found it is from collectory service, one step before DOI

https://github.com/AtlasOfLivingAustralia/pipelines-airflow/blob/7_variables/dags/ala/doi_service.py#L222

The current code (new changed): resource_url = collectory_server + '/ws/dataResource/' + datasets

the previous was: resource_url = collectory_server + '/dataResource/' + datasets

And the registry_url in DevEnveriment Airflow is defined as

Image

After change registry_url to https://collections-test.ala.org.au/

The download process is completed.

However, I am not sure this change on variable registry_url will fail other DAGs. Changing variable or change code itself , both work on Events download

djtfmartin commented 1 year ago

Thanks @qifeng-bai

variable needs to be "https://collections-test.ala.org.au/ws/" for the rest of the DAGs and certain pipelines (Interpretation).

I'll change doi_service.py accordingly

djtfmartin commented 1 year ago

Fixed in this commit

https://github.com/AtlasOfLivingAustralia/pipelines-airflow/commit/ba076b859d87edbf4958a4edd1e385b736e429b3