search

stream-project / WP4R

Work Package 4 (Leitung:Infai)
1 stars 0 forks source link

Create DCAT endpoint for DSMS #6

Closed TBoonX closed 3 years ago

TBoonX commented 3 years ago
simeonackermann commented 3 years ago

See https://github.com/stream-project/WP4R/issues/24 for DCAT endpoint requirements

yoavnash commented 3 years ago

Hi @TBoonX, can you refer me to the swagger yml file that is used for this entry point: https://nomad-lab.eu/prod/rae/dcat/ ? We will try to unify our API so that it's consistent with the one of Nomad.

TBoonX commented 3 years ago

@yoavnash its hosted by FHI. @markus1978 or @mospring would you provide the file?

markus1978 commented 3 years ago

I attached the swagger.json. I needed to rename it to .txt due to github security.

Be aware that the swagger is generated from our API implementation (not the other way around). If we need to change the API, the swagger will change.

The download link on the API dashboard is kinda broken. For some reason an exception is thrown if the swagger is downloaded with certain headers. E.g. can download with curl, can't open with Chrome. But that is an unrelated issue, we have to fix at our end.

dcat_swagger.json.txt

yoavnash commented 3 years ago

Got it, thanks!

yoavnash commented 3 years ago

Done. Endpoint is available here: http://dsms.stream-dataspace.net/

TBoonX commented 3 years ago

The website works well and with curl it works also fine. But with CKAN I have issues. It returns 500 INTERNAL SERVER ERROR. Here is the response CKAN gets when calling the /catalog/ route: (Thats how much I was able to retrieve with Wireshark)

DSMS_Error.txt

@yoavnash Do you have an idea what the issue is?

yoavnash commented 3 years ago

Not really. Do you know how CKAN's HTTP requests look like?

TBoonX commented 3 years ago
Bildschirmfoto 2021-04-27 um 17 18 17
yoavnash commented 3 years ago

@yzuuang please have look. I would suggest trying to imitate this request with an HTTP client app (postman or Talend) and see what you get.

TBoonX commented 3 years ago

It seems that the accept header is missing which creates an error. I propose to make text/turtle the default

yzuuang commented 3 years ago

@TBoonX, thank you for this information. The '*' was not considered when handling the format. Now it should be fixed, but I am waiting for someone who has the right to update it on the server.

yoavnash commented 3 years ago

I just updated the server. Thanks @yzuuang @TBoonX please see if the problem is now solved.

TBoonX commented 3 years ago

Works! I was able to retrieve 5 datasets. Is this correct? I also started the harvest on stream-dataspace.net. It will be visible under https://stream-dataspace.net/harvest/dsms-dcat-harvester

yoavnash commented 3 years ago

Five is correct. Great!

TBoonX commented 3 years ago

Hello again. While using the DCAT endpoint I realized that the return of the routes have to be changed. For NOMAD we already changed it and it works. Now also the DSMS needs this change. This and the following comments https://github.com/stream-project/WP4R/issues/2#issuecomment-834353721 are explaining the needed change. Is this change possible to do this month? Sorry for the late request.

yoavnash commented 3 years ago

No problem. @yzuuang, could you look into this?

yoavnash commented 3 years ago

Hi @TBoonX, I deployed the changes that @yzuuang has made in http://dsms.stream-dataspace.net/ Can you check whether they fit your requirements?

TBoonX commented 3 years ago

It works, thank you! There are just a few minor issues:

yoavnash commented 3 years ago

The first point is done. For the second one, we first need to collect the raw data. We'll keep you updated.

yzuuang commented 3 years ago

There is the information about the distribution now.

TBoonX commented 3 years ago

It works! Thanks. I just found one tiny thing: dcat:title of dcat:Distribution is wrong, it has to be dct:title

yoavnash commented 3 years ago

Should be ok now

TBoonX commented 3 years ago

Works.