search

FraunhoferISST / DataspaceConnector

This is an IDS Connector reference implementation.
https://www.isst.fraunhofer.de/de/geschaeftsfelder/datenwirtschaft/technologien/Dataspace-Connector.html
Apache License 2.0
103 stars 80 forks source link

Artifact data not retrievable #275

Closed mkollenstart closed 3 years ago

mkollenstart commented 3 years ago

Describe the bug After creating all the relevant resources at the provider side, when the consumer tries to consume the data it is throwing an error message. Also the previously working /data endpoint at the provider side is not working anymore.

To Reproduce Provider side:

> provider="https://dataweek.euh4d.dataspac.es"

> catalog=`curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$provider/api/catalogs" -d '{}' | jq -r '."_links"."self"."href"'`
> echo "Catalog ID: $catalog"
Catalog ID: https://dataweek.euh4d.dataspac.es/api/catalogs/4c9ec68b-751a-46b6-b6bb-d9b3908cc40f

> representation=`curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$provider/api/representations" -d '{"title": "Dataweek Test representation"}' | jq -r '."_links"."self"[0]."href"'`
> echo "Representation ID: $representation"
Representation ID: https://dataweek.euh4d.dataspac.es/api/representations/77ce4f00-4ea6-49f8-91ed-e252fb995a91

> offer=`curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$provider/api/offers" -d '{"title":"Dataweek Test resources","description":"This is an example resource containing weather data.","keywords":["weather","data","sample"],"publisher":"https://openweathermap.org/","language":"EN","licence":"http://opendatacommons.org/licenses/odbl/1.0/","sovereign":"https://openweathermap.org/","endpointDocumentation":"https://example.com"}' | jq -r '."_links"."self"."href"'`
> echo "Offer ID: $offer"
Offer ID: https://dataweek.euh4d.dataspac.es/api/offers/86eef355-57f9-4987-8bd3-817bc4c8f14d

> artifact=`curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$provider/api/artifacts" -d '{"value": "Dataweek Test Value"}' | jq -r '."_links"."self"."href"'`
> echo "Artifact ID: $artifact"
Artifact ID: https://dataweek.euh4d.dataspac.es/api/artifacts/9adaa7b2-9c3e-4bd6-acfc-a52bf1da644a

> contract=`curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$provider/api/contracts" -d '{"title": "Dataweek Contract", "start": "2021-05-01T00:00:00.000Z", "end": "2021-07-01T00:00:00.000Z"}' | jq -r '."_links"."self"."href"'`
> echo "Contract ID: $contract"
Contract ID: https://dataweek.euh4d.dataspac.es/api/contracts/27f89022-b827-4bdd-8d35-78e5173b2620

> rule=`curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$provider/api/rules" -d '{"value":"{\"@type\":\"ids:Permission\",\"@id\":\"https://w3id.org/idsa/autogen/permission/cf1cb758-b96d-4486-b0a7-f3ac0e289588\",\"ids:action\":[{\"@id\":\"idsc:USE\"}],\"ids:description\":[{\"@value\":\"provide-access\",\"@type\":\"http://www.w3.org/2001/XMLSchema#string\"}],\"ids:title\":[{\"@value\":\"Example Usage Policy\",\"@type\":\"http://www.w3.org/2001/XMLSchema#string\"}]}"}' | jq -r '."_links"."self"."href"'`
> echo "Rule ID: $rule"
Rule ID: https://dataweek.euh4d.dataspac.es/api/rules/001d82fb-87b0-4aeb-9b9a-beb6ff594884

> curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$catalog/offers" -d "[\"$offer\"]"
{"_embedded":{"resources":[{"creationDate":"2021-05-21T07:36:31.430+0000","modificationDate":"2021-05-21T07:36:31.430+0000","title":"Dataweek Test resources","description":"This is an example resource containing weather data.","keywords":["weather","data","sample"],"publisher":"https://openweathermap.org/","language":"EN","licence":"http://opendatacommons.org/licenses/odbl/1.0/","version":1,"sovereign":"https://openweathermap.org/","endpointDocumentation":"https://example.com","additional":{},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/offers/86eef355-57f9-4987-8bd3-817bc4c8f14d"},"contracts":{"href":"https://dataweek.euh4d.dataspac.es/api/offers/86eef355-57f9-4987-8bd3-817bc4c8f14d/contracts{?page,size}","templated":true},"representations":{"href":"https://dataweek.euh4d.dataspac.es/api/offers/86eef355-57f9-4987-8bd3-817bc4c8f14d/representations{?page,size}","templated":true},"catalogs":{"href":"https://dataweek.euh4d.dataspac.es/api/offers/86eef355-57f9-4987-8bd3-817bc4c8f14d/catalogs{?page,size}","templated":true}}}]},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/catalogs/4c9ec68b-751a-46b6-b6bb-d9b3908cc40f/offers?page=0&size=30"}},"page":{"size":30,"totalElements":1,"totalPages":1,"number":0}}

> curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$offer/catalogs" -d "[\"$catalog\"]"
{"_embedded":{"catalogs":[{"creationDate":"2021-05-21T07:36:30.663+0000","modificationDate":"2021-05-21T07:36:30.663+0000","title":"","description":"","additional":{},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/catalogs/4c9ec68b-751a-46b6-b6bb-d9b3908cc40f"},"offers":{"href":"https://dataweek.euh4d.dataspac.es/api/catalogs/4c9ec68b-751a-46b6-b6bb-d9b3908cc40f/offers{?page,size}","templated":true}}}]},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/offers/86eef355-57f9-4987-8bd3-817bc4c8f14d/catalogs?page=0&size=30"}},"page":{"size":30,"totalElements":1,"totalPages":1,"number":0}}

> curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$offer/representations" -d "[\"$representation\"]"
{"_embedded":{"representations":[{"creationDate":"2021-05-21T07:36:31.155+0000","modificationDate":"2021-05-21T07:36:31.155+0000","remoteId":"genesis","title":"Dataweek Test representation","mediaType":"","language":"EN","additional":{},"_links":{"self":[{"href":"https://dataweek.euh4d.dataspac.es/api/representations/77ce4f00-4ea6-49f8-91ed-e252fb995a91"},{"href":"https://dataweek.euh4d.dataspac.es/api/representations/77ce4f00-4ea6-49f8-91ed-e252fb995a91"}],"artifacts":{"href":"https://dataweek.euh4d.dataspac.es/api/representations/77ce4f00-4ea6-49f8-91ed-e252fb995a91/artifacts{?page,size}","templated":true},"offers":{"href":"https://dataweek.euh4d.dataspac.es/api/representations/77ce4f00-4ea6-49f8-91ed-e252fb995a91/offers{?page,size}","templated":true}}}]},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/offers/86eef355-57f9-4987-8bd3-817bc4c8f14d/representations?page=0&size=30"}},"page":{"size":30,"totalElements":1,"totalPages":1,"number":0}}

> curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$representation/artifacts" -d "[\"$artifact\"]"
{"_embedded":{"artifacts":[{"creationDate":"2021-05-21T07:36:31.773+0000","modificationDate":"2021-05-21T07:36:31.773+0000","remoteId":"genesis","title":"","numAccessed":0,"byteSize":40,"checkSum":2757627060,"additional":{},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/artifacts/9adaa7b2-9c3e-4bd6-acfc-a52bf1da644a"},"data":{"href":"https://dataweek.euh4d.dataspac.es/api/artifacts/9adaa7b2-9c3e-4bd6-acfc-a52bf1da644a/data"},"representations":{"href":"https://dataweek.euh4d.dataspac.es/api/artifacts/9adaa7b2-9c3e-4bd6-acfc-a52bf1da644a/representations{?page,size}","templated":true},"agreements":{"href":"https://dataweek.euh4d.dataspac.es/api/artifacts/9adaa7b2-9c3e-4bd6-acfc-a52bf1da644a/agreements{?page,size}","templated":true}}}]},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/representations/77ce4f00-4ea6-49f8-91ed-e252fb995a91/artifacts?page=0&size=30"}},"page":{"size":30,"totalElements":1,"totalPages":1,"number":0}}

> curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$offer/contracts" -d "[\"$contract\"]"
{"_embedded":{"contract":[{"creationDate":"2021-05-21T07:36:32.054+0000","modificationDate":"2021-05-21T07:36:32.054+0000","title":"Dataweek Contract","start":"2021-05-21T07:36:32.048+0000","end":"2021-07-01T00:00:00.000+0000","consumer":"","additional":{},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/contracts/27f89022-b827-4bdd-8d35-78e5173b2620"},"rules":{"href":"https://dataweek.euh4d.dataspac.es/api/contracts/27f89022-b827-4bdd-8d35-78e5173b2620/rules{?page,size}","templated":true},"offers":{"href":"https://dataweek.euh4d.dataspac.es/api/contracts/27f89022-b827-4bdd-8d35-78e5173b2620/offers{?page,size}","templated":true}}}]},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/offers/86eef355-57f9-4987-8bd3-817bc4c8f14d/contracts?page=0&size=30"}},"page":{"size":30,"totalElements":1,"totalPages":1,"number":0}}

> curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$contract/rules" -d "[\"$rule\"]"
{"_embedded":{"rules":[{"creationDate":"2021-05-21T07:36:32.361+0000","modificationDate":"2021-05-21T07:36:32.361+0000","title":"","value":"{\"@type\":\"ids:Permission\",\"@id\":\"https://w3id.org/idsa/autogen/permission/cf1cb758-b96d-4486-b0a7-f3ac0e289588\",\"ids:action\":[{\"@id\":\"idsc:USE\"}],\"ids:description\":[{\"@value\":\"provide-access\",\"@type\":\"http://www.w3.org/2001/XMLSchema#string\"}],\"ids:title\":[{\"@value\":\"Example Usage Policy\",\"@type\":\"http://www.w3.org/2001/XMLSchema#string\"}]}","additional":{},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/rules/001d82fb-87b0-4aeb-9b9a-beb6ff594884"},"contracts":{"href":"https://dataweek.euh4d.dataspac.es/api/rules/001d82fb-87b0-4aeb-9b9a-beb6ff594884/contracts{?page,size}","templated":true}}}]},"_links":{"self":{"href":"https://dataweek.euh4d.dataspac.es/api/contracts/27f89022-b827-4bdd-8d35-78e5173b2620/rules?page=0&size=30"}},"page":{"size":30,"totalElements":1,"totalPages":1,"number":0}}

> curl -s --user $credentials --output - "$artifact/data"
��Dataweek Test Value%

Consumer side

> consumer="https://dataweek-consumer.euh4d.dataspac.es"
> provider="https://dataweek.euh4d.dataspac.es/api/ids/data"

> resource=`curl -s --user $credentials -X "POST" --data-urlencode "recipient=$provider" --data-urlencode "elementId=$catalog" "$consumer/api/ids/description" | jq -r '."ids:offeredResource"[0]'`
resourceId=`echo $resource | jq -r '."@id"'`
> echo "Resource ID: $resourceId"
Resource ID: https://dataweek.euh4d.dataspac.es/api/offers/86eef355-57f9-4987-8bd3-817bc4c8f14d

> artifact=`echo $resource | jq -r '."ids:representation"[0]."ids:instance"[0]."@id"'`
> permissions=`echo $resource | jq -r '."ids:contractOffer"[0]."ids:permission" | map_values(. + {"ids:target": "'"$artifact"'"})'`

> echo "Artifact ID: $artifact"
> echo "Permissions: $permissions"
Artifact ID: https://dataweek.euh4d.dataspac.es/api/artifacts/9adaa7b2-9c3e-4bd6-acfc-a52bf1da644a
Permissions: [
  {
    "@type": "ids:Permission",
    "@id": "https://dataweek.euh4d.dataspac.es/api/rules/001d82fb-87b0-4aeb-9b9a-beb6ff594884",
    "ids:action": [
      {
        "@id": "idsc:USE"
      }
    ],
    "ids:description": [
      {
        "@value": "provide-access",
        "@type": "http://www.w3.org/2001/XMLSchema#string"
      }
    ],
    "ids:title": [
      {
        "@value": "Example Usage Policy",
        "@type": "http://www.w3.org/2001/XMLSchema#string"
      }
    ],
    "ids:target": "https://dataweek.euh4d.dataspac.es/api/artifacts/9adaa7b2-9c3e-4bd6-acfc-a52bf1da644a"
  }
]

> agreement=`curl -s --user $credentials -H "Content-Type: application/json" -X "POST" "$consumer/api/ids/contract?recipient=\`echo $provider | jq -sRr @uri\`&resourceIds=\`echo $resourceId | jq -sRr @uri\`&artifactIds=\`echo $artifact | jq -sRr @uri\`&download=false" -d "$permissions" | jq -r '."_links"."self"."href"'`
> echo "Agreement Location: $agreement"
Agreement Location: https://dataweek-consumer.euh4d.dataspac.es/api/agreements/620e37a1-798b-4bf3-84b5-59d9579ffd85

> localartifact=`curl -s --user $credentials -X "GET" "$agreement/artifacts" | jq -r '."_embedded"."artifacts"[0]."_links"."data"."href"'`
> echo "Artifact Location: $localartifact"
Artifact Location: https://dataweek-consumer.euh4d.dataspac.es/api/artifacts/97c6b711-954c-48df-9f65-c455ed133d6a/data

> curl -s --user $credentials --output - "$localartifact"
{"message":"An error occurred. Please try again later."}

When I now repeat the last command of the provider side, that previously worked:

curl -s --user $credentials --output - "$artifact/data"
{"message":"An error occurred. Please try again later."}

The logs of the provider connector, for both calls show this stacktrace:

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) [spring-security-web-5.4.6.jar!/:5.4.6]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) [spring-web-5.3.6.jar!/:5.3.6]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) [spring-web-5.3.6.jar!/:5.3.6]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.3.6.jar!/:5.3.6]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.3.6.jar!/:5.3.6]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93) [spring-boot-actuator-2.4.5.jar!/:2.4.5]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.6.jar!/:5.3.6]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707) [tomcat-embed-core-9.0.45.jar!/:?]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.45.jar!/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.45.jar!/:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
mkollenstart commented 3 years ago

The problem seems to have something to do with Postgres.

When I'm using memory backed H2 databases the whole process works correctly, but as soon as I enable Postgres for the provider the error returns.

The used Spring JPA configuration for Postgres is:

SPRING_DATASOURCE_URL=jdbc:postgresql://*********:5432/connectordb
SPRING_DATASOURCE_USERNAME=connector
SPRING_DATASOURCE_PASSWORD=************
SPRING_DATASOURCE_PLATFORM=postgres
SPRING_DATASOURCE_DRIVER_CLASS_NAME=org.postgresql.Driver
SPRING_JPA_DATABASE_PLATFORM=org.hibernate.dialect.PostgreSQLDialect

And the Postgres version is 13.3

The configuration itself doesn't seem to be the problem, since all the requests prior to the error are all working.

brianjahnke commented 3 years ago

It seems the database changes the format of the uris. Thus a query on the provider side fails/returns wrong elements.