search

eclipse-tractusx / tractus-x-umbrella

Where all Tractus-X comes together
Apache License 2.0
12 stars 24 forks source link

Update helm-test post install when upgrading EDC from 0.7.1 to 0.7.7 #159

Closed matbmoser closed 2 weeks ago

matbmoser commented 3 weeks ago

Task

Relates to https://github.com/eclipse-tractusx/tractus-x-umbrella/issues/134 and https://github.com/eclipse-tractusx/tractus-x-umbrella/pull/153

evegufy commented 2 weeks ago

it's not about some test that is failing, it's about a failing post install job which is supposed to be importing/seeding the test data https://github.com/eclipse-tractusx/tractus-x-umbrella/tree/umbrella-2.0.2/charts/tx-data-provider/resources

kss682 commented 2 weeks ago 
  File "/opt/scripts/upload.py", line 253, in create_registry_asset
    print_response(response_)
Policy default-policy already exists. Skipping creation.
Create policy test on EDC http://umbrella-dataprovider-edc-controlplane:8081/management/v2/policydefinitions
<Response [409]>
[{"message":"Policy with ID test already exists","type":"ObjectConflict","path":null,"invalidValue":null}]
http://umbrella-dataprovider-edc-controlplane:8084
http://umbrella-dataprovider-edc-controlplane:8081
Query Catalog for registry asset http://umbrella-dataprovider-edc-controlplane:8081/management/v2/catalog/request
<Response [500]>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Internal Server Error</title>
</head>
<body><h2>HTTP ERROR 500 Internal Server Error</h2>
<table>
<tr><th>URI:</th><td>/management/v2/catalog/request</td></tr>
<tr><th>STATUS:</th><td>500</td></tr>
<tr><th>MESSAGE:</th><td>Internal Server Error</td></tr>
<tr><th>SERVLET:</th><td>EDC-management</td></tr>
</table>

</body>
</html>

  File "/opt/scripts/upload.py", line 168, in print_response
    raise Exception("Failed to call service")
Exception: Failed to call service

The error on the post install job

kss682 commented 2 weeks ago

For the catalog request payload. The only difference I could see is, missing key "@type". catalog doc 0.7.3 code

evegufy commented 2 weeks ago

@kss682 hadn't seen that log yet, I get the helm test point in the context of the post install job now. I noticed another thing, not sure if relevant:

For some reason, I can't find the catalog request endpoint in 0.7.3 and once it's available again - from 0.7.4 onwards - it's v3, not v2 (as in 0.7.1) anymore ... 0.7.7

Maybe it's worth checking the versions of all endpoints here

kss682 commented 2 weeks ago

@evegufy @matbmoser 

Query Catalog for registry asset http://umbrella-dataprovider-edc-controlplane:8081/management/v3/catalog/request
<Response [500]>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Internal Server Error</title>
</head>
<body><h2>HTTP ERROR 500 Internal Server Error</h2>
<table>
<tr><th>URI:</th><td>/management/v3/catalog/request</td></tr>
<tr><th>STATUS:</th><td>500</td></tr>
<tr><th>MESSAGE:</th><td>Internal Server Error</td></tr>
<tr><th>SERVLET:</th><td>EDC-management</td></tr>
</table>

</body>

The issue persists even after changing it to v3. It seems like the issue is because of the this breaking change https://github.com/eclipse-edc/Connector/pull/4185

evegufy commented 2 weeks ago

Hi @kss682 thank you for testing v3, I was just guessing.

Do you need support for upgrading the post install job for seeding the test data to the breaking change?

If so, I'm not so deep into EDC stuff. @matbmoser can you or @saudkhan116 support here? If not, there are also some other committers we could ask, I think.

kss682 commented 2 weeks ago

@evegufy I was reading into the commit, not sure if this would be the only change. "edc:operandLeft": "https://w3id.org/edc/v0.0.1/ns/type" to "edc:operandLeft": {"@id": "https://w3id.org/edc/v0.0.1/ns/type"} in here. All such occurrences would need to be changed.

evegufy commented 2 weeks ago

@kss682 ok good, will you test that change? Once you tested it and it still doesn't work, we'd ask for support from someone.

kss682 commented 2 weeks ago

@evegufy Hello, I did try testing with the changes but the error persists. shall post the same in the chat room.

evegufy commented 2 weeks ago

Hi @kss682 as we have this issue already, please post the error here as comment. I'll write a quick message in the matrix chat thread that all should join the discussion here.

kss682 commented 2 weeks ago

Summary: We upgraded the tractusx-edc to 0.7.7 and tried testing the umbrella helm chart, we could see that the post install test job fails post-install-test script. From the discussions, it was mentioned that this could be due to a breaking change in this PR. Hence we modified the "edc:operandLeft": "https://w3id.org/edc/v0.0.1/ns/type" to "edc:operandLeft": {"@id": "https://w3id.org/edc/v0.0.1/ns/type"} in here. But the errors persists as before. controlplane logs Screenshot from 2024-11-08 14-22-53 Screenshot from 2024-11-08 14-23-49

Similar issue : https://github.com/eclipse-edc/Connector/issues/4160

ndr-brt commented 2 weeks ago

Summary: We upgraded the tractusx-edc to 0.7.7 and tried testing the umbrella helm chart, we could see that the post install test job fails post-install-test script. From the discussions, it was mentioned that this could be due to a breaking change in this PR. Hence we modified the "edc:operandLeft": "https://w3id.org/edc/v0.0.1/ns/type" to "edc:operandLeft": {"@id": "https://w3id.org/edc/v0.0.1/ns/type"} in here. But the errors persists as before. Similar issue : eclipse-edc/Connector#4160

@kss682 could you paste the actual json used for the call and copy and paste the logs in a triple tick block instead of screenshots?

kss682 commented 2 weeks ago

Payload: 

        payload_ = {
            "@context": edc_context(),
            "@type": "CatalogRequest",
            "edc:protocol": "dataspace-protocol-http",
            "edc:counterPartyAddress": f"{edc_url_}/api/v1/dsp",
            "edc:counterPartyId": f"{edc_bpn_}",
            "edc:querySpec": {
                "edc:filterExpression": {
                    "@type": "edc:Criterion",
                    "edc:operandLeft": {"@id": "https://w3id.org/edc/v0.0.1/ns/type"},
                    "edc:operator": {"@id": "odrl:eq"},
                    "edc:operandRight": "data.core.digitalTwinRegistry"
                }
            }
        }

Error logs: 

SEVERE 2024-11-08T13:05:12.292763896 JerseyExtension: Unexpected exception caught
java.lang.NullPointerException: Cannot invoke "jakarta.json.JsonString.getString()" because the return value of "org.eclipse.parsson.JsonObjectBuilderImpl$JsonObjectImpl.getJsonString(String)" is null
    at org.eclipse.parsson.JsonObjectBuilderImpl$JsonObjectImpl.getString(JsonObjectBuilderImpl.java:249)
    at org.eclipse.edc.validator.jsonobject.validators.MandatoryValue.lambda$validate$2(MandatoryValue.java:43)
    at java.base/java.util.Optional.map(Unknown Source)
    at org.eclipse.edc.validator.jsonobject.validators.MandatoryValue.validate(MandatoryValue.java:43)
    at org.eclipse.edc.validator.jsonobject.validators.MandatoryValue.validate(MandatoryValue.java:31)
    at org.eclipse.edc.validator.jsonobject.JsonObjectValidator.lambda$validate$0(JsonObjectValidator.java:60)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)

jakarta.ws.rs.InternalServerErrorException: Failed to compact JsonObject: When compacting an IRI would result in an IRI which could be confused with a compact IRI [code=IRI_CONFUSED_WITH_PREFIX].
    at org.eclipse.edc.web.jersey.providers.jsonld.JerseyJsonLdInterceptor.lambda$compact$2(JerseyJsonLdInterceptor.java:87)
    at org.eclipse.edc.spi.result.AbstractResult.orElseThrow(AbstractResult.java:119)
    at org.eclipse.edc.web.jersey.providers.jsonld.JerseyJsonLdInterceptor.compact(JerseyJsonLdInterceptor.java:87)
    at org.eclipse.edc.web.jersey.providers.jsonld.JerseyJsonLdInterceptor.aroundWriteTo(JerseyJsonLdInterceptor.java:79)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
    at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:85)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:61)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
ndr-brt commented 2 weeks ago

Payload:

        payload_ = {
            "@context": edc_context(),
            "@type": "CatalogRequest",
            "edc:protocol": "dataspace-protocol-http",
            "edc:counterPartyAddress": f"{edc_url_}/api/v1/dsp",
            "edc:counterPartyId": f"{edc_bpn_}",
            "edc:querySpec": {
                "edc:filterExpression": {
                    "@type": "edc:Criterion",
                    "edc:operandLeft": {"@id": "https://w3id.org/edc/v0.0.1/ns/type"},
                    "edc:operator": {"@id": "odrl:eq"},
                    "edc:operandRight": "data.core.digitalTwinRegistry"
                }
            }
        }

@kss682 I'd need the actual payload in json, not python code

kss682 commented 2 weeks ago

@ndr-brt Unfortunately some other post-install job is failing in our azure env that I can't generate the exact payload. But here is the payload without the BPN ID and provider dsp url. 

{
  "@context": {
    "dct": "http://purl.org/dc/terms/",
    "tx": "https://w3id.org/tractusx/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "dcat": "https://www.w3.org/ns/dcat/",
    "odrl": "http://www.w3.org/ns/odrl/2/",
    "dspace": "https://w3id.org/dspace/v0.8/",
    "cx-common": "https://w3id.org/catenax/ontology/common#"
  },
  "@type": "CatalogRequest",
  "edc:protocol": "dataspace-protocol-http",
  "edc:counterPartyAddress": "/api/v1/dsp",
  "edc:counterPartyId": "",
  "edc:querySpec": {
    "edc:filterExpression": {
      "@type": "edc:Criterion",
      "edc:operandLeft": {
        "@id": "https://w3id.org/edc/v0.0.1/ns/type"
      },
      "edc:operator": {
        "@id": "odrl:eq"
      },
      "edc:operandRight": "data.core.digitalTwinRegistry"
    }
  }
}
wolf4ood commented 2 weeks ago

do you know why the edc:counterPartyId is empty? if you remove the filter does it actually works the catalog request?

ndr-brt commented 2 weeks ago

@kss682 I see different issues:

shaped like this, it likely didn't work correctly in 0.7.1 as well a working one would be (please fill in correct values for counter party id and address)

{
  "@context": {
    "dct": "http://purl.org/dc/terms/",
    "tx": "https://w3id.org/tractusx/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "dcat": "https://www.w3.org/ns/dcat/",
    "odrl": "http://www.w3.org/ns/odrl/2/",
    "dspace": "https://w3id.org/dspace/v0.8/",
    "cx-common": "https://w3id.org/catenax/ontology/common#"
  },
  "@type": "CatalogRequest",
  "edc:protocol": "dataspace-protocol-http",
  "edc:counterPartyAddress": "<counter party url>",
  "edc:counterPartyId": "<counter party id>",
  "edc:querySpec": {
    "edc:filterExpression": {
      "@type": "edc:Criterion",
      "edc:operandLeft": "https://w3id.org/edc/v0.0.1/ns/type",
      "edc:operator": "=",
      "edc:operandRight": "data.core.digitalTwinRegistry"
    }
  }
}
kss682 commented 2 weeks ago

Hello @ndr-brt , @wolf4ood , @evegufy I have been trying to set this up locally on a minikube cluster (data-exchange values file)and as there were few issues with the azure env.

The current issue is with the wallet. (Not sure if this occurring because the previous one got resolved) Error on control plane

DEBUG 2024-11-11T14:36:03.498133907 DIM Token expired, need to refresh.
DEBUG 2024-11-11T14:36:03.498880696 [Hashicorp Vault] Failed to resolve secret 'edc-wallet-secret': [Failed to get secret with status 404]
SEVERE 2024-11-11T14:36:03.499069479 JerseyExtension: Unexpected exception caught
org.eclipse.edc.spi.EdcException: Failed to fetch client secret from the vault with alias: edc-wallet-secret
 at org.eclipse.tractusx.edc.identity.mapper.BdrsClientImpl.lambda$resolve$0(BdrsClientImpl.java:101)
 at org.eclipse.edc.spi.result.AbstractResult.orElseThrow(AbstractResult.java:119)
 at org.eclipse.tractusx.edc.identity.mapper.BdrsClientImpl.resolve(BdrsClientImpl.java:101)
 at org.eclipse.tractusx.edc.identity.mapper.BdrsClientAudienceMapper.resolve(BdrsClientAudienceMapper.java:42)
 at org.eclipse.edc.protocol.dsp.http.dispatcher.DspHttpRemoteMessageDispatcherImpl.dispatch(DspHttpRemoteMessageDispatcherImpl.java:120)
 at org.eclipse.edc.connector.core.message.RemoteMessageDispatcherRegistryImpl.dispatch(RemoteMessageDispatcherRegistryImpl.java:48)
 at org.eclipse.edc.connector.controlplane.services.catalog.CatalogServiceImpl.requestCatalog(CatalogServiceImpl.java:44)
 at org.eclipse.edc.connector.controlplane.api.management.catalog.BaseCatalogApiController.requestCatalog(BaseCatalogApiController.java:53)
 at org.eclipse.edc.connector.controlplane.api.management.catalog.v3.CatalogApiV3Controller.requestCatalogV3(CatalogApiV3Controller.java:43)
 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
 at java.base/java.lang.reflect.Method.invoke(Unknown Source)
 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)

To reproduce the error: Follow the steps in here umbrella and run it for values-adopter-data-exchange, after changing the edc version in /tractusx-data-provider to 0.7.7. Then querying the below request gives the error pasted above.

curl --location 'http://dataprovider-controlplane.tx.test/management/v3/catalog/request' \
--header 'Content-Type: application/json' \
--header 'X-Api-Key: TEST2' \
--data-raw '{
  "@context": {
    "dct": "http://purl.org/dc/terms/",
    "tx": "https://w3id.org/tractusx/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "dcat": "https://www.w3.org/ns/dcat/",
    "odrl": "http://www.w3.org/ns/odrl/2/",
    "dspace": "https://w3id.org/dspace/v0.8/",
    "cx-common": "https://w3id.org/catenax/ontology/common#"
  },
  "@type": "CatalogRequest",
  "edc:protocol": "dataspace-protocol-http",
  "edc:counterPartyAddress": "https://dataprovider-controlplane.tx.test/api/v1/dsp",
  "edc:counterPartyId": "BPNL00000003AYRE",
  "edc:querySpec": {
    "edc:filterExpression": {
      "@type": "edc:Criterion",
      "edc:operandLeft": "https://w3id.org/edc/v0.0.1/ns/type",
      "edc:operator": "=",
      "edc:operandRight": "data.core.digitalTwinRegistry"
    }
  }
}'

But querying the ssi-dim-wallet manually works. 

curl -X 'POST' \
  'https://ssi-dim-wallet-stub.tx.test/oauth/token' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'client_id=BPNL00000003AYRE&client_secret=edc-wallet-secret&grant_type='

Screenshot from 2024-11-11 15-44-48

wolf4ood commented 2 weeks ago

@kss682

I think the vault is a separated issue probably in your setup. Anyway I applied

the change similar to your PR + a fix in the policy definition due https://github.com/eclipse-edc/Connector/pull/4185

The changeset is here https://github.com/eclipse-tractusx/tractus-x-umbrella/compare/main...wolf4ood:tractus-x-umbrella:chore/upgrade_tx_edc_0_7_7

And I run the helm lint/test also which is green on my fork

https://github.com/wolf4ood/tractus-x-umbrella/actions/runs/11781562495

Let me know if this helps

Thanks

evegufy commented 2 weeks ago

@kss682 regarding Failed to resolve secret 'edc-wallet-secret', please make sure to have executed a helm dependency updatebefore the install of the umbrellla, could be that you're missing the this change in the tx-data-provider locally when installing.

could you also please integrate the change from @wolf4ood and test it?

kss682 commented 2 weeks ago

@evegufy making the changes to the policy definition as per @wolf4ood suggestion does resolve the issue and the checks are passing.