No offers appearing in catalog

[mujina93]

Hi, I set up 2 connectors, I tried to register some resources in the first one (provider) and get them from the second (consumer), but even if I have a catalog with registered offers in the first one, when the second retrieves the catalog, I see no offers in there.

Details of my setup

Both connectors are configured by setting the following fields in application.properties:

spring.security.user.name=*
spring.security.user.password=*
spring.security.app.name=*
spring.security.app.password=* 
configuration.keyStorePassword=*
server.ssl.enabled=false
server.ssl.key-store=*
server.ssl.key-store-password=*

And the following ones in config.json:

idsc:PRODUCTIVE_DEPLOYMENT
dsc:CONNECTOR_ONLINE
file:///conf/<keystore file>

They are both built with docker (Docker version 18.06.3-ce), and the connector version is 5.2.1 (the latest version does not run for me, see issue #401).

They both run behind a reverse proxy that does SSL termination.

At the provider, I registered a catalog, an offer, a representation, an artifact, a contract and a rule, linking those as in the provider-consumer python script example from this repo.

This is what I get if I request /api/catalogs?page=0&size=30 with a GET at the provider:

{'_embedded': {'catalogs': [{'creationDate': '2021-08-09T10:04:08.699+0000',
    'modificationDate': '2021-08-09T10:04:08.699+0000',
    'title': 'Parking Predictions Data Catalog',
    'description': 'Predictions offered by [ui!], containing parking spots in a given area, together with occupancy level and tendency',
    'additional': {},
    '_links': {'self': {'href': 'http://ids.urbanpulse.de/api/catalogs/3bb0d5fe-0ee2-4ea6-b385-d499b91aab06'},
     'offers': {'href': 'http://ids.urbanpulse.de/api/catalogs/3bb0d5fe-0ee2-4ea6-b385-d499b91aab06/offers{?page,size}',
      'templated': True}}}]},
 '_links': {'self': {'href': 'http://ids.urbanpulse.de/api/catalogs?page=0&size=30'}},
 'page': {'size': 30, 'totalElements': 1, 'totalPages': 1, 'number': 0}}

This seems to confirm that I have 1 registered catalog with an offer

And this is the response to a GET to /api/catalogs/3bb0d5fe-0ee2-4ea6-b385-d499b91aab06/offers:

{'_embedded': {'resources': []},
 '_links': {'self': {'href': 'http://ids.urbanpulse.de/api/catalogs/3bb0d5fe-0ee2-4ea6-b385-d499b91aab06/offers?page=0&size=30'}},
 'page': {'size': 30, 'totalElements': 0, 'totalPages': 0, 'number': 0}}

From this it seems that the offer is not appearing among the ones parented to this catalog.

And when I request the catalog's data through the consumer, I first get the provider's description by posting with the consumer to /api/ids/description?recipient=<provider>/api/ids/data&elementId=, then I extract the catalog(s) from the ids:resourceCatalog field in the json I got as response. Finally, I make a POST to /api/ids/description?recipient=<provider>/api/ids/data&elementId=<the catalog I extracted>.

As a response I get:

{'@context': {'ids': 'https://w3id.org/idsa/core/', 'idsc': 'https://w3id.org/idsa/code/'}, '@type': 'ids:ResourceCatalog', '@id': 'http://ids.urbanpulse.de/api/catalogs/3bb0d5fe-0ee2-4ea6-b385-d499b91aab06', 'ids:offeredResource': []}

which shows 0 offers for this catalog.

The documentation says

As stated here [broken link!], an offered resource is only complete if it contains at least one contract offer and at least one representation with at least one artifact. Otherwise, it will not be listed in the IDS self-description because there is no complete data offer.

In my case, my offer is linked to a representation and to a contract, but still it's not appearing in the catalog.

I thought it might have to do with the fact that the urls are with http (since the connectors are set up without SSL), while they are actually available at https, but this is not the case, since the connector doesn't do any additional request at the offers' urls when I request the catalog (and the http ones are redirected anyways to https).

What could be the reason for this? Am I missing something in the protocol through which the consumer should access the data from the provider?

[juliapampus]

In my case, my offer is linked to a representation and to a contract, but still it's not appearing in the catalog. Does your representation also contain an artifact? And does your contract contain a rule?

[mujina93]

Hi!

Yes, as I said

I registered a catalog, an offer, a representation, an artifact, a contract and a rule, linking those as in the provider-consumer python script example

I created the catalog, the offer, a representation, an artifact, a contract, and a rule. And I linked those in the following manner (like in the python example):

catalog <- offer(ed resource) <- representation <- artifact
            ^
            |
         contract <- rule

I should be able the entities I linked when GETting things like /api/offers/{id}/contracts, /api/offers/{id}/representations, /api/catalog/{id}/offers, etc. Right? Unfortunately the lists are always empty.

The way I create the link A <- B is by POSTing at <my url>/api/<A>/{id of A}/<B>, with json body ["<my url>/<B>/{id of B}"].

For example, to link offer 1404db42-6765-4b19-b1ca-d568b59b349e to catalog 250c39da-54b9-4627-8129-5396a2e1 I POST at http://.../api/catalogs/250c39da-54b9-4627-8129-5396a2e17c33/offers with body ["http://.../api/offers/1404db42-6765-4b19-b1ca-d568b59b349e"].

Do you see something wrong in what I am doing?

[mujina93]

I found the issue. It was our reverse proxy + a thing that I didn't know about http redirection.

Apparently, almost all clients implement the wrong behavior when handling 301 and 302 redirections. That is, they turn POSTs into GETs, losing the data payload, when they shouldn't.

Therefore, what was happening was that my POST to link 2 things (e.g. an offer to a catalog) was being downgraded to a GET, and was arriving to the connector as a GET without the payload.

Setting our reverse proxy to redirect with 307 lets POSTs be unchanged, and now everything works!