Extend the Writer API to write single attributes of a record

[sseppi]

As an Open Data Hub Data Editor I want to edit the single attributes descriptions and metadata of a station, to improve the description of the station itself.

This API extension is needed to allow the Open Data Hub Content Editor to edit the description and metadata of the Stations by using the Open Data Hub Data Browser.

We do have to find a strategy to avoid that the data input through the data browser gets overwritten by the next sync with the data provided by the API of the Data Provider and at the same time not loose the original information provided by the API of the Data Provider

• [x] define the Software Architect who will lead the startegy definition
• [ ] create a strategy proposal
• [ ] implement a prototype
• [ ] go into production

[sseppi]

@rcavaliere please verify if the description contains all information we need to implement properly the issue.

[rcavaliere]

@sseppi looks fine. At the end we can summarize this as follows:

• [ ] every station should keep the metadata as automatically provided by the Data Provider [via API]
• [ ] through the Data Browser an authorized user can manually append additional metadata, but without changing the other existing metadata. The user can only edit the metadata that has been created through the Data Browser

I have put the user story the in "To do" lane. @clezag and @dulvui, who can take care of this?

[sseppi]

@rcavaliere how do you suggest to deal with the stations that doesn't have any metadata sent by a provider (e.g. the static eCharging we are now managing in the spreadsheets)?

[rcavaliere]

@sseppi you mean the structure metadata in the API?

[sseppi]

@rcavaliere I mean that, if possible, we should implement a logic that allows to change all fields, excluding the measurements, and blocks only the one that are sent by the data provider so we could use the Data Browser also to enrich the data or input new stations (e.g. the static eCharging we are now managing in the spreadsheets).

[rcavaliere]

@sseppi yes this is the logic we need to introduce. However, all stations have a minimum of mandatory metadata they have to foresee (these are the fields with "s" in front, e.g. "sname"). But I would not separate between mandatory and optional metadata. Everything that comes from a Data Provider automatically can not be changed. Through the Data Browser you can just add metadata (and edit them, of course)

[sseppi]

@rcavaliere fine for me.

Since we have to touch the APIs, I would suggest to link this issue to the following one, which is a riquired change to clarify better where the data comes from and who is responsible for it:

https://github.com/noi-techpark/bdp-core/issues/271

[clezag]

I've discussed this with @dulvui, and we came to a few conclusions:

• I would take this issue
• Doing this right requires a few parts we don't have yet, some of them are currently being worked on:
  • Databrowser writing mobility data
  • Databrowser using a separate API to write (not BDP writer)
  • Permission management in general on writer side
  • Field level ACL
  • Tracking provider, source, license
  • Metadata API and definitions for databrowser
  • Separate metadata records for each provider/source
• We don't really know where to start since everything surrounding it is not yet implemented or even defined
• Instead of losing ourselves in thinking up elaborate systems, we should start small with a use case and build up from there

Because there is so much missing, we would propose to do this in the simplest manner possible, and replace/extend it once we have the other things in place. This is assuming we have the use case and it's important to get it up and running in a reasonable time.

Our proposal would be:

1. let the databrowser access the BDP writer API with a separate user
2. restrict the stations this user can modify by stationtype/origin with a simple hardcoded condition
3. disable pushing data or data types for this account, again using a simple condition
4. modify the data collector so that it only overwrites the fields it provides itself. Where it makes sense, we extend the writer API and client library to support this as a standard feature.

2. will be replaced by proper permission management (https://github.com/noi-techpark/odh-api-core/issues/384) 3. will be done right once we understand better how the databrowser will write mobility data, probably through a separate new API

Happy to hear your feedback

[rcavaliere]

@clezag I completely agree with your proposal. We have a very specific use case, I can provide all the details of it in a different user story. As said, the Data Browser should only add metadata (incl. e.g. images), but without changing what is automatically provided by the source. This should significantly simplify the implementation.

[sseppi]

@clezag fine for me to start from a concrete use case we already have and then generalize it.

The only things I would be careful are:

restrict the stations this user can modify by stationtype/origin with a simple hardcoded condition

I would like to avoid as much as possible hard-coded condition that are difficult to maintain and document. Instead to put a hard-coded condition. For the initial PoC development, I would propose to allow to modify only the metadata of all stations and provide write rights only to core team.

will be done right once we understand better how the databrowser will write mobility data, probably through a separate new API

It would be nice to not have a new separate API, but extend the Ninja API with CRUD functionality.

[clezag]

I've pushed a PoC implementation to test.

Currently, only the stations endpoint is accessible using this. Access is configurable by station type, origin and if someone is able to deactivate existing stations. All these can be configured on Keycloak without touching the application. It works largely identical to the Content API, only the format of how you define resources in Keycloak is different.

This does not fully cover the initial requirements, but the idea is that now we got all the building blocks in place and working.
When we get the next use case, we can tweak it and implement the concrete rules (e.g. can write only a certain field or certain station ID).

[rcavaliere]

@sseppi @ohnewein our proposal with @clezag is NOT to implement this because of the new v2 architecture. At the moment we don't have a use case for this, therefore it is not necessary to implement something like that

[sseppi]

@sseppi @ohnewein our proposal with @clezag is NOT to implement this because of the new v2 architecture. At the moment we don't have a use case for this, therefore it is not necessary to implement something like that

@rcavaliere in the last weeks me and Martin talked with a company that wants to share in both direction their and our Data Catalogue via API using one of the standards that are in use also by the Data Spaces. We are now making a evaluation about what is needed, but is seems that we need to have the information about data provider and license at a data level. For this reason, we probably need to keep working on this issue, to find a proper solution for it.