search

noi-techpark / it.bz.opendatahub.databrowser

Explore and navigate through Open Data you need to build your next service.
https://databrowser.opendatahub.com
GNU Affero General Public License v3.0
8 stars 7 forks source link

MetaData API - data model and validation #312

Closed gappc closed 1 year ago

gappc commented 1 year ago

The MetaData API model allows almost all properties of a record to be null (see schema definition at https://tourism.opendatahub.com/swagger/index.html#/MetaData/SingleMetaData).

That design has some severe drawbacks, as vital information for MetaData API usage may be missing on some records, like Id, ApiIdentifier, Shortname, ApiUrl, PathParam and many more. For example, the value of ApiIdentifier in https://databrowser.opendatahub.testingmachine.eu/dataset/raw/tourism/v1/MetaData/it.bz.opendatahub.odhactivitypoi.lts.beaconpoi is null, which leads to problems and edge cases in the Data Browser that have to be fixed there. Such problems would probably not exist if the model would be more restrictive.

@RudiThoeni, @sseppi: in my opinion, MetaData API should allow nullable values only where it make sense and where it does no harm.

Further, MetaData API validation in the backend could be improved. It is for example possible to set arbitrary strings as ApiFilter and one can see that some ApiFilter value start with a ?, some others start with a letter. Not having proper backend validation pushes the duty of data-cleanup to the client, which is not ideal because then any client needs to implement the cleanup and handle edge cases.

The task of this issue is to discuss the MetaData model and validation and then to implement the results.

sseppi commented 1 year ago

@gappc thank you for your feedback! I suggest to start the discussion in the meeting on Thursday.

RudiThoeni commented 1 year ago

@gappc Yes i can do this, there is space for improvements since the metadata api was done quick without much testing. -For nullable properties i have to check because the framework is marking strings as nullable automatically -i changed the ApiFilter to an Array and removed all special chars

RudiThoeni commented 1 year ago

@gappc i set ApiIdentifier, ApiFilter, Id, OdhType, ApiVersion, PathParam as non nullable ApiUrl is built out of them and is readonly true

gappc commented 1 year ago

@RudiThoeni: the changes for nullable / readonl look good, thank you.

One more question: what do you say about the data validation I mentioned? Should we perhaps discuss it in a dedicated issue?

gappc commented 1 year ago

@sseppi, @mrabans, @RudiThoeni, @pkritzinger, @mmalfertheiner, @MatteoBiasi: I fixed the handling of MetaData API for ApiFilters, the https://databrowser.opendatahub.testingmachine.eu/dataset-overview works again as intended, sry for the delay.

sseppi commented 1 year ago

@gappc thank you for the notification! I will test it and let you know.

RudiThoeni commented 1 year ago

@gappc Since the validation is in the backend i can always add it later when i have more time, let us keep open this issue in the meantime

gappc commented 1 year ago

@RudiThoeni @sseppi can we close this issue?

sseppi commented 1 year ago

I think we can close it, @RudiThoeni do you agree?