[Datahub] - Add filter options on the download menu

tomalary commented 6 months ago

On each datasets page, the user can access to options to filter download resources and not only per file extension. The filters can be set up (for example per year).

The filter is a drop-down menu with suggestion and autocomplete search on each drop-down menu.

Draft of the component...

For the display of the download links the component check the tag in the metadata. Then we create the download links and they are displayed. If the tag are not found the component work as the usual downloading component.

As our solution is to create the download links by using API Entrepot (API developed by IGN that allow to load/process/share data services), we are open to discuss about an enhancement that suits everyone.

If this feature is not clear please let us know or ask questions to clarify.

This work is sponsored by IGN.

jahow commented 6 months ago

Hi, thanks for your interest,

Could you please explain what are the tags you mention that let you apply these filters? are these standard ISO fields? Also, how many download links do you typically expect in order for such filtering to become valuable?

IGNF-Xavier commented 5 months ago

Hi, API entrepôt will allow the owner of datasets to configure prepackage zip-files of data (PPK) depending on territories, date of products, format of files, etc. Once done, these PPK can be downloaded at a specific URL thanks to API Telechargement The user of the component in the dasasets page will be able to choose the PPK needed according to user-defined filters :

polygon : x1,y1;x2,y2;… ( EPSG:4326),
editionDateFrom : YYYY-MM-DD,
editionDateTo : YYYY-MM-DD,
lang : FR ou EN
crs : epsg,
zone : FRA (for France entière) or D075 (for Paris)...
format : SHP (for Shapefile)...

The swagger of API telechargement is here : https://data.geopf.fr/telechargement/swagger-ui/index.html (today 16/01, it won't work because of maintenance). The point of access of the capabilities of this download-service will be stored in the metadata of the dataset. If completed, the component appears, if not, the component is not displayed.

When you consider that there are 101 departments in France, maybe the results of this component will need to have pagination.

jahow commented 5 months ago

Thanks for the clarification, it definitely makes more sense! Do we agree that records with this feature would be service metadata, and not datasets?

There has been a discussion for a while about how/if we should show services in the datahub; currently they're filtered out. A component such as the one mentioned here would make sense for other kinds of services (WMS, OGC API etc.), so we could address this use case globally and do a specific implementation for the IGN APIs.

Also, could you please clarify if this work will be funded by IGN?

IGNF-Xavier commented 5 months ago

We (IGN) are currently developing this download component specifically for the datasets pages because there will be plenty of datasets with different years (and eventually format and territories). For instance, a new edition of BD TOPO is available every 6 months since many years, on each departement with different CRS. So in the medatata of BD TOPO, there will be a link for this download service https://[data.geopf.fr/telechargement/resource/BDTOPO?](https://data.geopf.fr/telechargement/resource/BDTOPO?)

Maybe we will dadapt this component for this specific PPK-download service page adding the filter dataset.

Then we (IGN) will develop a map-based download component principally used for datasets which have a high weight (for exemple LidarHD). Moreover this type of dataset are available as tiles of 1km to 1km (for LidarHD)

jahow commented 5 months ago

Ok, I see, then this interface would simply show up next to an API link pointing at the IGN Géoplateforme, regardless of the metadata type we're in. I think such a "capabilities explorer" component would be very useful even outside of the IGN use cases, e.g. listing layers in a WMS endpoint.

Would you agree to participate in a preliminary UX work to properly design this component and its integration in the datahub?

IGNF-Xavier commented 5 months ago

I think there are 2 subjects :

a component n°1 in the dataset page which provides URL of zip-files thanks to a search API telechargement
a component n°2 in the service page which allow to explore the getcapabilities of a service (WMS for example).

For Geoplateforme (the governance is not only supported by IGN) :

Component n°1 is described in this issue. A priori for march 2024
component n°2 is another problem. For the getcapabilities of Geoplateforme on public endpoint, the xml will become bigger and bigger as time goes by. We are under investigation in order to propose :
1. a tool which allow the user/developer to maintain chosen layers in the XML of the getcapabilities and store it
2. a way to record a XML getcap for the organisation during the workflow of API Entrepôt and store it
3. a way to make XML by the governance of Geoplateforme for authorities data in differents thematics with automatic updates if new layers are proposed. In fact, all the layers are in the getcapabilities of the service but we provide a way to obtain XML getcap with only layers pertinent or needed by the users

For 2 : the getcapabilities will be available on each dataset page and on the organisation page (we want to suggest the community to have a specific for each organization with a description (exeample : https://www.data.gouv.fr/fr/organizations/institut-national-de-la-statistique-et-des-etudes-economiques-insee/) For 2 and 3 : the component n°2 will allow to search different XML getcap : organisation, thematic and data authorities.

IGNF-Xavier commented 5 months ago

But to answer : (sorry) yes we agree to participate in a preliminary UX Work. Note that the development teams at IGN for Geoplateforme may be different for each component

jahow commented 5 months ago

@IGNF-Xavier Could you please get in touch with someone at Camptocamp? (your contact information is not public) Thanks!

IGNF-Xavier commented 5 months ago

👍 @fgravin knows @tomalary and me

jahow commented 5 months ago

Meetings note are here (FR): https://semestriel.framapad.org/p/api-telechargement-datahub-a5o8?lang=fr

mmohadIGN commented 5 months ago

This is the workflow we plan to do. Did it seem alright to you? @fgravin and @jahow

---
title: Workflow IGN download API
---
%% doc mermaid ici https://mermaid-js.github.io/mermaid/#/flowchart?id=flowcharts-basic-syntax
flowchart TD
    A("record-apis-component") -->|choose proxy compenent API| B("feature/record/proxy-apis-component")
    B -->|do| C(feature/record/ign-api-dl)
    C -->|begin| D(IGN API client)
    C -->|display| E(ui/ign-dl-form)
    D <-->|look at| C

jahow commented 5 months ago

Thanks for working on this and giving more details:

I think there is no need to introduce a "proxy" component; currently the gn-ui-record-api-form is used exclusively but I think it would make sense to only use this component if the protocol is ogcFeatures, and introduce a new component called e.g. gn-ui-record-ign-api-form based on the IGN-specific protocol code; the choice between both can be done in the record-apis-component
a dedicated ign-dl-form presentation component might not be needed; IIRC the component will mostly be comprised of several dropdown/search fields and a list of results; the smart component (e.g. gn-ui-record-ign-api-form) could hold the logic, interact with the IGN API client and place the relevant presentation components; this would avoid introducing another layer of complexity

Pinging @fgravin because there may be things that were discussed before that I don't know about :slightly_smiling_face:

fgravin commented 5 months ago

I think there is no need to introduce a "proxy" component; currently the gn-ui-record-api-form is used exclusively but I think it would make sense to only use this component if the protocol is ogcFeatures, and introduce a new component called e.g. gn-ui-record-ign-api-form based on the IGN-specific protocol code; the choice between both can be done in the record-apis-component

Where would you do the switch statement (API type => corresponding form) ? in datahub-record-apis ? Was just wondering if it's not more a feature than a datahub thiing.

a dedicated ign-dl-form presentation component might not be needed; IIRC the component will mostly be comprised of several dropdown/search fields and a list of results; the smart component (e.g. gn-ui-record-ign-api-form) could hold the logic, interact with the IGN API client and place the relevant presentation components; this would avoid introducing another layer of complexity

Yes I told them to start with implementing the feature component, then perhaps we could split it with some UI components or not. Might not be needed though.

jahow commented 5 months ago

Where would you do the switch statement (API type => corresponding form) ? in datahub-record-apis ? Was just wondering if it's not more a feature than a datahub thiing.

Right, good point! then a generic gn-ui-record-ign-api-form in the feature/record lib would make sense, and the switch would be done here.

jahow commented 1 month ago

Hi @mmohadIGN @tomalary, this PR might interfere with your work there? https://github.com/geonetwork/geonetwork-ui/pull/875

We're going to merge this soon so as to make it part of the next release, let us know if there's anything that can be done to help with potential conflicts.

mmohadIGN commented 1 month ago

Hi @mmohadIGN @tomalary, this PR might interfere with your work there? #875

We're going to merge this soon so as to make it part of the next release, let us know if there's anything that can be done to help with potential conflicts.

Hi,

Thanks, it should be okay for us, but we will tell you if not

geonetwork / geonetwork-ui

[Datahub] - Add filter options on the download menu #766