search

FREVA-CLINT / freva-nextgen

The next generation Freva
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Create PUT methods for the databrowser API #32

Open antarcticrainforest opened 4 months ago

antarcticrainforest commented 4 months ago

Improving the freva-rest API

In the existing freva system, users are able to add custom metadata to Apache Solr, which has been quite popular. To enhance this functionality in the new freva-rest API (databrowser API), we propose implementing a PUT method. This method will allow users to add their own metadata seamlessly. The new PUT endpoint will validate against the schema and integrate the metadata into Apache Solr.

Avoiding File Name Creation

The current UserData class in freva manages metadata by merging input files with user-provided metadata, resulting in a new file structure under the crawl_my_data root directory. However, this approach necessitates creating new file names, often accomplished through soft links, copying, or moving data, which introduces complications. To streamline this process for the metadata-crawler, we propose extending the PUT method to also update MongoDB, linked with the REST API. This way, the metadata-crawler can access metadata directly from MongoDB, eliminating the need for new file names.

Introducing a user Facet

Additionally, we recommend implementing a new user facet in Solr. This enhancement enables users to manage their project data independently without affecting the primary metadata. Currently, we use complex naming conventions like project=<username>. user project, which can be cumbersome. With the user facet, users can seamlessly index their project data under their own namespace within Solr. This change not only simplifies data management but also allows for easy indexing of intake-esm catalogue files. All catalogue entries can be grouped under theuser` facet, ensuring they are distinct from the main metadata collection and eliminating any risk of interference.

antarcticrainforest commented 4 months ago

I have created a PR https://github.com/FREVA-CLINT/freva-service-config/pull/16