In the existing freva system, users are able to add custom metadata to Apache Solr, which has been quite popular. To enhance this functionality in the new freva-rest API (databrowser API), we propose implementing a PUT method. This method will allow users to add their own metadata seamlessly. The new PUT endpoint will validate against the schema and integrate the metadata into Apache Solr.
Avoiding File Name Creation
The current UserData class in freva manages metadata by merging input files with user-provided metadata, resulting in a new file structure under the crawl_my_data root directory. However, this approach necessitates creating new file names, often accomplished through soft links, copying, or moving data, which introduces complications. To streamline this process for the metadata-crawler, we propose extending the PUT method to also update MongoDB, linked with the REST API. This way, the metadata-crawler can access metadata directly from MongoDB, eliminating the need for new file names.
Introducing a user Facet
Additionally, we recommend implementing a new user facet in Solr. This enhancement enables users to manage their project data independently without affecting the primary metadata. Currently, we use complex naming conventions like project=<username>. user project, which can be cumbersome. With the user facet, users can seamlessly index their project data under their own namespace within Solr. This change not only simplifies data management but also allows for easy indexing of intake-esm catalogue files. All catalogue entries can be grouped under theuser` facet, ensuring they are distinct from the main metadata collection and eliminating any risk of interference.
Improving the freva-rest API
In the existing freva system, users are able to add custom metadata to Apache Solr, which has been quite popular. To enhance this functionality in the new freva-rest API (databrowser API), we propose implementing a PUT method. This method will allow users to add their own metadata seamlessly. The new PUT endpoint will validate against the schema and integrate the metadata into Apache Solr.
Avoiding File Name Creation
The current UserData class in freva manages metadata by merging input files with user-provided metadata, resulting in a new file structure under the crawl_my_data root directory. However, this approach necessitates creating new file names, often accomplished through soft links, copying, or moving data, which introduces complications. To streamline this process for the metadata-crawler, we propose extending the PUT method to also update MongoDB, linked with the REST API. This way, the metadata-crawler can access metadata directly from MongoDB, eliminating the need for new file names.
Introducing a user Facet
Additionally, we recommend implementing a new user facet in Solr. This enhancement enables users to manage their project data independently without affecting the primary metadata. Currently, we use complex naming conventions like
project=<username>.
user project, which can be cumbersome. With the user facet, users can seamlessly index their project data under their own namespace within Solr. This change not only simplifies data management but also allows for easy indexing of intake-esm catalogue files. All catalogue entries can be grouped under theuser` facet, ensuring they are distinct from the main metadata collection and eliminating any risk of interference.