search

sci-ndp / scidx-api

sciDX API
Apache License 2.0
1 stars 2 forks source link

[FEATURE] Add data mapping and processing info #47

Closed Andreufb closed 1 month ago

Andreufb commented 1 month ago

Description

To enable efficient filtering and processing of data, we need comprehensive mapping information of the desired data. This includes insights into the data format and structure, allowing us to extract the required data in a standardized manner. Additionally, this information will facilitate the application of user-defined filters and abstractions.

Steps to Reproduce

  1. Update the KafkaDataSourceRequest model to include mapping and processing fields.
  2. Modify the create_kafka_datasource endpoint to handle the new fields.
  3. Update the add_kafka service function to process and store the mapping and processing information.
  4. Follow similar steps for the URL resource:
    • Add file_type, mapping, and processing fields to the URLRequest model.
    • Update the post_url endpoint.
    • Modify the add_url service function to handle and store the new fields.

Expected Behavior

We expect to implement a feature that provides comprehensive mapping information for the data. This feature should offer insights into the data format and structure, facilitating the extraction of the desired data in a standardized manner. It should also support the application of user-defined filters and abstractions to the data.

Actual Behavior

(Not applicable for this feature request.)

Possible Solution

Implement a mapping system that includes detailed information about the data format and structure. This system should be flexible enough to support various data types (e.g., stream, CSV, TXT, JSON, NetCDF) and allow for user-defined filters and abstractions.

Additional Context

This feature will standardize data extraction and processing, making it easier to apply filters and abstractions across different data sources.

rbardaji commented 1 month ago

Not sure what you want to do. Do you want to modify the POST /kafka endpoint? To do what exactly? @Andreufb (or @saleemalharir1, or @irodero), can you clarify this issue? Thanks.

Andreufb commented 1 month ago

yes, as we discussed monday morning I am updating the POST of kafka and URL to have mapping and processing fields so later on we are able to apply filtering to the data. I am also updating the Search to correctly show this new fields.