The objective of this task is to implement a mechanism that allows users to query data sources and schedule new transformation jobs based on those queries.
Features
Select Sources: Users should be able to create and define queries to select data sources according to various criteria, facilitating targeted data processing.
Schedule New Jobs: The sources identified through the queries can be utilized to initiate new transformation jobs, ensuring efficient handling of relevant data.
Prevent Concurrent Processing: Implement locking mechanisms during transformations to prevent concurrent processing and race conditions. Locks should be released once the transformation is complete or has failed, allowing for further actions.
Metadata Requirements
Ensure that the operational metadata has sufficient granularity and that the queries are expressive enough to support Data Operations effectively.
User Interface Proposal
Propose an interface or API that enables users to edit and inspect the results of the queries for scheduling new jobs. This interaction should be user-friendly and accessible through a UI or a Jupyter notebook, designed for users with basic Python knowledge.
Deliverables
Examples of Selection Queries: Provide examples of queries that can be used to select notices and batches.
Demonstrator: Create a demonstrator that showcases a common operation, illustrating how to select and schedule sources for transformation.
Constraints
During transformations, ensure that notices are locked to prevent concurrent processing and race conditions.
Clean up any obsolete or unused components in the codebase.
Description
The objective of this task is to implement a mechanism that allows users to query data sources and schedule new transformation jobs based on those queries.
Features
Metadata Requirements
Ensure that the operational metadata has sufficient granularity and that the queries are expressive enough to support Data Operations effectively.
User Interface Proposal
Propose an interface or API that enables users to edit and inspect the results of the queries for scheduling new jobs. This interaction should be user-friendly and accessible through a UI or a Jupyter notebook, designed for users with basic Python knowledge.
Deliverables
Examples of Selection Queries: Provide examples of queries that can be used to select notices and batches.
Demonstrator: Create a demonstrator that showcases a common operation, illustrating how to select and schedule sources for transformation.
Constraints