Integrate TEDMON as a new data source in the pipeline to enable periodic scheduling of its package transformations. The implementation should support scheduling transformations for multiple packages within a folder or bucket, with the ability to select specific packages based on defined patterns.
Features
Source Configuration: Define a file pattern to filter which packages to consume from a folder or S3. By default, all packages are included.
Package Handling: For each package, extract the XML notices and schedule a new batch job to process all notices.
Description
Integrate TEDMON as a new data source in the pipeline to enable periodic scheduling of its package transformations. The implementation should support scheduling transformations for multiple packages within a folder or bucket, with the ability to select specific packages based on defined patterns.
Features