Add batch support - Githubissues

TODO:

create pipelines/batch/create_batch.py, modelled after scripts/create_batch.py

[ ] script should expect a base path argument (i.e., the timestream/jobs/ part)
[ ] script should optionally accept a date argument as well but default to current date.time (YYYYMMDD.HH0000)
[ ] the script should loop through the S3 bucket under the provided path arguments to find all available files (different API than os.listdir())
[ ] The script should submit one batch job per dataset (the folder underneath date.time)
[ ] The script should error if len(files) in the dataset path >100 and should log that it failed
[ ] The script should log when the batch job is created

The script will be run on a cron job

After this script is done, we can work on integrating the ingest logic to get it to run (hooking it up with the runner.py script / PipelineRegistry)

a2edap / ingest-timestream