Multi processing implementation works with DAG cvt-pipeline-small_multiprocessing
Part of the batch processing implementation is in the comments of retrieve_images.py
TODO for later
Postprocessing doesn't work, need to make implementation for glob on coco_instancesresults*.json files
Fix when you want to restart DAG. Idea by Thomas:
file_names = read_csv(path_to_csv)
try:
for file_name in file_names:
process(file_name)
except Exception as e:
# Clean up processed files to prepare for restart
for file_name in file_names:
# Do something to get the path to the processed file
processed_file_name = filename += "processed"
os.rm(processed_file_name)
# Re-raise error to log trace back, and bring task to failed state
raise e
Tested and works tot postprocessing POD
TODO for later