The current processor writes files in Data/Catalogs/FullData/
This is very dangerous in the context of a WEB service where we can have different processes running in parallel.
Having these processes writing their own files at the same time, location and names will lead to a disaster.
I propose to add a session_dir parameter to process_one_source and to change the code in order that all files written during the processing will be located in that session folder.
The session folders will be handled at API level in a way that will prevent any data overlap.
The current processor writes files in Data/Catalogs/FullData/ This is very dangerous in the context of a WEB service where we can have different processes running in parallel. Having these processes writing their own files at the same time, location and names will lead to a disaster.
I propose to add a
session_dir
parameter toprocess_one_source
and to change the code in order that all files written during the processing will be located in that session folder. The session folders will be handled at API level in a way that will prevent any data overlap.