ErwanQuintin / STONKS

1 stars 0 forks source link

Working enclosed in session #4

Open lmichel opened 1 year ago

lmichel commented 1 year ago

The current processor writes files in Data/Catalogs/FullData/ This is very dangerous in the context of a WEB service where we can have different processes running in parallel. Having these processes writing their own files at the same time, location and names will lead to a disaster.

I propose to add a session_dir parameter to process_one_source and to change the code in order that all files written during the processing will be located in that session folder. The session folders will be handled at API level in a way that will prevent any data overlap.

lmichel commented 1 year ago

This issue is blocking for the service deployment

lmichel commented 1 year ago

fixed