Summary
If the data loader python script fails to an exception, it will simply hang in the console and need to be force killed. This is not good for GitHub actions usage as the script will sit there for 6 hours until GitHub times out the workflow. This is partially due to the supabase client not destructing gracefully, and also due to a lack of exception handling throughout the code.
The objective of this ticket is to address the problem in the short and long terms
Short term solution
[x] Make the github action workflow timeout shorter in order to protect against script runaway
Long term solution
[ ] Add exception handling to joblib parallelization
[ ] Add exception handling to any third party API calls
[x] Figure out how to destruct the supabase API client correctly
39 fixed the issues where supabase would hang due to polling for the JWT refresh token, and shortened the job timeouts to 30 mins. However, we still need to add better exception handling to make the scripts more robust, so leaving this open for now. These robustness changes are not MVP though
Summary If the data loader python script fails to an exception, it will simply hang in the console and need to be force killed. This is not good for GitHub actions usage as the script will sit there for 6 hours until GitHub times out the workflow. This is partially due to the supabase client not destructing gracefully, and also due to a lack of exception handling throughout the code.
The objective of this ticket is to address the problem in the short and long terms
Short term solution
Long term solution