@Jinnkunn, I modified your script to extract monthly data from PostgreSQL and store it in SQLite. So far, I have been able to extract data up until April 2020. However, the script fails to extract data for the following months due to insufficient memory. I am wondering if there are ways to optimize the script to be more efficient or use the disk more effectively to avoid memory issues. Can you please review the script and suggest changes so that I can proceed with the data extraction required for Alexandra? There is an ongoing issue that I am currently addressing in relation to the problem.
I suggest committing data insertion every 24 hours or weekly, deleting unnecessary variables during the pipeline, and explicitly calling the garbage collector in Python. Please update me on the progress.
Go to /meridian/sqlite-databases/ there you will find the .py file for data extraction.
If you need to use AISdb, you can use "source AISdb/*/activate" on the same folder.
@Jinnkunn, I modified your script to extract monthly data from PostgreSQL and store it in SQLite. So far, I have been able to extract data up until April 2020. However, the script fails to extract data for the following months due to insufficient memory. I am wondering if there are ways to optimize the script to be more efficient or use the disk more effectively to avoid memory issues. Can you please review the script and suggest changes so that I can proceed with the data extraction required for Alexandra? There is an ongoing issue that I am currently addressing in relation to the problem.
I suggest committing data insertion every 24 hours or weekly, deleting unnecessary variables during the pipeline, and explicitly calling the garbage collector in Python. Please update me on the progress.