search

ODM2 / ODM2DataSharingPortal

A Python-Django web application enabling users to upload, share, and display data from their environmental monitoring sites via the app's ODM2 database. Data can either be automatically streamed from Internet of Things (IoT) devices, manually uploaded via CSV files, or manually entered into forms.
BSD 3-Clause "New" or "Revised" License
31 stars 8 forks source link

Upload of small historical csv times out #696

Open SRGDamia1 opened 10 months ago

SRGDamia1 commented 10 months ago

Attempting to upload a CSV with approximately 4300 lines of data leads to a time-out after 5 minutes. The data in the file is from 8/18/23 to 9/2/23 at SHVC2S. Everything in the CSV appears to be correctly formatted and I'm getting no errors other than the timeout. Is this a problem with the CSV upload itself or is it related to compressing and decompressing chunks to insert historical data?

tpwrules commented 10 months ago

This is likely due to database access inefficiencies. I was able to make this over 90 times faster on my setup using the code here. I don't recall if that's a complete fix that 100% preserves site functionality however, but I believe that commit is for this specific problem.

aufdenkampe commented 1 month ago

@SRGDamia1 and @ptomasula, this might become a quick fix once we complete our testing of: