LRO Instance Testing - Backend

[daniel-slaugh]

I did some load testing on the LRO instance and found the following bugs:

• [x] Going to the SiteDetails page of Mendon Road as a guest, the photos endpoint returns a 403 forbidden
• [x] Database queries are still taking 60+ seconds on large datasets (200,000 obs+) when successful, but more often than not, a dataset will churn for a minute then return a 502 bad gateway or 504 Gateway Timeout. After making a couple large queries, no api endpoints can be reaches for a couple minutes. I'm guessing the server gets overwhelmed and needs to clear out.
• [x] Copying state as URL saves without the base URL (https://lro.hydroserver2.org), but it works locally. That means the VITE_APP_PROXY_BASE_URL is empty in production?
• [x] This dataset has duplicate values for every time point: https://lro.hydroserver2.org/time-series-analyst/?datastreams=98c599df-34b7-43eb-aba7-67ef381a3430&PLs=Quality+controlled+data&OPs=Blue-green+algae+%28cyanobacteria%29%2C+phycocyanin&selectedDateBtnId=0&dataZoomStart=62.46651056583777&dataZoomEnd=62.49671058690937

[kjlippold]

I've been digging into the backend issues trying to optimize the DB queries. Currently I've got queries of one full datastream with 360,000 values taking 4 seconds, and it seems to be handling several concurrent queries without slowing down, other than the expected network bottlenecks and client memory limitations. Querying through the API is still taking a long time (20-60 seconds), so the bottleneck now seems to be somewhere in the application server.

I was seeing some out-of-memory errors associated with concurrent requests so I increased the EC2 instance from t3.micro to t3.small and that's helping with the out-of-memory errors. I'm still seeing 504 errors sometimes, so I'm going to continue doing some testing to try to identify what's still causing those.

We should probably give an option to display one or two years of data, and require the user to manually expand the date range beyond that instead of having an "All Time" button. Most clients will choke on multiple "All Time" datastreams even if we manage to speed up the requests, and we're at a point where most solutions involve spending more money on better servers rather than code optimizations. We don't want our client apps to fetch more data than the user needs, but right now I think most users will just click "All Time" if they need more than one month of data instead of entering a custom range.

[kjlippold]

PROXY_BASE_URL and photo issues were caused by a deployment bug for the LRO site. Both have been resolved.

Duplicate values issue has been resolved. https://github.com/hydroserver2/hydroserver-webapp-back/commit/2556cab76445c7f7406b1115c416625f2a72c3fa