Long TTFB causing the website to respond slowly

[LeoHerrmann]

I spent some time testing this application on my local machine and noticed it to be still loading very slowly. Therefore, the website's slow performance is unlikely to be caused by large file sizes or a bad connection. Testing using lighthouse exposes the website's Time-to-First-Byte (TTFB) to be the largset bottleneck. Long TTFB times are often caused by a script on the server taking much time to execute and prevent the client from starting to render the page.

To inspect the issue further, I edited /processdata/views.py and commented out every line including getdata, plots and maps and replaced the return values with placeholder values. Running the lighthouse tests again shows TTFB times to be drastically reduced. So my assumption is that the server needs to wait until all of the data has been fetched before sending a response.

A possible solution would be to make the index function of views.py only render an empty template and have separate URLs on the server for the data and fetch the data from the client using ajax requests. This way, the client no longer needs to wait for the server to fetch all the data and can render the website and fetch the data at the same time. The disadvantage of this method is that processing load and data usage might be slightly increased on the client.

[BrianRuizy]

@Rahmsauce, thank you for your very elaborate issue submission!

A possible solution would be to make the index function of views.py only render an empty template and have separate URLs on the server for the data and fetch the data from the client using ajax requests.

I agree, and it was something I considered had this project gained more traction, as it did. Thus, efficiency is much more crucial. Do you have an idea on how to implement that? I believe another approach could be to import the csv's into Django database models. We would have to see what would be most cost effective? Or what could be adopted quicker.

Do you wish to be co-assigned to this?

edit: Other ideas that come to mind are...

• working with cache
• storing daily reports (CSVs) on AWS S3 (free-tier, 5gb)

[LeoHerrmann]

I think I could implement a more client-based solution. You can co-assign me if you want me to give it a try.

Caching can result in major performance gains. However, it of course doesn't help people loading the website for the first time. Also, when the cached data is outdated, it needs to be fetched again anyways. I'd consider caching when the website's is already running fast to speed it up for people accessing it very frequently.

I'm not quite sure if I understand the Django database and daily reports methods correctly. How are they going to benefit performance? Wouldn't these solutions still require the client to wait until the server has fetched all the data?

[BrianRuizy]

@Rahmsauce, Go for it! The more client-based approach.

[BrianRuizy]

Successful review deployment! And, significantly improved load time.

Solves issue: #47