Extreme timeout issues

[davwheat]

I've been noticing that, after attempting to log in to the web dashboard, the entire server enters a period of not handling any requests to it, most likely as it's doing some logic behind the scenes.

This also causes my entire raspberry pi to stop responding to any commands (SSH connections, Samba connections, other websites). This continues for about 2-5 minutes before it restores itself and all is normal again.

Thankfully, as my site is behind Cloudflare, all my content is cached so this doesn't affect usability much, but it's very frustrating if I want to upload an image/file from my phone via the web dashboard while not at my PC.

Why is this happening:

As I said, I'm guessing the server enumerates the full list of uploaded items when someone logs into the dashboard. I have over 1200 items uploaded, totalling 1.7 GB. Eek.

Example login request:

This took 1.7 minutes to complete... It only stopped because Cloudflare detected the timeout and showed me a 524 error page.

[DJkikisa]

I would say it's not normal for 1200 items, but you can fix with adding ajax to datatables .

[theaquarium]

Yeah, it's odd that this is happening with 1200 items, since I have the same amount and don't have any issues. However, performance is definitely a concern I have, and I have a couple ideas about what could be done:

1) Most likely, if you run ls -la on that directory right now, it won't take 5 minutes. This makes this problem a bit more confusing, since ls will be fetching the same information that we are. My guess is that since we use finfo_file to read file info to determine whether to draw a tooltip, that's the main bottleneck. Since finfo reads MIME types, it has to open and read the first few bytes of each file, which is much slower than just getting info like size from the file system. This can be fixed by just checking via file extension.

2) It's reasonable to suspect that reading file info might be slow itself even without finfo_file. This could be fixed by creating an index of all uploads that has some basic info about each file. This means that the page will just have to read from the index instead of scanning the directory each time. The index would be updated whenever a file is uploaded, renamed, or deleted. This means that users wouldn't be able to really change the files manually, because it would break the index. In addition to generating the index on first load, users could be given the object to regenerate the index if they have to change something (like with a button on the dashboard). Index mode would be an off-by-default configuration option.

3) Finally, there's likely a performance drawback from generating all the HTML for the data table at once and sending it over. Luckily, DataTables allows for server side processing. This means that server would be doing all the sorting and would just send over the rows to display. Using this along with index mode would greatly speed up time to first byte, as well as page load time. This would also be in a configuration option.

I think this should mostly fix all the possible performance issues and will overall streamline the uploader even for systems without extreme performance issues like this.