WebUI infinite timeout issues with servers with lots of files

[StrangeOne101]

From a lot of heavy, heavy debugging, I've finally found out why and where exactly the WebUI is killing itself on my servers. You can start the web UI, login to the panel, and it never finishes loading. The page will then never again load even when the process is still running. After a lot of debugging and going through the code, I've found that the webUI tries to cache all server files. So this is what kills it!

Problem

When the server_files are called to be indexed on a server with lots of files, it takes forever to finish and stops the entire WebUI. On my servers, I have dynmap installed. Which means I have literally hundreds of thousands of png files in my server, and it is going through and filtering every single one of them to try find .jar and .phar files. The indexing never finishes. Or at least not in any reasonable amount of time, and even if it did, it is going to continue to get longer as my server grows.

Solution

I assume the problem lies around this point here: https://github.com/hexparrot/mineos-node/blob/3d62232c63d62925f342370af2b02d0bee6b0a5c/mineos.js#L1472 The solution would be to limit the amount of subdirectories it can search. In reality, I think it would only need to search the plugins folder or the bedrock server equivalent for files. Anything else isn't needed for this task.

If there is a need to search more than that, then maybe set a limit on the amount of subdirectories to search with a configurable value or something.

Note: I am not a node expert by any means on this. So while I don't know how you would implement this in node, the concept in itself is feasible in my opinion.

Thanks for reading. :)

[miiichael]

I was assuming all the filesystem walking was due to wanting to work out diskspace usage. Which of course you can't do unless you actually walk the entire server directory. :P

I do also notice that it walks /proc/ repeatedly (more than once per second).

[hexparrot]

The solution proposed is supposed to be the solution in-place, but I can't seem to get the module to work as intended:

https://github.com/hexparrot/mineos-node/commit/d7806f34165f73bc128ac071cd86879d04b231dd

726 and 735 demonstrate the explicitly ignoring of such directories, but unsure why it's ignored.

I don't think its 1472, but that's because I thought 'readdir' isn't recursive and that the profile directories (different from server directories) shouldn't have the hundreds of thousands of files within. This one has eluded me, largely because I still never work with dynmap in any degree that has ever created an appreciable amount of wait time.

[spgill]

I get this timeout behavior all the time and it makes my server's CPU usage spike up to almost 100% for ~15 seconds at a time (which is probably partly due to my OS's file system structure). It happens whenever I refresh the web UI or whenever I change some options. (for reference, I host two servers with dynmap so I too have a sh*t ton of files in my server directories)

I like the idea of limiting sub-directory depth, but could we maybe get some option to just disable file tree traversal for mineos? I'm not sure what the reasons are for finding the disk usage, but if it's solely for displaying the "Space used by live server files:" on the server page, I frankly don't need it. 😅

[hexparrot]

@spgill If you can find a way for disabling file tree traversal while maintaining new file detection, I will happily implement that. "Space used by live server files" isn't the culprit, usually.

I haven't been able to figure out how to do new file detection (and detection of files deleted and recreated) with any other tool than fireworm (the npm module).

[miiichael]

One workaround could be to simply tell the filesystem walker to not follow symlinks. Then admins can just move dynmap tile directories (the largest contributor to file count) out from mineos's prying eyes. Converting dynmap to use sqlite (or other SQL storage) for map tiles is currently the only viable alternative.

[nazzyz]

I'll just toss in a comment about this as well. Probably won't help contribute to a resolution but I'll just make it as a note. I put together a modpack that included Millenaire, https://www.curseforge.com/minecraft/mc-mods/millenaire The mod itself installs hundreds of txt and png files into .minecraft\mods\millenaire folder. I have never had a problem running any servers on my mineos install until I put together this server pack. Spent about a week troubleshooting what in the world was making the WebUI freeze up. I figured it may have been that I was on FreeNAS 11.2, so I upgraded the server to 11.3 latest. Still had the problem when I loaded the modpack. I wiped the whole jail and started a fresh install of mineos and once again loaded up only, the single modpack. Still had the problem. Lastly, I thought maybe it was hardware related so I added more RAM to the system. Currently running only the single server modpack which is fine for about a day until the UI freezes up again. My only solution to getting it back is to copy the var/games/minecraft/servers/servername to another location. Remove the server folder from /servers, restart the jail, then I can log back in again. I have to go through the process of re-importing the server folder back in again then rinse and repeat just about everyday. Don't know what information I can give but would be most obliged to help out if needed.