sk0rabu
commented
2 months ago Update - configured my PC to never turn screen off, and left it there overnight. The server still crashed after the said interval, and I should correct myself that instead of a mouse click I might need a key press to bring the server back.
There is an extra line output by the shell: ERROR reading output (state change): The sever problably exited.
Update 2 - my previous setup was linking the temp folder to my HDD. The transfer rate via webdav was 5MB/s and it would lead to the server crashing every 2-6h. (Upload speed from local to Mega server was anywhere between 100Mbps to 1Gbps)
Suspecting that HDD might have been the bottleneck, yesterday I moved the local cache folder to an SSD. Now the speeds via webdav had increased six-fold, but thanks to the transfer queue being fed quicker, the server has also been crashing faster. (Usually within 2h)
At the same time, since I have been dealing with files 5-20GB large in the past days, I'm struggling to get them uploaded via webdav. The transfers seem to not register and I had to repeatedly attempt to reupload those files.
As an example, running rclone check some time after a fresh sync session gave a result showing some files had been uploaded, but a while later (after the crash?) running the same command again would tell me that MORE files need to be uploaded...
Update (Last?) - it turns out that rclone would wait for Mega's response that the files were uploaded successfully, but the default timeout was very short (either 1 min or 5); and if that expires it considers it a connection / IO error and IMMEDIATELY RETRIES TO UPLOAD THE "OFFENDING" FILE. In the meanwhile megacmd might still be working on the previous transfer, if the IO throughput to the WebDAV is higher than upload speeds to Mega servers. This results in files being repeatedly queued up in megacmd, and if the transfer queue grows too large, the server gets shut down. Also if the same file was transferred twice, the copy on the Mega server would be removed, which might be why I wasn't seeing any progress if crash had happened.
Will consolidate my findings and upload my request.
Consolidated request:
httputfile.getxfer.AAA.BBB.mega)Thanks.
Original post:
Hi,
This is related to another issue that I reported earlier.
My current workflow is - run megacmd, start server & webdav, run rclone (which had webdav addr as remote), and upload files via webdav thru rclone.
This all works fine but the files being transferred all seemed to be cached locally (in Windows they are stored under
C:\Users\$USERNAME\AppData\Local\MEGAcmd\.megaCmdby default; the filenames look likehttputfile.getxfer.AAA.BBB.mega). However, megacmd won't automatically clear the cache nor offers an cmdline option to let users do so, resulting in an ever-enlarging temp folder (hence my workaround to link the local temp folder to a different drive).Which brings the other problem I mentioned - the megacmd server will automatically shut itself down, bringing down the webdav server as well. I ran the rclone jobs in the background and locked the desktop, and usually hours (3 - 6) later they would start complaining that "the connections were refused". And if I switched focus onto the megacmd window it will report that "MEGAcmd Server not running. Initiating in the background..." and moments later the server would come back up and the file transfer would resume.
I think a reply to another similar issue mentioned that the server was shut down as the growing number of transfers caused megacmd's memory to run out (?) but I'm wondering if anything can be done to address the growing cache problem and in turn server auto shutdown problem. Is it possible to have the server auto recover itself so that I won't have to click on the megacmd shell window to trigger its restart?
Thanks.