Upload refused because of maintenance-mode (but server is not in maintenance-mode)

[bruennlein]

Actual behaviour Android app refuses to auto-upload pictures and videos and says the server is in maintenance-mode. Same behavior, when trying to upload manually from the app. Uploading via browser works from the same device. Tried to turn maintenance-mode on and off again, no effect. For some reason, 6 photos of about 6-8MB and two videos with about 330MB went through. But the most files are stuck. Using the retry-button from inside the app (menu->uploads) has no effect.

Steps to reproduce Enable auto-upload and take some photos and videos

Environment data Android version: 9

Device model: Nokia 8

Stock or customized system: Stock

Nextcloud app version: 3.6.0 and 3.6.1

Nextcloud server version: 15.0.7

[bruennlein]

I´ve just updated the app to Version 3.6.1. After the update, the upload of 10 Videos worked just fine (biggest file: 700MB). Now, all files are stuck again with the same message. App restart (forced app-restart using the system-settings menu ) solved the problem for now (this did not help in 3.6.0). This "Restart-Cheat" doesn´t work every time.

[bruennlein]

Problem still exists, but some newer photos got uploaded. Can't say, why it works for some files.

[augustseptember]

Same problem here

[bruennlein]

I´ve got some news. It seems like this happens to pictures in autoupload, where the first try was in "maintenance-mode". They seem to be stuck forever. It looks like, they are "tagged". When I try manual uploads or when I take photos now, the upload seems to work just fine (I´ve made an update after I had this problem from 3.6.0 to 3.6.1).

[MrSmoke]

I have the same thing when trying to upload larger files manually (I have 3 failed, which are all over 400MB)

[snoord]

Confirming that I am also having this issue.

Sometimes the files upload fine, but certain photos and videos will continually fail with the "Server in maintenance mode" message.

Phone = Samsung Galaxy S9+ (SM-G965F) Android version = 9 Kernel version = 4.9.59-15540155 Nextcloud app version = 3.7.0 RC3 Nextcloud server version = 16.0.1

My Nextcloud server instance is deployed as a Docker container in Swarm mode. The image used is the "linuxserver/nextcloud" image. It is accessible via the Traefik reverse proxy (also containerised and in the same Swarm).

[bruennlein]

Hi Guys,

the auto-upload works for new pictures since a while now. But pictures, which got stuck in the past, are still not uploaded. On new pictures, the auto-upload is going through all pictures in the queue. Pictures, which got stuck in the past, still showing the mainenance-mode error. When the queue reaches the new pictures, the upload works just fine for them.

[tobiasKaminsky]

Can you click on them manually to retry? They should be re-tested , so that the old status gets invalitated

[bruennlein]

Hello Tobias,

klicking them lead to the same problem. But I can´t reproduce this at the moment, because the elements disapeard from the upload list. I can´t say if they have disapeared because of a device-restart or if they finally got uploaded. At the moment the auto-upload works very reliable. I´ll see, if I can find out, if they got uploaded. I have to search them by date and compare the files on my device with the files on the server. This could take a while...

Cheers...

[Samathy]

I´ve got some news. It seems like this happens to pictures in autoupload, where the first try was in "maintenance-mode". They seem to be stuck forever. It looks like, they are "tagged". When I try manual uploads or when I take photos now, the upload seems to work just fine (I´ve made an update after I had this problem from 3.6.0 to 3.6.1).

I'm also seeing this exact behavior - photos which failed to upload once keep failing, but new photos are uploaded immediately with no problems. Clicking on them retries, but they fail again. For me, the first attempt to upload these files failed because the server genuinely was in maintenance mode, intentionally. Uploading the files manually works fine.

I'm running versions later than @bruennlein, so it would appear that this issue has not been addressed and persists.

Nextcloud Version: "15.0.10" Android App Version: "3.7.0" Stock or customized: Stock snap image, Debian 9. Android Version: 9

[stale[bot]]

This request did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[darmbrust]

This should be reopened, it still constantly happens for me as well. The server logs errors about streams being null, when this happens.

[darmbrust]

[no app in context] Error: Sabre\DAV\Exception\ServiceUnavailable: Could not open file at <<closure>>

 0. /opt/storage/website/nextcloud/apps/dav/lib/Upload/AssemblyStream.php line 292
    OCA\DAV\Connector\Sabre\File->get()
 1. /opt/storage/website/nextcloud/apps/dav/lib/Upload/AssemblyStream.php line 142
    OCA\DAV\Upload\AssemblyStream->getStream(OCA\DAV\Connector\Sabre\File {})
 2. <<closure>>
    OCA\DAV\Upload\AssemblyStream->stream_read(8192)
 3. /opt/storage/website/nextcloud/3rdparty/icewind/streams/src/Wrapper.php line 91
    undefinedundefinedfread(null, 8192)
 4. /opt/storage/website/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php line 98
    Icewind\Streams\Wrapper->stream_read(8192)
 5. <<closure>>
    Icewind\Streams\CallbackWrapper->stream_read(8192)
 6. /opt/storage/website/nextcloud/lib/private/Files/Storage/Local.php line 488
    undefinedundefinedfile_put_contents("/mnt/nextCloudS ... t", null)
 7. /opt/storage/website/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 630
    OC\Files\Storage\Local->writeStream("files/VID_20190 ... t", null, null)
 8. /opt/storage/website/nextcloud/apps/dav/lib/Connector/Sabre/File.php line 191
    OC\Files\Storage\Wrapper\Wrapper->writeStream("files/VID_20190 ... t", null)
 9. /opt/storage/website/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php line 156
    OCA\DAV\Connector\Sabre\File->put(null)
10. /opt/storage/website/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php line 316
    OCA\DAV\Connector\Sabre\Directory->createFile("VID_20190922_174412.mp4", null)
11. /opt/storage/website/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php line 130
    Sabre\DAV\Tree->copyNode(OCA\DAV\Upload\FutureFile {}, OCA\DAV\Files\FilesHome {}, "VID_20190922_174412.mp4")
12. /opt/storage/website/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php line 161
    Sabre\DAV\Tree->copy("uploads/dans_ph ... e", "files/dans_phon ... 4")
13. /opt/storage/website/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 642
    Sabre\DAV\Tree->move("uploads/dans_ph ... e", "files/dans_phon ... 4")
14. <<closure>>
    Sabre\DAV\CorePlugin->httpMove(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
15. /opt/storage/website/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
    undefinedundefinedcall_user_func_array([Sabre\DAV\CorePlugin {},"httpMove"], [Sabre\HTTP\Requ ... }])
16. /opt/storage/website/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 479
    Sabre\Event\EventEmitter->emit("method:MOVE", [Sabre\HTTP\Requ ... }])
17. /opt/storage/website/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 254
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
18. /opt/storage/website/nextcloud/apps/dav/lib/Server.php line 316
    Sabre\DAV\Server->exec()
19. /opt/storage/website/nextcloud/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
20. /opt/storage/website/nextcloud/remote.php line 163
    undefinedundefinedrequire_once("/opt/storage/we ... p")

MOVE /nextcloud/remote.php/dav/uploads/phone/81d7c0aaeb56f41b205482db7811533c/.file
from 192.168.10.1 by phone at 2019-09-24T16:10:13+00:00

When I look in the folder store, this folder exists: /nextcloud/remote.php/dav/uploads/phone/81d7c0aaeb56f41b205482db7811533c/

However, the sub file (or folder?) named .file it is looking for does not exist.

[darmbrust]

For me, it also fails again even if I try to manually upload the failed file.

[darmbrust]

Usually, after a number of days, this will randomly start working again for the file in question, but there is no rhyme or reason as to when or why. It seems that the unique file key is being kept by the server somewhere, and it has it locked, thinking there is an upload or operation in process - even though the upload failed... and then refuses all future operations on that file until the lock is released. But I have no idea how or what releases the lock... it may be that the lock doesn't go away until my phone stops trying to upload the file for a certain amount of time, like when I am off of wifi...

[darmbrust]

There is a bug filed on the server over here...

https://github.com/nextcloud/server/issues/16383

[AndyScherzinger]

also looping in @nextcloud/server-triage for help

[darmbrust]

my wild guess about file locks doesn't seem to be the root cause, I've played around with manually deleting locks from the occ_file_locks table, and it still fails in the same way.

[stale[bot]]

This request did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[darmbrust]

no auto close....

[darmbrust]

FYI, the thing that I have found which will fix the issue for particular stuck files for me - my autoupload is set to only upload when on wifi. So, if I shut my wifi off for about 12 hours, and then re-enable it, the files that were previously stuck in the lock / broken loop then properly upload and work fine.

Obviously, the server is holding some sort of lock on files during an upload, and it isn't properly clearing the lock when an upload gets interrupted, by say, going out of range on wifi.

[burner-]

I have 17.0.1 server, 3.9.0 android app, nokia 8 phone and I got same error with 464MB file. Also I notice that all windows clients give now error of same file that "The downloaded file is empty despite that the server announced it should ha..." and it give popup each time when I save file to nextcloud folder. So it looks that server somehow failed that file sync and now it does not allow retry that upload and downloading clients also give error again and again.

[jomo]

Can confirm I also ran into the issue @burner- had, a few files are empty despite the server claiming a different file size.

The Android app also sometimes fails to upload a file with "Locked" instead of "Server in maintenance mode"

[stale[bot]]

This request did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[darmbrust]

Hey stale bot, go away. This isn't fixed.

[AndyScherzinger]

@darmbrust stale != fixed. The couldn't be reproduced and no further info on how it could be reproduced has ever been provided so it is yet "unfixable" thus the stale bot closing it. (and this loop will keeping repeating until stale-based close is accepted or necessary info is being provided).

[darmbrust]

You are joking, right? Obviously, if I don't kick the stale bot, this will be closed, and all useful info is lost. No additional info is required, it is pretty trivial to reproduce this problem. Its reported by multiple users.

I know that the reason that it hasn't been fixed is because it hasn't yet found a developer with the time and skill to actually look at the code, and figure out where the lock is.

I'm not complaining, but its pretty laughable that the issue management paradigm seems to be if we ignore them, they must not be real.....

[AndyScherzinger]

You are joking, right? Obviously, if I don't kick the stale bot, this will be closed, and all useful info is lost.

Nope, the issue will just be closed, but the info is still there

No additional info is required, it is pretty trivial to reproduce this problem. Its reported by multiple users.

Wrong, It works just fine for me, @tobiasKaminsky @ezaquarii, etc. So just because people report this issue (for them happening) it doesn't mean it just happens for anybody else.

I know that the reason that it hasn't been fixed is because it hasn't yet found a developer with the time and skill to actually look at the code, and figure out where the lock is.

Only right with regards to time. @tobiasKaminsky rewrote the logic as in moving to chunked uploads. Maintenance mode is an information we should get from the server and just display as such. So yes something seems to go wrong here, hence the reports. Still, reproduction means that you need to know the steps and the steps can't be "wait until this randomly happens".

I'm not complaining, but its pretty laughable that the issue management paradigm seems to be if we ignore them, they must not be real.....

Again, wrong, the paradigm is if it can't be reproduced or nobody is willing to invest as much time as it might be necessary to find the cause (which might still point to too little information) than the issue just won't be resolved. That doesn't imply the issue isn't real but is rather a automated (via bot) decision to not put in the effort to fix it. While this might be frustrating for users being hit by this issue it is also not a realistic scenario to think any bugs any software have will be resolved over time.

[tobiasKaminsky]

@darmbrust Can you create us a test account, test if the problem occurs also there and if so send the credentials to tobias at nextcloud dot com with a reference to this issue?

[darmbrust]

a test account on my server? I could, if you think it would help.

How recently were the changes made WRT chunked uploads mentioned above?

If you have time to look into this, I should re-verify that I have the latest version of everything, and reproduce it again before you spend time on it....

Though in the past, it didn't take anything more to reproduce it than trying to upload a large file, killing the wifi half way through the transfer (assuming configured for wifi only uploads), and then letting it try to resume when the wifi returned.

[bruennlein]

I have the problem again. I'm currently in an environment, where I often change the network (Mobile data, Hotel-WiFi, OpenVPN), but currently there is nothing more could say. I'll try to find out more...

@developers: is there anything special, I cluld check?

[petitminion]

same problem here. Nextcloud is in a lxc container, data is in a virtual drive mount into the container.

[aquatic7]

This problem is 2 years old and no fix has been provided or found yet. There are several open topics on this, but none provide any solution. I've had this issue on the same server for 2 years now on my snap installation. Happened after an upgrade. Same issue on all android devices trying to autoupload pictures and videos. It is so frustrating that it ain't working. So in addition I am now using Mega.nz which kinda defeats having a Nextcloud other than to sync contacts and calendar (which works perfectly).

[aymercury]

Same happens here. Nextcloud 18.0.1 (docker), android app 3.10.1. Auto upload of camera photos fails, server in maint mode, which is a lie. Manual upload of same problem files fails in the same way.

[mattg66]

Same issue here using external SMB storage with global credentials. Auto upload fails as server is in maintenance mode, when in fact it is not and file uploads work via web browser. Manual upload also fails via the app as others report.

[tobiasKaminsky]

@nextcloud/server-triage do you have an idea? Android client is just trying to use webdav to upload it. It shows "maintenance" info if we receive a 503.

[rullzer]

@nextcloud/server-triage do you have an idea? Android client is just trying to use webdav to upload it. It shows "maintenance" info if we receive a 503.

503 can happen in other cases as well. For example the backend storage is not available etc.

You should always check status.php to see if the maintenance mode is activated.

[74954]

This issue is not related to whether the site is actually in maintenance mode. Nextcloud via the browser works without issue, just in the app, it always reports maintenance when using external SMB storage

[jomo]

I tried debugging this issue.

What I found is that the PROPFIND response returns hundreds of chunk files with a 200 OK status (although 404 Not Found for quota), but the files don't exist on the filesystem.

PROPFIND response

```xml /remote.php/dav/uploads/user/937f626f2f410ab8cc0aa475954010ca/ Fri, 17 Jan 2020 11:05:18 GMT HTTP/1.1 200 OK /remote.php/dav/uploads/user/937f626f2f410ab8cc0aa475954010ca/0000000000000000-0000000010239999 Fri, 17 Jan 2020 10:25:45 GMT 10240000 "f7e243ef0ac5d9fddb1c402b19c10316" application/octet-stream HTTP/1.1 200 OK HTTP/1.1 404 Not Found /remote.php/dav/uploads/user/937f626f2f410ab8cc0aa475954010ca/0000003246080000-0000003251808419 Fri, 17 Jan 2020 11:05:18 GMT 5728419 "6ea0e66c7afab52c37fe60526479d88b" application/octet-stream HTTP/1.1 200 OK HTTP/1.1 404 Not Found /remote.php/dav/uploads/user/937f626f2f410ab8cc0aa475954010ca/.file Fri, 17 Jan 2020 11:05:18 GMT 3251808419 "cac50c40cd233" application/octet-stream HTTP/1.1 200 OK HTTP/1.1 404 Not Found ```

In this case, the directory 937f626f2f410ab8cc0aa475954010ca/ existed and was created on Jan 17, but it contained no files when I ran ls -la.

The following MOVE request for 937f626f2f410ab8cc0aa475954010ca/.file then failed with:

fopen(/path/to/nextcloud_data/user/uploads/937f626f2f410ab8cc0aa475954010ca/0000000000000000-0000000010239999):
failed to open stream: No such file or directory at /var/www/nextcloud/lib/private/Files/Storage/Local.php#302

---

I assume this error results in a 503 response (which is correct), but the Android app incorrectly interprets it as "Maintenance mode".

[jomo]

I'd like to add that the nextcloud_data directoy in my case is mounted on the local filesystem and the nextcloud user can properly read and write files in the uploads directory and sub-directories.

So this doesn't seem to be related to external SMB storage.

[tobiasKaminsky]

What I found is that the PROPFIND response returns hundreds of chunk files with a 200 OK status (although 404 Not Found for quota), but the files don't exist on the filesystem.

Thanks for debugging! @rullzer this seems to show outdated chunk information, but the files were deleted meanwhile (maybe because of auto cleanup?) But why are then the infos via dav still available?

[rullzer]

No clue. The file should be there then... Autocleanup also kills them fromt he filecache.

[tobiasKaminsky]

@jomo is this working after auto-cleanup gets triggered? I think it is after 24h of no write into specific chunk folder

[darmbrust]

In my experience, the problem goes away if I disable wifi for a period of time (which disables the upload attempts in my config) - so something that happens after a timeout period clears the issue. It very well could be the auto-cleanup...

I suspect the auto-cleanup doesn't happen for the file in question if you don't disable upload, because it keeps trying every few minutes.

I have tried manually deleting orphaned chunks of files I've found on the server file store before, to see if that helps, but it doesn't seem to (I'm sure, because I didn't clean up the in-memory state of knowing about those chunks by manually deleting them)

[tobiasKaminsky]

I have no clues :-( If there are really leftovers from chunk upload, I fear that I cannot do anything from Android side :/

[stale[bot]]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[jomo]

bump

[Queuecumber]

I also just got hit with this after it working fine for a while, is there a workaround?

[darmbrust]

For me, I have mine set to only upload on wifi. So, if I turn off wifi for 12+ hours, the problem tends to clear, since it doesn't try to upload then. Whatever is holding the lock must have a timeout. After the lock is gone, the problem file uploads fine.

I haven't tested enough to be sure of the exact amount of time required to clear the lock.

[Queuecumber]

I wasn't able to upload any file to any folder under any network conditions though. Luckily I'm still in the testing phase of this deployment so I nuked the server which I assume has to fix it