search

cloud-py-api / mediadc

Nextcloud Media Duplicate Collector application
https://apps.nextcloud.com/apps/mediadc
GNU Affero General Public License v3.0
93 stars 6 forks source link

Specific job setting does result in 503 Service Unavailable error ("Storage is temporarily not available") #187

Open rugk opened 10 months ago

rugk commented 10 months ago

Describe the bug I have run some jobs successfully, but a specific job (with maybe a lot of files), cannot be started?

To Reproduce I have no idea. I can reproduce it when I select two specific directories. Other settings do not seem to matter.

And the server response I can see in the devtools comes very fast, it does not seem as if it is indexing something or so. It took 853 ms.

The following request is executed (copied as fetch):

await fetch("https://*****/apps/mediadc/api/v1/tasks/run", {
    "credentials": "include",
    "headers": {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0",
        "Accept": "application/json, text/plain, */*",
        "Accept-Language": "de,en-US;q=0.7,en;q=0.3",
        "Content-Type": "application/json",
        "requesttoken": "***************",
        "Sec-Fetch-Dest": "empty",
        "Sec-Fetch-Mode": "cors",
        "Sec-Fetch-Site": "same-origin",
        "Sec-GPC": "1",
        "Alt-Used": "***************",
    },
    "body": "{\"targetDirectoryIds\":\"[\\\"132366\\\",\\\"1329883\\\"]\",\"excludeList\":{\"user\":{\"mask\":[],\"fileid\":[]},\"admin\":{\"mask\":[],\"fileid\":[]}},\"collectorSettings\":{\"hashing_algorithm\":\"dhash\",\"similarity_threshold\":\"99\",\"hash_size\":\"16\",\"target_mtype\":0,\"finish_notification\":true}}",
    "method": "POST",
    "mode": "cors"
});

POST data JSON:

{"targetDirectoryIds":"[\"132366\",\"1329883\"]","excludeList":{"user":{"mask":[],"fileid":[]},"admin":{"mask":[],"fileid":[]}},"collectorSettings":{"hashing_algorithm":"dhash","similarity_threshold":"99","hash_size":"16","target_mtype":0,"finish_notification":true}}

The server reply is this:

503 Service Unavailable HTTP/3 503 Service Unavailable

Response headers ``` x-content-type-options: nosniff x-frame-options: SAMEORIGIN date: Wed, 06 Sep 2023 19:38:35 GMT referrer-policy: no-referrer x-robots-tag: noindex, nofollow status: 503 Service Unavailable content-type: text/html; charset=UTF-8 strict-transport-security: max-age=63072000; includeSubDomains; preload x-permitted-cross-domain-policies: none content-encoding: gzip server: Caddy, Caddy pragma: no-cache x-powered-by: PHP/8.2.10 cache-control: no-store, no-cache, must-revalidate expires: Thu, 19 Nov 1981 08:52:00 GMT x-xss-protection: 1; mode=block vary: Accept-Encoding content-security-policy: **** ```

grafik

Storage is temporarily not available

Full HTML below:

Full HTML ```html Nextcloud

Nextcloud

Fehler

  • Storage is temporarily not available

Nextcloud – ein sicherer Ort für all deine Daten

```

Also in the GUI that button keeps on spinning endlessly (nothing else happens): grafik

Nothing can be found in the Nextcloud log about this. I even set the log level to debug in that Python API thing (default seems to be Warn), but it did not log more: grafik

Expected behavior Create a task or at least show (or log) a useful error.

Desktop (please complete the following information):

More system reports (copied):

System report from admin settings ```Json { "nextcloud-version": "27.0.2.1", "app-versions": { "cloud_py_api-version": "0.1.8", "mediadc-version": "0.3.6" }, "is-videos-supported": false, "is-snap": false, "arch": "amd64", "webserver": "Caddy/v2.7.4", "database": { "type": "mysql", "version": "10.6.15", "size": "642736128" }, "php-version": "8.2.10", "php-interpreter": "php", "python-interpreter-setting": "/usr/bin/python3", "os": "Linux", "os-release": "6.4.7-200.fc38.x86_64", "machine-type": "x86_64" } ``` Python API: ```json { "nextcloud-version": "27.0.2.1", "app-versions": { "cloud_py_api-version": "0.1.8" }, "is-videos-supported": false, "is-snap": false, "arch": "amd64", "webserver": "Caddy/v2.7.4", "database": { "type": "mysql", "version": "10.6.15", "size": "642736128" }, "php-version": "8.2.10", "php-interpreter": "php", "python-interpreter-setting": "/usr/bin/python3", "os": "Linux", "os-release": "6.4.7-200.fc38.x86_64", "machine-type": "x86_64" } ```

Additional context Nextcloud issue report data: https://gist.github.com/rugk/93b16842749408755ac64bdf7331f462

Unrelated(?), because in my Nextcloud log it does not happen at the same time when I trigger the request this error appears:

{
    "reqId": "ksGAxoR4j99BDwyu6uGP",
    "level": 3,
    "time": "2023-09-06T20:06:27+02:00",
    "remoteAddr": "****",
    "user": "rugk",
    "app": "mediadc",
    "method": "DELETE",
    "url": "/apps/mediadc/api/v1/tasks/3",
    "message": "Can't terminate CollectorTask background process.\n{\"id\":3,\"type\":\"manual\",\"owner\":\"rugk\",\"target_directory_ids\":\"[\\\"485\\\"]\",\"exclude_list\":\"{\\\"user\\\":{\\\"mask\\\":[],\\\"fileid\\\":[]},\\\"admin\\\":{\\\"mask\\\":[],\\\"fileid\\\":[]}}\",\"collector_settings\":\"{\\\"hashing_algorithm\\\":\\\"dhash\\\",\\\"similarity_threshold\\\":100,\\\"hash_size\\\":16,\\\"target_mtype\\\":0,\\\"finish_notification\\\":true,\\\"duplicated\\\":false}\",\"files_scanned\":38,\"files_total\":5722,\"files_total_size\":12232414183,\"deleted_files_count\":0,\"deleted_files_size\":0,\"created_time\":1694023458,\"finished_time\":0,\"updated_time\":1694023515,\"py_pid\":253,\"errors\":\"\"}",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0",
    "version": "27.0.2.1",
    "data": {
        "app": "mediadc"
    },
    "id": "64f8dc0a6c349"
}

[mediadc] Fehler: Can't terminate CollectorTask background process. {"id":3,"type":"manual","owner":"rugk","target_directory_ids":"[\"485\"]","exclude_list":"{\"user\":{\"mask\":[],\"fileid\":[]},\"admin\":{\"mask\":[],\"fileid\":[]}}","collector_settings":"{\"hashing_algorithm\":\"dhash\",\"similarity_threshold\":100,\"hash_size\":16,\"target_mtype\":0,\"finish_notification\":true,\"duplicated\":false}","files_scanned":38,"files_total":5722,"files_total_size":12232414183,"deleted_files_count":0,"deleted_files_size":0,"created_time":1694023458,"finished_time":0,"updated_time":1694023515,"py_pid":253,"errors":""}

DELETE /apps/mediadc/api/v1/tasks/3 from ***** by rugk at 2023-09-06T20:06:27+02:00

But that was just that I had to delete one (stopped, and before of that stop the) task to execute the new one (also BTW don't get why already finished tasks count against the limit of running at most 3 tasks?Edit: Oh I misinterpreted that, it is 1 for running. forget that.). And I pressed the delete or stop button multiple times so…