search

Sebclem / hassio-nextcloud-backup

Hass.io Add-on: Backup your snapshots to Nextcloud
GNU General Public License v3.0
127 stars 20 forks source link

Unable to upload backup above 1G in size :( #123

Open vincegre opened 2 years ago

vincegre commented 2 years ago

A clear and concise description of what the bug is:

extension has worked fine till backup size has passed 1Go in size (same problem on 2 HA instances and different Nextcloud servers). If I upload myself the backup on the NC it should go I have no problem either in web or using Nextcloud sync client. Problem is independant of NC version (problem happens with NC 21/22/23

Steps to reproduce the behavior:

happens all the time as soon as backup file goes above 1Go

A clear and concise description of what you expected to happen.

backup to continue to work as it was in the past !

System information

Logs in HA during a backup:

[12:57:56] INFO: Starting Node...
> nexcloud-backup@0.8.0 start
> node ./bin/www
[2022-04-08 12:57:58] [info]:   Log level: debug
[2022-04-08 12:57:58] [info]:   Backup timeout: 5400000
[2022-04-08 12:57:58] [info]:   Satus : Go !
[2022-04-08 12:57:58] [info]:   Initializing and checking webdav client...
[2022-04-08 12:57:58] [info]:   Starting auto clean cron...
[2022-04-08 12:57:58] [info]:   Starting Cron...
[2022-04-08 12:57:58] [info]:   Hassio API : Go !
[2022-04-08 12:57:59] [debug]:  Nextcloud connection:  Success !
[2022-04-08 12:57:59] [debug]:  Path /Config_softs/ already exist.
[2022-04-08 12:57:59] [debug]:  Path /Config_softs/HA/ already exist.
[2022-04-08 12:57:59] [info]:   Nextcloud connection : Go !
[2022-04-08 12:58:27] [info]:   Stopping addons...
[2022-04-08 12:58:27] [info]:   ... Ok
[2022-04-08 12:58:27] [info]:   Creating new snapshot...
[2022-04-08 12:58:27] [debug]:  Addon to backup:
[2022-04-08 12:58:27] [debug]:  a0d7b954_adguard,core_check_config,5c53de3b_esphome,core_configurator,c4d25c74_edge,a0d7b954_glances,a0d7b954_grocy,core_mariadb,core_mosquitto,229cc4d7_nextcloud_backup,a0d7b954_phpmyadmin,core_samba,core_ssh,a0d7b954_unifi
[2022-04-08 12:58:27] [debug]:  Folders to backup:
[2022-04-08 12:58:27] [debug]:  homeassistant,ssl,share,media,addons/local
[2022-04-08 13:16:07] [info]:   Snapshot created with id c795316f
[2022-04-08 13:16:07] [info]:   Downloading snapshot c795316f...
[2022-04-08 13:16:07] [debug]:  Snapshot size: 2081.25
[2022-04-08 13:16:49] [info]:   Download success !
[2022-04-08 13:16:49] [debug]:  Snapshot dl size : 2081.25
[2022-04-08 13:16:52] [info]:   Uploading snap...
[2022-04-08 13:16:52] [debug]:  ...URI: https://host.hiden/remote.php/webdav/Config_softs/HA/Manual/Manual-2022.4.1-2022-04-08_1258.tar
[2022-04-08 13:16:52] [debug]:  ...rejectUnauthorized: true
[2022-04-08 13:17:58] [error]:  Fail to upload snapshot to nextcloud (HTTPError: Response code 500 (Internal Server Error)) !
[2022-04-08 13:17:58] [error]:  HTTPError: Response code 500 (Internal Server Error)
    at Request._onResponseBase (/opt/nextcloud_backup/node_modules/got/dist/source/core/index.js:899:31)
    at Request._onResponse (/opt/nextcloud_backup/node_modules/got/dist/source/core/index.js:934:24)
    at ClientRequest.<anonymous> (/opt/nextcloud_backup/node_modules/got/dist/source/core/index.js:948:23)
    at Object.onceWrapper (events.js:520:26)
    at ClientRequest.emit (events.js:412:35)
    at ClientRequest.origin.emit (/opt/nextcloud_backup/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)
    at HTTPParser.parserOnIncomingClient (_http_client.js:647:27)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:127:17)
    at TLSSocket.socketOnData (_http_client.js:515:22)
    at TLSSocket.emit (events.js:400:28)
(node:352) UnhandledPromiseRejectionWarning: Fail to upload snapshot to nextcloud (HTTPError: Response code 500 (Internal Server Error)) !
(Use `node --trace-warnings ...` to show where the warning was created)
(node:352) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:352) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Side note: why logs show a non-sense URL (https://host.hiden) for upload that is not the one setup in extension ??

Thanks

Vincèn

Sebclem commented 2 years ago

Hi,

I think it's related to your Nextcloud setup, did you find anything in your Nextcloud logs ?

Ps: the hidden.host is just something I have added for privacy when you share your logs here

vincegre commented 2 years ago

I think it's related to your Nextcloud setup, did you find anything in your Nextcloud logs ?

Well as stated in my bug report I have no problem to upload files at these NC instances either with the synchronisation client or through the web interface that are quite larger than HA backups so I don't think NC is the problem !

All I could find in NC but not sure it's related :/ Do you use the lib axios for NC access ??

{"reqId":"7WwiOracS5sqgHQjARAd","level":0,"time":"2022-04-08T11:17:12+00:00","remoteAddr":"51.154.57.249","user":"--","app":"camerarawpreviews","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"7WwiOracS5sqgHQjARAd","level":0,"time":"2022-04-08T11:17:12+00:00","remoteAddr":"51.154.57.249","user":"--","app":"serverDI","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"The requested alias \"PreviewManager\" is deprecated. Please request \"OCP\\IPreview\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"7WwiOracS5sqgHQjARAd","level":0,"time":"2022-04-08T11:17:12+00:00","remoteAddr":"51.154.57.249","user":"--","app":"maps","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"j3rQvQlRjouqFpP3EWZc","level":0,"time":"2022-04-08T11:17:13+00:00","remoteAddr":"51.154.57.249","user":"--","app":"camerarawpreviews","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"j3rQvQlRjouqFpP3EWZc","level":0,"time":"2022-04-08T11:17:13+00:00","remoteAddr":"51.154.57.249","user":"--","app":"serverDI","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"The requested alias \"PreviewManager\" is deprecated. Please request \"OCP\\IPreview\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"j3rQvQlRjouqFpP3EWZc","level":0,"time":"2022-04-08T11:17:13+00:00","remoteAddr":"51.154.57.249","user":"--","app":"maps","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"XgwrCNqweCHiBIGvNlyq","level":0,"time":"2022-04-08T11:17:17+00:00","remoteAddr":"51.154.57.249","user":"--","app":"camerarawpreviews","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"XgwrCNqweCHiBIGvNlyq","level":0,"time":"2022-04-08T11:17:17+00:00","remoteAddr":"51.154.57.249","user":"--","app":"serverDI","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"The requested alias \"PreviewManager\" is deprecated. Please request \"OCP\\IPreview\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"XgwrCNqweCHiBIGvNlyq","level":0,"time":"2022-04-08T11:17:17+00:00","remoteAddr":"51.154.57.249","user":"--","app":"maps","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"pAc5Tn5Es5Ur66KF4d8X","level":0,"time":"2022-04-08T11:17:18+00:00","remoteAddr":"51.154.57.249","user":"--","app":"camerarawpreviews","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"pAc5Tn5Es5Ur66KF4d8X","level":0,"time":"2022-04-08T11:17:18+00:00","remoteAddr":"51.154.57.249","user":"--","app":"serverDI","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"The requested alias \"PreviewManager\" is deprecated. Please request \"OCP\\IPreview\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"pAc5Tn5Es5Ur66KF4d8X","level":0,"time":"2022-04-08T11:17:18+00:00","remoteAddr":"51.154.57.249","user":"--","app":"maps","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"9HkhbjnDSe1OkT0iIfa2","level":0,"time":"2022-04-08T11:17:22+00:00","remoteAddr":"51.154.57.249","user":"--","app":"camerarawpreviews","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"9HkhbjnDSe1OkT0iIfa2","level":0,"time":"2022-04-08T11:17:22+00:00","remoteAddr":"51.154.57.249","user":"--","app":"serverDI","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"The requested alias \"PreviewManager\" is deprecated. Please request \"OCP\\IPreview\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"9HkhbjnDSe1OkT0iIfa2","level":0,"time":"2022-04-08T11:17:22+00:00","remoteAddr":"51.154.57.249","user":"--","app":"maps","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"ACong6b5pZBi5QadlHwt","level":0,"time":"2022-04-08T11:17:23+00:00","remoteAddr":"51.154.57.249","user":"--","app":"camerarawpreviews","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"ACong6b5pZBi5QadlHwt","level":0,"time":"2022-04-08T11:17:23+00:00","remoteAddr":"51.154.57.249","user":"--","app":"serverDI","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"The requested alias \"PreviewManager\" is deprecated. Please request \"OCP\\IPreview\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"ACong6b5pZBi5QadlHwt","level":0,"time":"2022-04-08T11:17:23+00:00","remoteAddr":"51.154.57.249","user":"--","app":"maps","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"CXCpnaVrFFa97tjN27XG","level":0,"time":"2022-04-08T11:17:27+00:00","remoteAddr":"51.154.57.249","user":"--","app":"camerarawpreviews","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"CXCpnaVrFFa97tjN27XG","level":0,"time":"2022-04-08T11:17:27+00:00","remoteAddr":"51.154.57.249","user":"--","app":"serverDI","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"The requested alias \"PreviewManager\" is deprecated. Please request \"OCP\\IPreview\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"CXCpnaVrFFa97tjN27XG","level":0,"time":"2022-04-08T11:17:27+00:00","remoteAddr":"51.154.57.249","user":"--","app":"maps","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"Zo1GY4KIU2Syc0doBAA4","level":0,"time":"2022-04-08T11:17:28+00:00","remoteAddr":"51.154.57.249","user":"--","app":"camerarawpreviews","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"Zo1GY4KIU2Syc0doBAA4","level":0,"time":"2022-04-08T11:17:28+00:00","remoteAddr":"51.154.57.249","user":"--","app":"serverDI","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"The requested alias \"PreviewManager\" is deprecated. Please request \"OCP\\IPreview\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"Zo1GY4KIU2Syc0doBAA4","level":0,"time":"2022-04-08T11:17:28+00:00","remoteAddr":"51.154.57.249","user":"--","app":"maps","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"8NyS2Hab0MmriC64c4iJ","level":0,"time":"2022-04-08T11:17:32+00:00","remoteAddr":"51.154.57.249","user":"--","app":"camerarawpreviews","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"8NyS2Hab0MmriC64c4iJ","level":0,"time":"2022-04-08T11:17:32+00:00","remoteAddr":"51.154.57.249","user":"--","app":"serverDI","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"The requested alias \"PreviewManager\" is deprecated. Please request \"OCP\\IPreview\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"8NyS2Hab0MmriC64c4iJ","level":0,"time":"2022-04-08T11:17:32+00:00","remoteAddr":"51.154.57.249","user":"--","app":"maps","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Manual/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"4pOnuylU8TxSQXvH3KJd","level":0,"time":"2022-04-08T11:17:33+00:00","remoteAddr":"51.154.57.249","user":"--","app":"camerarawpreviews","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"axios/0.24.0","version":"23.0.3.2"}
{"reqId":"4pOnuylU8TxSQXvH3KJd","level":0,"time":"2022-04-08T11:17:33+00:00","remoteAddr":"51.154.57.249","user":"--","app":"serverDI","method":"PROPFIND","url":"/remote.php/webdav/Config_softs/HA/Auto/","message":"The requested alias \"PreviewManager\" is deprecated. Please request \"OCP\\IPreview\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"axios/0.24.0","version":"23.0.3.2"}

Ps: the hidden.host is just something I have added for privacy when you share your logs here

I better understand and may I suggest to replace that very bad taste of fake url by something easier like "yourhosthere» or "hidden for confidentiality" for example ?

vdwals commented 2 years ago

Hi, I experience the same issue. I can download the backups from Home Assistant and upload them via the Web UI, the same endpoint effectively the addon would use. But the addon return 413. I could not find a related entry in the nextcloud logs. Nextcloud is running behind the Nginx Proxy Manager, but it does not matter if I use the external accessible URL or the internal IP address. Also the nextcloud system information shows the correct limits (16 gb in my instance), but only the addon is not able to upload 1.6 or 1.8 gb files.

Sebclem commented 2 years ago

Hi, I experience the same issue. I can download the backups from Home Assistant and upload them via the Web UI, the same endpoint effectively the addon would use. But the addon return 413. I could not find a related entry in the nextcloud logs. Nextcloud is running behind the Nginx Proxy Manager, but it does not matter if I use the external accessible URL or the internal IP address. Also the nextcloud system information shows the correct limits (16 gb in my instance), but only the addon is not able to upload 1.6 or 1.8 gb files.

For 413 related error, please see #9

sippeeey commented 1 year ago

i had the same error with HTTP 413 Error. I used the official nextcloud docker image. My Nextcloud runs with apache webserver behind a traeffic proxy and i must add LimitRequestBody 0 in apache webserver configuration. i do this with a separate docker volume /docker/nextcloud/nextcloud.conf:/etc/apache2/conf-enabled/nextcloud.conf

now it works

mhzawadi commented 3 months ago

Hello 👋🏼

I too have hit the upload limit on my nextcloud server, I have for the time being upped the limit.

But on looking at the code, I can see that chunked upload is a thing. But its not used from what I can see, why?

Sebclem commented 3 months ago

Hi, Chunked upload is a thing only on the main branch, I'm still doing some testing before making a first "beta" release since I have changed a lot of things.

If you still want to test the development version with chunked upload, a completely rewritten UI and typescript, you can add this as a repository and install the main.xxx version : https://github.com/Sebclem/sebclem-hassio-addon-repository#main This is still an early dev version , I'm still trying to solve an issue with error handling that makes the add-on crash if an error happens during Cron backup, so don't use this version as your primary backup solution

mhzawadi commented 3 months ago

Thats great news, I'll get back in my box now