search

nextcloud / backup

Backup now. Restore later.
GNU Affero General Public License v3.0
249 stars 37 forks source link

S3 upload still failes with OOM for 1.0.6 #237

Open ibayer opened 2 years ago

ibayer commented 2 years ago

The latest release still has the OOM bug when uploading to remote storage.

image

  > Health status: 0 correct, 363 missing and 0 faulty files
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00001-4J8cz2vkcCkabEn: ok
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00002-KNuSXKImJx65wKm: PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 104857824 bytes) in /var/www/html/lib/private/Files/Storage/Common.php on line 208
PHP Fatal error:  Uncaught Error: Class "OC\Log\ExceptionSerializer" not found in /var/www/html/lib/private/Log.php:311
Stack trace:
#0 /var/www/html/lib/private/Log/ErrorHandler.php(93): OC\Log->logException(Object(Error), Array)
#1 [internal function]: OC\Log\ErrorHandler::onError(2, 'fopen(/tmp/oc_t...', '/var/www/html/a...', 641)
#2 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(641): fopen('/tmp/oc_tmp_6Qo...', 'r')
#3 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(501): OCA\Files_External\Lib\Storage\AmazonS3->writeBack('/tmp/oc_tmp_6Qo...', 'backups/2022032...')
#4 [internal function]: OCA\Files_External\Lib\Storage\AmazonS3->OCA\Files_External\Lib\Storage\{closure}()
#5 /var/www/html/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php(119): call_user_func(Object(Closure))
#6 [internal function]: Icewind\Streams\CallbackWrapper->stream_close()
#7 {main}
  thrown in /var/www/html/lib/private/Log.php on line 311
Vrumf0ndel commented 2 years ago

As a temporary workaround, you could increase the related setting in your php.ini file from 512M to a higher value, e.g. 768M. But be aware it's only a workaround and it does not attempt to fix the underlying problem. It can also cause your web server to run out of RAM.

; Maximum amount of memory a script may consume
; http://php.net/memory-limit
memory_limit=768M
ibayer commented 2 years ago

Would be interesting to see if this makes the app work or if other bugs pop up after this.

Knowing nothing about php I tried to find this file but :/srv/nextcloud# find . -name "php.ini" didn't point me to any php.ini file. I guess I can create this file somewhere but I'm not sure what would be the appropriate place with the official nextcloud docker container.

Vrumf0ndel commented 2 years ago

Just type php --ini and it will give you a lot of information. The information you're looking for is Loaded Configuration File:: e.g. Loaded Configuration File: /etc/php7/php.ini

Keep in mind that after restart of your docker container the changes are gone. So it's only for testing purposes.

ibayer commented 2 years ago

/etc/php7/php.ini doesn't exist and loaded is (none). :shrug: 

Configuration File (php.ini) Path: /usr/local/etc/php
Loaded Configuration File:         (none)
Scan for additional .ini files in: /usr/local/etc/php/conf.d
Additional .ini files parsed:      /usr/local/etc/php/conf.d/docker-php-ext-apcu.ini,
/usr/local/etc/php/conf.d/docker-php-ext-bcmath.ini,
/usr/local/etc/php/conf.d/docker-php-ext-exif.ini,
/usr/local/etc/php/conf.d/docker-php-ext-gd.ini,
/usr/local/etc/php/conf.d/docker-php-ext-gmp.ini,
/usr/local/etc/php/conf.d/docker-php-ext-imagick.ini,
/usr/local/etc/php/conf.d/docker-php-ext-intl.ini,
/usr/local/etc/php/conf.d/docker-php-ext-ldap.ini,
/usr/local/etc/php/conf.d/docker-php-ext-memcached.ini,
/usr/local/etc/php/conf.d/docker-php-ext-opcache.ini,
/usr/local/etc/php/conf.d/docker-php-ext-pcntl.ini,
/usr/local/etc/php/conf.d/docker-php-ext-pdo_mysql.ini,
/usr/local/etc/php/conf.d/docker-php-ext-pdo_pgsql.ini,
/usr/local/etc/php/conf.d/docker-php-ext-redis.ini,
/usr/local/etc/php/conf.d/docker-php-ext-sodium.ini,
/usr/local/etc/php/conf.d/docker-php-ext-zip.ini,
/usr/local/etc/php/conf.d/nextcloud.ini,
/usr/local/etc/php/conf.d/opcache-recommended.ini

edit: I have put it here :/var/www/html# vim /usr/local/etc/php/conf.d/php.ini let see what happens.

ibayer commented 2 years ago

Seems to upload 1 more file before blowing up. :)

docker exec -it --user www-data nextcloud-f9ec7847-3b83-bf79-2049-578f2b137b3e php occ backup:point:upload 20220323114316-full-abXVdLfbkxKV5ZE
- uploading 20220323114316-full-abXVdLfbkxKV5ZE to external folders
 - checking external folder 19:backups
  > restoring point found
  > Health status: 2 correct, 360 missing and 1 faulty files
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00003-AySDZQMtYbrCAyc: ok
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00004-Ohm7SQUAocykJVg: ok
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00005-phENEUkPuNnLZ72: ok
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00006-YGr1UcnBOAp8w1w: PHP Fatal error:  Allowed memory size of 805306368 bytes exhausted (tried to allocate 104857704 bytes) in /var/www/html/lib/private/Files/Storage/Common.php on line 208
PHP Fatal error:  Uncaught Error: Class "OC\Log\ExceptionSerializer" not found in /var/www/html/lib/private/Log.php:311
Stack trace:
#0 /var/www/html/lib/private/Log/ErrorHandler.php(93): OC\Log->logException(Object(Error), Array)
#1 [internal function]: OC\Log\ErrorHandler::onError(2, 'fopen(/tmp/oc_t...', '/var/www/html/a...', 641)
#2 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(641): fopen('/tmp/oc_tmp_wlw...', 'r')
#3 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(501): OCA\Files_External\Lib\Storage\AmazonS3->writeBack('/tmp/oc_tmp_wlw...', 'backups/2022032...')
#4 [internal function]: OCA\Files_External\Lib\Storage\AmazonS3->OCA\Files_External\Lib\Storage\{closure}()
#5 /var/www/html/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php(119): call_user_func(Object(Closure))
#6 [internal function]: Icewind\Streams\CallbackWrapper->stream_close()
#7 {main}
  thrown in /var/www/html/lib/private/Log.php on line 311
Vrumf0ndel commented 2 years ago

I'm using different docker container, so it looks somehow different.

Vrumf0ndel commented 2 years ago

Looks like a bug in the upload routines where memory is not freed after single data chunk upload is finished.

ibayer commented 2 years ago

Jep, feels like a memory leak. Not something that a higher memory limit can fix. But I didn't check the code and know very little about php...

Anyway, thanks for the idea. :+1:

varet80 commented 2 years ago

On the same train here! up to 8GB memory.. still fails

domaximus commented 2 years ago

set memory to -1 (unlimited) didn't fix it for me. triggering a manual upload uploads several files, but eventually the process stops with an event "Uploading Killed".

Any suggestions?

fvillena commented 2 years ago

Same problem on Backup 1.1.3

imagen

SpaghettiCodeBooy commented 1 year ago

Will this ever be fixed? This is really a deal-breaker for using NextCloud

danepowell commented 1 year ago

In the interest of consolidating discussion, would you mind closing this and upvoting #308?