Closed onny closed 3 years ago
I am also affected by this. I run Arch Linux. I even tried a fresh install (but old data directory). The web interface works as expected.
I also run Arch Linux, and in the web interface, of O go to the page admin/overview, it says, among other things
Last background job execution ran 9 days ago. Something seems wrong. Check the background job settings
And if go to settings/admin#backgroundjobs I see that the choice "cron" is checked, and if try to check webcron or ajax, and leave the page to come back, cron is still checked. So even the webinterface doesn't work correctly.
debian 10 here, same issue: tried with php 7.0 7.3 and 8.0 no joy, this is a nextcloud issue not an operative system one...
I upgraded from 21.0.0 RC2 to 21.0.0 and the issue presented itself on the upgrade at the end of the update process (that is the first time occ is run AFTER the files changed from rc2 to stable), this suggests me the problem might be somewhere in the changelog between the rc2 and the production ?
As a matter of fact, there are whole bunch of things that don't work with web interface. I can't update, enable, disable, remove or download applications.
As it might be relevant, here is some info about the php cli configuration:
I can reproduce this bug with Nextcloud stable21 and php v7.4 on Arch Linux.
Another finding: The occ command works fine if the instance is not yet installed. After installing (e.g. using web installer or occ install command) it becomes unusable.
Currently I suspect the upgrade process somehow corrupted the Mysql database? I get following erros from the mysql/mariadb daemon:
InnoDB: Page [...] log sequence number [...] is in the future!
Feb 22 17:42:03 mysql mariadbd[327]: 2021-02-22 17:42:03 254850 [ERROR] InnoDB: Page [page id: space=2026, page number=45620] log sequence number 352586303314 is in the future! Current system log sequence number 320954826491.
Feb 22 17:42:03 mysql mariadbd[327]: 2021-02-22 17:42:03 254850 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
If I stop the database, occ runs until connection timeout to Mysql server. So my guess is Occ crashes after connecting to the database ...
Okay maybe it is something different. Nevertheless I put this at line 133 in lib/private/AppFramework/Utility/SimpleContainer.php
:
echo "<pre>".print_r(debug_backtrace(),true)."</pre>";
Which is giving me this stack trace showing some kind of iteration with arrays including the PHP environment and configuration array?! https://onny.project-insanity.org/files/stacktrace_short.txt
Information which might be useful in this case:
/etc/webapps/nextcloud/config/config.php
. /usr/share/webapps/nextcloud/config
is a symlink to the config folder located in /etc/webapps
-folder/var/lib/nextcloud/data
This is what I get from logging mysql queries:
210222 21:05:09 6 Connect nextcloud@http.pi on nextcloud using TCP/IP
6 Query SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED
6 Query SET SESSION AUTOCOMMIT=1
6 Query SELECT * FROM `oc_appconfig`
Nothing more ...
Edit: oc_appconfig has a lot of keys so I don't know where to look at. In case some value in there is causing the issue ...
I hit OOM when upgrading to latest nextcloud by running occ upgrade. Is there any workaround or fix?
I upgraded from 21.0.0 RC2 to 21.0.0 and the issue presented itself on the upgrade at the end of the update process (that is the first time occ is run AFTER the files changed from rc2 to stable), this suggests me the problem might be somewhere in the changelog between the rc2 and the production ?
I second this. I have been testing using the stable21 branch last week and it worked fine until a couple days ago. The regression was introduced recently.
Maybe bisecting from 21RC2 to the current version is a good starting point.
Same issue here. Arch Linux ARM, but with Postgres backend, so it might not be related to mysql at all.
occ
attempts to allocate a lot more memory on startup than usual on 21.0.0, I tried increasing memory_limit
in php.ini
but that just causes it to get oomkilled at some point.
Disabling all extensions except for pgsql
and pdo_pgsql
(without these two occ
just crashes) still results in the same behaviour.
Even with memory_limit
set to -1
(so no limit at all), occ
will fail and trigger OOMKiller on my server with 32GiB RAM and 32GiB Swap…
Same problem on FreeBSD 12.2, php 7.3.
Upgrades to NC 21 RC 1 and NC 21 RC 2 went through without problems. Now it's just the main release NC 21.0.0:
PHP Fatal error: Allowed memory size of 8489271296 bytes exhausted (tried to allocate 20480 bytes) in /usr/local/www/nextcloud-test/lib/private/AppFramework/Utility/SimpleContainer.php on line 133
Fatal error: Allowed memory size of 8489271296 bytes exhausted (tried to allocate 20480 bytes) in /usr/local/www/nextcloud-test/lib/private/AppFramework/Utility/SimpleContainer.php on line 133
and a few more reports on the forum: https://help.nextcloud.com/t/occ-command-php-fatal-error-allowed-memory-size-of-xxx-bytes-exhausted/108521
Also having this problem on FreeBSD 12.2, with php 7.4.
Downgrading (using FreeBSD binary packages built myself) restores functionality, the DB looks unmodified by the botched update.
I had exactly the same issue after I upgraded from NC 20.0.7 to 21.0.0 release. The cron.php also did not run.
CentOS 7 - postgreSQL 13 - php 7.4 or 8.0 (did not matter)
I "solved" the problem by downgrading to NC 21.0.0 RC2
@onny Try to enable apc.enable_cli=1 in /etc/php/8.0/cli/conf.d/20-apcu.ini. After I added that line to the file occ worked and the memory didn't get filled anymore. Also see the warning here in the manual.
@andreaslutsch your solution worked for me (21.0.0, ArchLinux). For reference for others, you can enable the option by adding the parameter --define apc.enable_cli=1
to your php cli call (e.g. php --define apc.enable_cli=1 occ [...]
).
@onny Do you think, this issue is fixed? For me it is a regression and thus a bug when software changes it behavior and is crashing, when there is something missing in the environment. At least this should be documented, if apc is a hard requirement (and checked by the application when starting and throwing an error message). Enabling apc is in this case just a "workaround". So can you please reopen the issue so it can be fixed?
The apc.enable_cli workaround does not work here (21.0.0, Alpine Linux 3.13, PHP 7.4.15).
@waja it looks like a regression for me because it worked without this option before and it's only recommended for the cronjob.
Similar issue here on Ubuntu server 20.10, php 7.4, MariaDB 10.3, whatever amount of memory I allocate to my Nextcloud instance gets exhausted.
I tried the apc_enable_cli workaround but it didn't change anything it still ends up being OOM-killed
the command I'm being asked to run is php occ db:add-missing-indices
.
Also, no Cron jobs can complete, they also run out of memory.
Thanks for your help
I tried the apc_enable_cli workaround but it didn't change anything it still ends up being OOM-killed
Just in case: it is apc.enable_cli
with a dot. It's working for me with the changed config file. Maybe also check, if you are using the correct php version and try running it with explicit version command e.g. php7.4 -f /var/www/nc/cron.php
.
Edit: added version info
OMV 5.6.0-1/Debian GNU/Linux 10 (buster)
PHP 8.0.2 (cli)
I ran git bisect and the commit a745d5813355fe60c9fc3ef2f9dff69e85a3f8d0 seems to be introducing this regression.
I tried the apc_enable_cli workaround but it didn't change anything it still ends up being OOM-killed
Just in case: it is
apc.enable_cli
with a dot. It's working for me with the changed config file. Maybe also check, if you are using the correct php version and try running it with explicit version command e.g.php7.4 -f /var/www/nc/cron.php
.
Thanks for your reply. sorry I did use apc.enable_cli, I typed it above rather than copy/paste. I also tried both php and php7.4
EDIT: my mistake it worked with apc.enable_cli=1
I confirm that adding apc.enable_cli=1
into /etc/php/7.4/mods-available/apcu.ini
on my Debian/testing solved this issue.
Thanks for your reply. sorry I did use apc.enable_cli, I typed it above rather than copy/paste. I also tried both php and php7.4
Do you have APCu enabled on your Nextcloud instance? Otherwise this line will have no effect. Maybe other cache backends are affected aswell.
config.php: 'memcache.local' => '\OC\Memcache\APCu',
I ran git bisect and the commit a745d58 seems to be introducing this regression.
this was submitted by @nickvergessen : could you take a look at this or find someone to attribute this issue? There is quite a number of people affected.
It also affects cron.php, not only occ
Adding apc.enable_cli=1
also fixed it for me but only for php7.4, interestingly though not for php8.0 as soon as I switch the php version the bug appears again and consumes all the memory.
I added it to the cron command */5 * * * * php --define apc.enable_cli=1 -f /var/www/nextcloud/cron.php
to /etc/php/8.0/cli/php.ini
and /etc/php/8.0/mods-available/apcu.ini
same for 7.4.
@Grotax If I understood correctly, apc.enable_cli=1
should be added only in it's configuration file in /etc/php/8.0/cli/conf.d/20-apcu.ini
because if you restart the php-fpm service, it will rewrite the /etc/php/8.0/mods-available/apcu.ini
file. If you add it here you don't need to add the define in the cron command, as it's already defined in the conf file.
Also if you just installed php 8.0, make sure to configure it and install all necessary modules, I also used php 7.4 that worked before and forgot to update php 8.0 settings after installing it.
@andreaslutsch thanks for your response under ubuntu /etc/php/8.0/cli/conf.d/*
are just symlinks to /etc/php/8.0/mods-available/*
As you said correctly I don't need to use it in all those places, it was just for debugging.
Your assumption that I just installed php8.0 is correct but ubuntu enables the php mods through the packages, unlike arch where you enable them in the php.ini I still checked via diff, the only meaningful diff was the memory limit 1024M vs 128M so yea I adjusted that.
EDIT: sudo -u www-data php8.0 --define apc.enable_cli=1 ./occ
results in 100%cpu load and memory consumption for me
EDIT2: Found it, apcu was for some reason not installed correctly for php8.0 ...
@tflidd well if apcu is configured in the config.php it also needs to be available all the time which includes CLI like cron and occ? 😬 Not sure the code is doing something wrong, we use code like that in many places already.
@tflidd well if apcu is configured in the config.php it also needs to be available all the time which includes CLI like cron and occ? 😬 Not sure the code is doing something wrong, we use code like that in many places already.
Is there no way of checking if that is enabled in CLI mode? Many used it wrongly before (when it was working) and ending up with an half-installed upgrade is a pretty bad user experience.
@nickvergessen You are right. I traced the code causing the recursion. There is an if block which allows executing cli code even if the configured cache backend is unavailable and exactly that check is causing the recursion and subsequent oom. The commit I mentioned earlier is just fine.
There even is a warning in the documentation which reminds admins to set apc.enable_cli=1
.
IMO, the cli should fail just like the server does, if the configured cache backend is not available. Otherwise, this issue will reoccur in the future.
I will propose a PR and mention you.
Within my update-routine I use:
/updater/updater.phar --no-interaction
Enabling apc.enable_cli did not fix the memory issue yet. In addition within my setup redis was not enabled for cli. After having enabled both redis and apc for cli the upgrade succeeded. Nevertheless the upgrade could not be continued by using /updater/updater.phar --no-interaction
as this kept throwing this memory error. After running occ upgrade
directly the upgrade was finished.
For versions up to 20.0.8 this was not necessary.
@Rumbelstilzchen Can confirm this. If you setup any cache backend and then disable it in php config (e.g. php.ini) occ will fail.
If you install with mysql and disable mysql in the php config it will also fail 😳 [/joke]
That was exactly my point. You configure nextcloud and say it's there and turn it off. It's totally expected to not work anymore.
I had the same OOM on upgrade from 20.0.7 to 21.0.0 on PHP 7.4.15 . I tried the apc workaround but it didn't work. Then had in my config:
'memcache.distributed' => '\\OC\\Memcache\\Memcached',
'memcached_servers' =>
array (
0 =>
array (
0 => '127.0.0.1',
1 => 11211,
),
),
I commented this out and it worked. I could execute occ upgrade and occ app:update on the command line. Once I enable it, I get the same error.
I can reconfirm that adding apc.enable_cli=1
into /etc/php/7.4/mods-available/apcu.ini
on Debian Buster solved this issue.
At the same time though make sure your SQL Database ist up to date to suit Nextcloud v21's needs.
One installation was hanging until MariaDB was updated and the tables all properly upgraded. Only then the update to v21 could be finished.
I can also confirm that adding
apc.enable_cli=1
to /etc/php/7.3/mods-available/apcu.ini
helps.
In my case the cron.php job brought down the entire server (not even the mouse moved)
My configuration: Debian Buster running in a LXD container on Ubuntu 20.04.
I can also confirm that adding
apc.enable_cli=1
to /etc/php/7.3/mods-available/apcu.ini
helps.
In my case the cron.php job brought down the entire server (not even the mouse moved)
My configuration: Debian Buster running in a LXD container on Ubuntu 20.04.
Exact same Situation here.
In my case the cron.php job brought down the entire server (not even the mouse moved)
I had the same problem and needed several restarts. I saw in htop that the cron/occ allocates all available RAM.
To prevent this behaviour I've added a memory limit to the cli php.ini
memory_limit = 4G
Edit: In addition to the apc settings that already fixed the problem. Just in case...
Using Nextcloud 21.0.0 (nextcloud-php80-21.0.0) on FreeBSD, I don't have that issue. The difference with @tflidd and @madpilot78 is the PHP version (I'm using PHP 8.0). However, php is using up an entire core ~non stop on the machine it runs on:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10246 www 1 117 19 327M 275M CPU1 1 202:52 86.67% php
# php -i | grep -i apc
PHP Warning: Missing arginfo for Imagick::__tostring() in Unknown on line 0
PHP Warning: Missing arginfo for Imagick::getimagehistogram() in Unknown on line 0
/usr/local/etc/php/ext-20-apcu.ini,
apcu
apc.coredump_unmap => Off => Off
apc.enable_cli => Off => Off
apc.enabled => On => On
apc.entries_hint => 4096 => 4096
apc.gc_ttl => 3600 => 3600
apc.mmap_file_mask => no value => no value
apc.preload_path => no value => no value
apc.serializer => php => php
apc.shm_segments => 1 => 1
apc.shm_size => 32M => 32M
apc.slam_defense => Off => Off
apc.smart => 0 => 0
apc.ttl => 0 => 0
apc.use_request_time => Off => Off
I fixed the issue by adding apc.enable_cli=1 to my php configuration.
From what I gather here I had apc enabled for the webserver php so the command line tool fails if it's not enabled, I think that's reasonable, they need the same setup. In my mind the issue is caused by a misconfiguration on my part, and it used to work just by chance before.
I have the same issue on docker image nextcloud:21-fpm-alpine
I fixed it with nc config: memory_limit=512M
Container file path: /usr/local/etc/php/conf.d/nextcloud.ini
Worked version: https://github.com/msergei/nextcloud
Same story here, I wish I had known about this. I had to do a manual upgrade to 21, and temporarily applied -1 to php memory size and luckily it worked. But the cron job was broken which lead me to here.
Arch, PHP 8.0.3
Same for me.
And If I increase the value of memory_limit
, the scripts get killed:
cedric@ssh1:~/www/cloud$ php /home/cedric/www/cloud/cron.php
Killed
cedric@ssh1:~/www/cloud$ php occ db:add-missing-indices
Killed
I just updated to 21.0.1.
For the database upgrade issue, the exception (Allowed memory size...) is raised on line 133 of /lib/private/AppFramework/Utility/SimpleContainer.php.
I have also run into this on my NextCloud instance while upgrading 20 to 21.. occ upgrade basically take all the memory, almost crashing the server and then is killed:
$ sudo -u www-data php /var/www/nextcloud/occ upgrade
Killed
What the... ?
EDIT: I have solved it, I have read this advice on other forum. I have edited this file:
/etc/php/7.4/cli/conf.d$ sudo vim 20-apcu.ini
Added one line:
apc.enable_cli=1
And then run occ upgrade again and it worked. Super weird.
Thank you @Gondy , it works. It's not ideal, but let's say it solves the problem right now.
How to use GitHub
Steps to reproduce
php ./occ
in Nextcloud root directory./occ
crashesExpected behaviour
Occ should work normally, for example printing available commands
Actual behaviour
Occ crashes with the following error:
To be more precise, this also happens running php cli against other files of Nextcloud, for example
index.php
orconsole.php
. Surprisingly Nextcloud is running fine through php-fpm and web server. It was even able to perform the upgrade to v21 just fine.Another user commented on the ArchLinux bug tracker:
Possibly not only related to ArchLinux setups. An other user reported the exact issue here, which was triggered on a Ubuntu installation.
Server configuration
Operating system: Archlinux x64
Web server: Caddy 2
Database: Mysql
PHP version: 8.0.2
Nextcloud version: (see Nextcloud admin page) 21.0.0
Updated from an older Nextcloud/ownCloud or fresh install: Yes
Where did you install Nextcloud from: Package repository
Signing status:
Signing status
``` Unable to access for now ```List of activated apps:
App list
``` accessibility/ drawio/ files_trashbin/ nextcloud_announcements/ settings/ twofactor_gateway/ activity/ encryption/ files_versions/ notes/ sharebymail/ updatenotification/ admin_audit/ extract/ files_videoplayer/ notifications/ spreed/ user_ldap/ apporder/ federatedfilesharing/ firstrunwizard/ oauth2/ support/ user_status/ calendar/ federation/ integration_discourse/ password_policy/ survey_client/ viewer/ cloud_federation_api/ files/ integration_github/ photos/ suspicious_login/ weather_status/ comments/ files_ebookreader/ integration_reddit/ polls/ systemtags/ workflowengine/ contacts/ files_external/ integration_twitter/ privacy/ tasks/ contactsinteraction/ files_pdfviewer/ keeweb/ provisioning_api/ text/ dashboard/ files_rightclick/ logreader/ recommendations/ theming/ dav/ files_sharing/ lookup_server_connector/ serverinfo/ twofactor_backupcodes/ ```Nextcloud configuration:
Config report
``` '****', 'datadirectory' => '/var/lib/nextcloud/data', 'dbtype' => 'mysql', 'version' => '21.0.0.18', 'dbname' => 'nextcloud', 'dbhost' => 'mysql.pi', 'mysql.utf8mb4' => true, 'dbuser' => 'nextcloud', 'dbpassword' => '****', 'installed' => true, 'maintenance' => false, 'loglevel' => 0, 'cron_log' => true, 'maxZipInputSize' => 5145728000, 'allowZipDownload' => true, 'memcache.local' => '\\OC\\Memcache\\APCu', 'allow_local_remote_servers' => true, 'trusted_domains' => array ( 0 => 'nextcloud.****.org', ), 'passwordsalt' => '***', 'secret' => '****', 'mail_domain' => '******.org', 'mail_smtpmode' => 'php', 'mail_from_address' => 'nextcloud', 'trashbin_retention_obligation' => 'auto', 'updatechecker' => false, 'app.mail.verify-tls-peer' => false, 'overwrite.cli.url' => 'https://nextcloud.***.org', 'default_phone_region' => 'DE', ); ```Are you using external storage, if yes which one: None, had configured SFTP mount point
Are you using encryption: No
Are you using an external user-backend, if yes which one: No
Client configuration
Browser: Latest Firefox
Operating system: NixOS
Nextcloud log (data/nextcloud.log)
Nextcloud log
``` [...] {"reqId":"pti6pSFymOb7PvbzzzmM","level":0,"time":"2021-02-21T18:08:04+00:00","remoteAddr":"2****","user":"--","app":"files_sharing","method":"GET","url":"/cron.php","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0","version":"21.0.0.18"} {"reqId":"pti6pSFymOb7PvbzzzmM","level":0,"time":"2021-02-21T18:08:04+00:00","remoteAddr":"*****","user":"--","app":"keeweb","method":"GET","url":"/cron.php","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0","version":"21.0.0.18"} {"reqId":"pti6pSFymOb7PvbzzzmM","level":0,"time":"2021-02-21T18:08:04+00:00","remoteAddr":"*****","user":"--","app":"onlyoffice","method":"GET","url":"/cron.php","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0","version":"21.0.0.18"} ```