nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.67k stars 4k forks source link

[Bug]: cron.php not working due to Cannot declare class OC\Command\CommandJob, because the name is already in use #44565

Open Quintasan opened 5 months ago

Quintasan commented 5 months ago

⚠️ This issue respects the following points: ⚠️

Bug description

My background jobs systemd service is constantly failing with:

PHP Fatal error: Cannot declare class OC\Command\CommandJob, because the name is already in use in /srv/www/cloud.frelia.org/lib/private/Command/CommandJob.php on line 31

Steps to reproduce

  1. sudo -u www-data php /srv/www/cloud.frelia.org/cron.php

Expected behavior

cron.php runs normally

Installation method

Community Manual installation with Archive

Nextcloud Server version

28

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.3

Web server

Nginx

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.frelia.org"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "version": "28.0.4.1",
        "logtimezone": "UTC",
        "installed": true,
        "maintenance": false,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "loglevel": 0,
        "overwriteprotocol": "https",
        "overwrite.cli.url": "https:\/\/cloud.frelia.org",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "theme": "",
        "updater.release.channel": "stable",
        "app_install_overwrite": [
            "calendar"
        ],
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_sendmailmode": "smtp",
        "mail_smtpport": "587",
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "default_phone_region": "PL",
        "has_rebuilt_cache": true,
        "updater.secret": "***REMOVED SENSITIVE VALUE***",
        "maintenance_window_start": 1
    }
}

List of activated Apps

Enabled:
  - activity: 2.20.0
  - admin_audit: 1.18.0
  - calendar: 4.6.7
  - circles: 28.0.0
  - cloud_federation_api: 1.11.0
  - comments: 1.18.0
  - contacts: 5.5.3
  - contactsinteraction: 1.9.0
  - dashboard: 7.8.0
  - dav: 1.29.1
  - deck: 1.12.2
  - federatedfilesharing: 1.18.0
  - federation: 1.18.0
  - files: 2.0.0
  - files_external: 1.20.0
  - files_pdfviewer: 2.9.0
  - files_reminders: 1.1.0
  - files_sharing: 1.20.0
  - files_trashbin: 1.18.0
  - files_versions: 1.21.0
  - firstrunwizard: 2.17.0
  - logreader: 2.13.0
  - lookup_server_connector: 1.16.0
  - mail: 3.5.7
  - maps: 1.3.1
  - nextcloud_announcements: 1.17.0
  - notes: 4.9.4
  - notifications: 2.16.0
  - oauth2: 1.16.3
  - password_policy: 1.18.0
  - photos: 2.4.0
  - privacy: 1.12.0
  - provisioning_api: 1.18.0
  - recommendations: 2.0.0
  - related_resources: 1.3.0
  - serverinfo: 1.18.0
  - settings: 1.10.1
  - sharebymail: 1.18.0
  - support: 1.11.1
  - survey_client: 1.16.0
  - systemtags: 1.18.0
  - tasks: 0.15.0
  - text: 3.9.1
  - theming: 2.3.0
  - twofactor_backupcodes: 1.17.0
  - twofactor_totp: 10.0.0-beta.2
  - updatenotification: 1.18.0
  - user_status: 1.8.1
  - viewer: 2.2.0
  - weather_status: 1.8.0
  - workflowengine: 2.10.0
Disabled:
  - bruteforcesettings: 2.8.0 (installed 1.1.0)
  - encryption: 2.16.0
  - files_rightclick: 0.15.1 (installed 1.6.0)
  - suspicious_login: 6.0.0
  - user_ldap: 1.19.0

Nextcloud Signing status

$ sudo -u www-data /usr/bin/php8.3 occ integrity:check-core -vvv                                                                                                                                                                                                                  
No errors found

Nextcloud Logs

{
  "reqId": "f3vl9SsZAHTLOuSpiQJl",
  "level": 3,
  "time": "2023-08-17T15:08:30+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "PHP",
  "method": "",
  "url": "--",
  "message": "Cannot declare class OC\\Command\\CommandJob, because the name is already in use at /srv/www/cloud.frelia.org/lib/private/Command/CommandJob.php#31",
  "userAgent": "--",
  "version": "27.0.1.2",
  "data": {
    "app": "PHP"
  }
}

Additional info

Based on the error above it appears I have had the issue since version 27 at least

So far I tried:

  1. Upgrading to PHP 8.2
  2. Upgrading to PHP 8.3
  3. Simply rebooting
  4. Restarting Redis
  5. Restarting PostgreSQL
  6. Restarting PHP-FPM
solracsf commented 5 months ago

Why is your configuration report outputing version 28.0.4.1 and you log 27.0.1.2 ?

Quintasan commented 5 months ago

Why is your configuration report outputing version 28.0.4.1 and you log 27.0.1.2 ?

Because I have a 90mb logfile running since at least one year. If you want me to upload the entire file then I can do so but it's full of the same messages.

Here's a fresh entry for this:

{
    "reqId": "353srMO2NBgDtEVrOoOQ",
    "level": 3,
    "time": "2024-03-29T08:40:27+00:00",
    "remoteAddr": "",
    "user": "--",
    "app": "PHP",
    "method": "",
    "url": "--",
    "message": "Cannot declare class OC\\Command\\CommandJob, because the name is already in use at /srv/www/cloud.frelia.org/lib/private/Command/CommandJob.php#31",
    "userAgent": "--",
    "version": "28.0.4.1",
    "data": {
        "app": "PHP"
    }
}
joshtrichards commented 5 months ago

Can you briefly enable log.backtrace and provide the stack trace associated with one of the errors?

https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#log-backtrace

Quintasan commented 5 months ago

@joshtrichards should this affect running via sudo -u www-data php /srv/www/cloud.frelia.org/cron.php? ~I added 'log.backtrace' => true, to config/config.php but I get no backtrace anyway~

EDIT

{
  "reqId": "g4qYPOX5Jldi3MClKOEy",
  "level": 3,
  "time": "2024-03-29T13:44:25+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "PHP",
  "method": "",
  "url": "--",
  "message": "Cannot declare class OC\\Command\\CommandJob, because the name is already in use at /srv/www/cloud.frelia.org/lib/private/Command/CommandJob.php#31",
  "userAgent": "--",
  "version": "28.0.4.1",
  "data": {
    "app": "PHP",
    "backtrace": [
      {
        "file": "/srv/www/cloud.frelia.org/lib/private/Log.php",
        "line": 139,
        "function": "log",
        "class": "OC\\Log",
        "type": "->"
      },
      {
        "file": "/srv/www/cloud.frelia.org/lib/private/Log/PsrLoggerAdapter.php",
        "line": 110,
        "function": "critical",
        "class": "OC\\Log",
        "type": "->"
      },
      {
        "file": "/srv/www/cloud.frelia.org/lib/private/Log/ErrorHandler.php",
        "line": 58,
        "function": "critical",
        "class": "OC\\Log\\PsrLoggerAdapter",
        "type": "->"
      },
      {
        "function": "onShutdown",
        "class": "OC\\Log\\ErrorHandler",
        "type": "->"
      }
    ]
  }
}

Doesn't look helpful though

joshtrichards commented 5 months ago

Yeah I was worried that might be the case since this is a PHP fatal error. sigh

I don't see how this is coming from our code base. We don't declare that class anywhere else and there haven't been other similar reports. (We'd no doubt have a ton fairly quickly after it appearing).

I don't see anything within the apps you have installed either.

A shot in the dark: Can you share the output of occ background-job:list?

Community Manual installation with Archive

What's the history of this installation? Has it been around awhile or is it fairly new? Also, how do you typically deploy updates?

joshtrichards commented 5 months ago

Also, what's the output of: grep CommandJob -r /srv/www/cloud.frelia.org --exclude-dir=data?

Quintasan commented 5 months ago

A shot in the dark: Can you share the output of occ background-job:list?

@joshtrichards as you would expect:

sudo -u www-data php occ background-job:list                                                                  255 ↵
PHP Fatal error:  Cannot declare class OC\Command\CommandJob, because the name is already in use in /srv/www/cloud.frelia.org/lib/private/Command/CommandJob.php on line 31

What's the history of this installation? Has it been around awhile or is it fairly new? Also, how do you typically deploy updates?

It has been around since... at least May 2018. At least that's what the timestamp on my data directory indicates. I usually deploy using the Web UI and before that I manually extracted tarballs.

Also, what's the output of: grep CommandJob -r /srv/www/cloud.frelia.org --exclude-dir=data?

╭─quintasan@frelia  /srv
╰─$ grep CommandJob -r /srv/www/cloud.frelia.org --exclude-dir=data
/srv/www/cloud.frelia.org/core/signature.json:        "lib\/private\/Command\/CommandJob.php": "78781dbfa98a7f9cf263a56118a721759ef81bfbfbd37a36972d5de003e85c2b500e02d26cf69442805c7a3fd14516de6dacbffe0dc00b7f80449118fb5f2412",
/srv/www/cloud.frelia.org/lib/composer/composer/autoload_static.php:        'OC\\Command\\CommandJob' => __DIR__ . '/../../..' . '/lib/private/Command/CommandJob.php',
/srv/www/cloud.frelia.org/lib/composer/composer/autoload_classmap.php:    'OC\\Command\\CommandJob' => $baseDir . '/lib/private/Command/CommandJob.php',
/srv/www/cloud.frelia.org/lib/private/Command/CommandJob.php:class CommandJob extends QueuedJob {
/srv/www/cloud.frelia.org/lib/private/Command/CronBus.php:                      return CommandJob::class;
joshtrichards commented 5 months ago

Any chance you're sharing your Redis server with a second Nextcloud instance?

Also can you see if changing your memcache.local to APCu makes any difference:

        "memcache.local": "\\OC\\Memcache\\APCu",

(I'm assuming this is a standalone server; in which case APCu is recommended for memcache.local anyhow).

Quintasan commented 5 months ago

Any chance you're sharing your Redis server with a second Nextcloud instance?

Nope. That's a single small VPS running a single Nextcloud instance.

Also can you see if changing your memcache.local to APCu makes any difference

Lemme do it.

Quintasan commented 5 months ago

Uh, I'm confused as hell right now:

# config.php
  'memcache.local' => '\OC\Memcache\APCu',
  'memcache.distributed' => '\OC\Memcache\Redis',
  'memcache.locking' => '\OC\Memcache\Redis',
  'redis' =>
  array (
    'host' => '/var/run/redis/redis-server.sock',
    'port' => 0,
  ),
╭─quintasan@frelia  /srv/www/cloud.frelia.org
╰─$ sudo -u www-data php /srv/www/cloud.frelia.org/cron.php
OCP\HintException: [0]: Memcache OC\Memcache\APCu not available for local cache (Is the matching PHP module installed and enabled?)

but php8.3-apcu is installed

EDIT

Once, again, I realized I'm a dummy so:

sudo -u www-data php --define apc.enable_cli=1 /srv/www/cloud.frelia.org/cron.php
PHP Fatal error:  Cannot declare class OC\Command\CommandJob, because the name is already in use in /srv/www/cloud.frelia.org/lib/private/Command/CommandJob.php on line 31

It appears we're back to square one

joshtrichards commented 5 months ago

Have you customized your PHP runtime configuration at all?

Share the output of:

php -v php -m php --ri "Zend OPcache"

Quintasan commented 5 months ago

@joshtrichards there you go

Details ``` ╭─quintasan@frelia ~ ╰─$ php -v PHP 8.3.4 (cli) (built: Mar 16 2024 08:40:08) (NTS) Copyright (c) The PHP Group Zend Engine v4.3.4, Copyright (c) Zend Technologies with Zend OPcache v8.3.4, Copyright (c), by Zend Technologies ╭─quintasan@frelia ~ ╰─$ php -m [PHP Modules] apcu bcmath calendar Core ctype curl date dom exif FFI fileinfo filter ftp gd gettext gmp hash iconv igbinary imagick intl json libxml mbstring mysqli mysqlnd openssl pcntl pcre PDO pdo_mysql pdo_pgsql pgsql Phar posix random readline redis Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer xml xmlreader xmlwriter xsl Zend OPcache zip zlib [Zend Modules] Zend OPcache ╭─quintasan@frelia ~ ╰─$ php --ri "Zend OPcache" Zend OPcache Opcode Caching => Up and Running Optimization => Enabled SHM Cache => Enabled File Cache => Disabled JIT => Disabled Startup => OK Shared memory model => mmap Cache hits => 0 Cache misses => 0 Used memory => 67888192 Free memory => 200547264 Wasted memory => 0 Interned Strings Used memory => 17303848 Interned Strings Free memory => 49805016 Cached scripts => 0 Cached keys => 0 Max keys => 16229 OOM restarts => 0 Hash keys restarts => 0 Manual restarts => 0 Start time => 2024-04-01T13:35:01+0000 Last restart time => none Last force restart time => none Directive => Local Value => Master Value opcache.enable => On => On opcache.use_cwd => On => On opcache.validate_timestamps => On => On opcache.validate_permission => Off => Off opcache.validate_root => Off => Off opcache.dups_fix => Off => Off opcache.revalidate_path => Off => Off opcache.log_verbosity_level => 1 => 1 opcache.memory_consumption => 256 => 256 opcache.interned_strings_buffer => 64 => 64 opcache.max_accelerated_files => 10000 => 10000 opcache.max_wasted_percentage => 5 => 5 opcache.force_restart_timeout => 180 => 180 opcache.revalidate_freq => 60 => 60 opcache.file_update_protection => 2 => 2 opcache.preferred_memory_model => no value => no value opcache.blacklist_filename => no value => no value opcache.max_file_size => 0 => 0 opcache.protect_memory => Off => Off opcache.save_comments => On => On opcache.record_warnings => Off => Off opcache.optimization_level => 0x7FFEBFFF => 0x7FFEBFFF opcache.opt_debug_level => 0 => 0 opcache.enable_file_override => Off => Off opcache.enable_cli => On => On opcache.error_log => no value => no value opcache.restrict_api => no value => no value opcache.lockfile_path => /tmp => /tmp opcache.file_cache => no value => no value opcache.file_cache_only => Off => Off opcache.file_cache_consistency_checks => On => On opcache.huge_code_pages => Off => Off opcache.preload => no value => no value opcache.preload_user => no value => no value opcache.jit => no value => no value opcache.jit_buffer_size => 128M => 128M opcache.jit_debug => 0 => 0 opcache.jit_bisect_limit => 0 => 0 opcache.jit_prof_threshold => 0.005 => 0.005 opcache.jit_max_root_traces => 1024 => 1024 opcache.jit_max_side_traces => 128 => 128 opcache.jit_max_exit_counters => 8192 => 8192 opcache.jit_hot_loop => 64 => 64 opcache.jit_hot_func => 127 => 127 opcache.jit_hot_return => 8 => 8 opcache.jit_hot_side_exit => 8 => 8 opcache.jit_blacklist_root_trace => 16 => 16 opcache.jit_blacklist_side_trace => 8 => 8 opcache.jit_max_loop_unrolls => 8 => 8 opcache.jit_max_recursive_calls => 2 => 2 opcache.jit_max_recursive_returns => 2 => 2 opcache.jit_max_polymorphic_calls => 2 => 2 opcache.jit_max_trace_length => 1024 => 1024 ```
joshtrichards commented 5 months ago

I see you have the extension igbinary installed. Are you using it for something? Is APCu maybe configured to use it?

php --ri apcu

Quintasan commented 5 months ago

@joshtrichards

╭─quintasan@frelia  ~
╰─$ php --ri apcu

apcu

APCu Support => Enabled
Version => 5.1.23
APCu Debugging => Disabled
MMAP Support => Enabled
MMAP File Mask =>
Serialization Support => php, igbinary
Build Date => Nov 25 2023 13:05:34

Directive => Local Value => Master Value
apc.enabled => On => On
apc.shm_segments => 1 => 1
apc.shm_size => 32M => 32M
apc.entries_hint => 4096 => 4096
apc.gc_ttl => 3600 => 3600
apc.ttl => 0 => 0
apc.smart => 0 => 0
apc.mmap_file_mask => no value => no value
apc.enable_cli => On => On
apc.slam_defense => Off => Off
apc.preload_path => no value => no value
apc.coredump_unmap => Off => Off
apc.use_request_time => Off => Off
apc.serializer => php => php
joshtrichards commented 1 month ago

I'm not sure why why you're experiencing this, but it seems to be something specific to your environment. There haven't been any similar reports. There is no evidence to suggest it's a bug in Nextcloud Server itself.

The error itself is generally a fairly straightforward condition to fix (from a development perspective, if it were a bug on this end), but the underlying cause in your situation appears to be unusual. It's possible it's even a bug in PHP. Given you've tried multiple PHP versions though, I'm skeptical of that. Something else about your environment though...

If you're motivated/able, you might try bringing up a fresh OS/hosting environment (with few changes beyond stock) and migrating your existing Nextcloud installation to it to see if the error returns. Precisely what OS/distribution (and version) is this anyhow?

You might try posting at the community help forum - https://help.nextcloud.com - see if some additional eyes turn up something about your environment.

I'm afraid there's not much else we can do from this end, since there is nothing to suggest it's a bug over here.

P.S. PHP recently improved the error reporting for this message, to indicate the source: php/php-src/commit/1c30c5e707c1c289676734df6f1056ac20f4479c (which would be very informative in your particular situation), but unfortunately that's only for the upcoming 8.4 AFAIK.

nextcloud-command commented 1 week ago

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.