File entry could not be inserted but could also not be selected with getId()

cartman29 commented 2 years ago

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

Steps to reproduce

1 - sudo -u USER1 php /var/www/nextcloud/occ files:scan USER 2 - Full scan ok 3 - Always same Error at end of scan

Expected behaviour

Do not Error at end of scan

Actual behaviour

Tell us what happens instead

Server configuration

Operating system: Ubuntu 20.04

Web server: Nginx 1.18

Database: psql 13.5

PHP version: 8.0.13

Nextcloud version: (see Nextcloud admin page) 22.2.3

Updated from an older Nextcloud:

Where did you install Nextcloud from:

Signing status:

Signing status

``` ```

List of activated apps:

App list

``` ```

Nextcloud configuration:

Config report

``` '************', 'passwordsalt' => '*********', 'secret' => '*****************', 'default_phone_region' => 'FR', 'trusted_domains' => array ( 0 => 'MYWEBSERVER', 1 => 'MYWEBSERVER.local', ), 'datadirectory' => 'SOMEWHERE', 'dbtype' => 'pgsql', 'version' => '22.2.3.0', 'overwrite.cli.url' => 'https://MYWEBSERVER', 'dbname' => 'NameDb', 'dbhost' => 'localhost', 'dbport' => '', 'dbtableprefix' => 'oc_', 'dbuser' => 'UserDb', 'dbpassword' => 'PassDb', 'installed' => true, 'filelocking.enabled' => 'true', 'memcache.local' => '\\OC\\Memcache\\APCu', 'memcache.locking' => '\\OC\\Memcache\\Redis', 'redis' => array ( 'host' => '/var/run/redis/redis.sock', 'port' => '0', ), 'memcache.distributed' => '\\OC\\Memcache\\Redis', 'app_install_overwrite' => array ( 0 => 'previewgenerator', ), 'preview_max_x' => '2048', 'preview_max_y' => '2048', 'jpeg_quality' => '60', 'filesystem_check_changes' => 0, 'maintenance' => false, 'theme' => '', 'loglevel' => 1, 'log_type' => 'file', 'logfile' => '/var/log/nextcloud/nextcloud.log', 'logtimezone' => 'Europe/Paris', 'log_rotate_size' => '104857600' ); ```

Are you using external storage, if yes which one: local

Are you using encryption: no

Are you using an external user-backend, if yes which one: don't know

Client configuration

Browser:

Operating system:

Logs

Web server error log

``` https://i.postimg.cc/3JKXFqsD/error.jpg ```

Nextcloud log (data/nextcloud.log)

Nextcloud log

``` {"reqId":"YkmKaFQJhcwlNWtJAOWc","level":3,"time":"2021-12-06T09:10:02+01:00","remoteAddr":"","user":"--","app":"files","method":"","url":"--","message":"File entry could not be inserted but could also not be selected with getId() in order to perform an update. Please try again.","userAgent":"--","version":"22.2.3.0","exception":{"Exception":"RuntimeException","Message":"File entry could not be inserted but could also not be selected with getId() in order to perform an update. Please try again.","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Files/Cache/Wrapper/CacheWrapper.php","line":135,"function":"insert","class":"OC\\Files\\Cache\\Cache","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":294,"function":"insert","class":"OC\\Files\\Cache\\Wrapper\\CacheWrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":224,"function":"addToCache","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":427,"function":"scanFile","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":388,"function":"handleChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":340,"function":"scanChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":510,"function":"scan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":521,"function":"OC\\Files\\Cache\\{closure}","class":"OC\\Files\\Cache\\Scanner","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":509,"function":"runBackgroundScanJob","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Utils/Scanner.php","line":185,"function":"backgroundScan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":89,"function":"backgroundScan","class":"OC\\Files\\Utils\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":125,"function":"runScanner","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/Job.php","line":51,"function":"run","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/TimedJob.php","line":58,"function":"execute","class":"OC\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":127,"function":"execute","class":"OC\\BackgroundJob\\TimedJob","type":"->"}],"File":"/var/www/nextcloud/lib/private/Files/Cache/Cache.php","Line":340,"CustomMessage":"--"},"id":"61add307359bd"} ```

Browser log

``` RuntimeException: File entry could not be inserted but could also not be selected with getId() in order to perform an update. Please try again. ```

PostgresSQL log

PostGreSQL

``` 2021-12-07 09:10:02.564 CET [39333] XXXXXX@YYYYYY ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « fs_storage_path_hash » 2021-12-07 09:10:02.564 CET [39333] XXXXXX@YYYYYY DÉTAIL: La clé « (storage, path_hash)=(3, 13c337fec*************1cf86af2c) » existe déjà. 2021-12-07 09:10:02.564 CET [39333] XXXXXX@YYYYYY INSTRUCTION : INSERT INTO "oc_filecache" ("mimepart", "mimetype", "mtime", "size", "etag", "storage_mtime", "permissions", "name", "parent", "checksum", "path_hash", "path", "storage") VALUES($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13) ```

##################

I have see the same error here : https://github.com/nextcloud/server/issues/22836

or here : https://github.com/nextcloud/server/issues/5448 (Integrity constraint violation: Duplicate entry)

or here : https://github.com/nextcloud/server/issues/19494

or here : https://github.com/nextcloud/server/issues/12369

But no answer at all. Can I have help please ?

solracsf commented 2 years ago

Seems tied to:

https://github.com/nextcloud/server/blob/842043180611b1e45898acdf29bd4162adf23434/lib/private/Files/Cache/Cache.php#L335-L341

I can also observe this with random actions (NC 22.2.3, MariaDB 10.4, PHP 8.0.13), here is a stack:

{
  "reqId": "k3ECPEoGdxdahL8ywyVb",
  "level": 3,
  "time": "2021-12-09T08:36:51+00:00",
  "remoteAddr": "91.168.179.154",
  "user": "ale.dal@ali.fr",
  "app": "no app in context",
  "method": "MOVE",
  "url": "/remote.php/dav/files/ale.dal@ali.fr/ONE/FOLDER",
  "message": "File entry could not be inserted but could also not be selected with getId() in order to perform an update. Please try again.",
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36",
  "version": "22.2.3.0",
  "exception": {
    "Exception": "RuntimeException",
    "Message": "File entry could not be inserted but could also not be selected with getId() in order to perform an update. Please try again.",
    "Code": 0,
    "Trace": [
      {
        "file": "/lib/private/Files/Cache/Scanner.php",
        "line": 294,
        "function": "insert",
        "class": "OC\\Files\\Cache\\Cache",
        "type": "->",
        "args": [
          "files/AFFAIRES/ARCHIVES/ONE",
          {
            "0": "And 5 more entries, set log level to debug to see all entries",
            "mimetype": "httpd/unix-directory",
            "mtime": 1635324600,
            "size": -1,
            "etag": "61b1c023df942",
            "storage_mtime": 1635324600
          }
        ]
      },
      {
        "file": "/lib/private/Files/Cache/Scanner.php",
        "line": 224,
        "function": "addToCache",
        "class": "OC\\Files\\Cache\\Scanner",
        "type": "->",
        "args": [
          "files/AFFAIRES/ARCHIVES/ONE",
          {
            "0": "And 4 more entries, set log level to debug to see all entries",
            "mimetype": "httpd/unix-directory",
            "mtime": 1635324600,
            "size": -1,
            "etag": "61b1c023df942",
            "storage_mtime": 1635324600
          },
          -1
        ]
      },
      {
        "file": "/lib/private/Files/Cache/Scanner.php",
        "line": 338,
        "function": "scanFile",
        "class": "OC\\Files\\Cache\\Scanner",
        "type": "->",
        "args": [
          "files/AFFAIRES/ARCHIVES/ONE",
          3,
          54576128,
          false,
          true
        ]
      },
      {
        "file": "/lib/private/Files/View.php",
        "line": 1349,
        "function": "scan",
        "class": "OC\\Files\\Cache\\Scanner",
        "type": "->",
        "args": [
          "files/AFFAIRES/ARCHIVES/ONE",
          false
        ]
      },
      {
        "file": "/lib/private/Files/View.php",
        "line": 1393,
        "function": "getCacheEntry",
        "class": "OC\\Files\\View",
        "type": "->",
        "args": [
          {
            "cache": null,
            "scanner": {
              "__class__": "OC\\Files\\Cache\\Scanner"
            },
            "watcher": null,
            "propagator": null,
            "updater": {
              "__class__": "OC\\Files\\Cache\\Updater"
            },
            "__class__": "OCA\\Files_Trashbin\\Storage"
          },
          "files/AFFAIRES/ARCHIVES/ONE",
          "/AFFAIRES/ARCHIVES/ONE"
        ]
      },
      {
        "file": "/lib/private/Files/View.php",
        "line": 1696,
        "function": "getFileInfo",
        "class": "OC\\Files\\View",
        "type": "->",
        "args": [
          "/ale.dal@ali.fr/files/AFFAIRES/ARCHIVES/ONE"
        ]
      },
      {
        "file": "/apps/activity/lib/FilesHooks.php",
        "line": 668,
        "function": "getOwner",
        "class": "OC\\Files\\View",
        "type": "->",
        "args": [
          "/AFFAIRES/ARCHIVES/ONE"
        ]
      },
      {
        "file": "/apps/activity/lib/FilesHooks.php",
        "line": 463,
        "function": "getSourcePathAndOwner",
        "class": "OCA\\Activity\\FilesHooks",
        "type": "->",
        "args": [
          "/AFFAIRES/ARCHIVES/ONE"
        ]
      },
      {
        "file": "/apps/activity/lib/FilesHooks.php",
        "line": 386,
        "function": "fileMoving",
        "class": "OCA\\Activity\\FilesHooks",
        "type": "->",
        "args": [
          "/ONE/FOLDER",
          "/AFFAIRES/ARCHIVES/ONE"
        ]
      },
      {
        "file": "/apps/activity/lib/FilesHooksStatic.php",
        "line": 79,
        "function": "fileMovePost",
        "class": "OCA\\Activity\\FilesHooks",
        "type": "->",
        "args": [
          "/ONE/FOLDER",
          "/AFFAIRES/ARCHIVES/ONE"
        ]
      },
      {
        "file": "/lib/private/legacy/OC_Hook.php",
        "line": 106,
        "function": "fileMovePost",
        "class": "OCA\\Activity\\FilesHooksStatic",
        "type": "::",
        "args": [
          {
            "oldpath": "/ONE/FOLDER",
            "newpath": "/AFFAIRES/ARCHIVES/ONE"
          }
        ]
      },
      {
        "file": "/lib/private/Files/View.php",
        "line": 861,
        "function": "emit",
        "class": "OC_Hook",
        "type": "::",
        "args": [
          "OC_Filesystem",
          "post_rename",
          {
            "oldpath": "/ONE/FOLDER",
            "newpath": "/AFFAIRES/ARCHIVES/ONE"
          }
        ]
      },
      {
        "file": "/apps/dav/lib/Connector/Sabre/Directory.php",
        "line": 442,
        "function": "rename",
        "class": "OC\\Files\\View",
        "type": "->",
        "args": [
          "/ONE/FOLDER",
          "/AFFAIRES/ARCHIVES/ONE"
        ]
      },
      {
        "file": "/3rdparty/sabre/dav/lib/DAV/Tree.php",
        "line": 160,
        "function": "moveInto",
        "class": "OCA\\DAV\\Connector\\Sabre\\Directory",
        "type": "->",
        "args": [
          "ONE",
          "files/ale.dal@ali.fr/ONE/FOLDER",
          {
            "__class__": "OCA\\DAV\\Connector\\Sabre\\Directory"
          }
        ]
      },
      {
        "file": "/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",
        "line": 612,
        "function": "move",
        "class": "Sabre\\DAV\\Tree",
        "type": "->",
        "args": [
          "files/ale.dal@ali.fr/ONE/FOLDER",
          "files/ale.dal@ali.fr/AFFAIRES/ARCHIVES/ONE"
        ]
      },
      {
        "file": "/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
        "line": 89,
        "function": "httpMove",
        "class": "Sabre\\DAV\\CorePlugin",
        "type": "->",
        "args": [
          {
            "__class__": "Sabre\\HTTP\\Request"
          },
          {
            "__class__": "Sabre\\HTTP\\Response"
          }
        ]
      },
      {
        "file": "/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 472,
        "function": "emit",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": [
          "method:MOVE",
          [
            {
              "__class__": "Sabre\\HTTP\\Request"
            },
            {
              "__class__": "Sabre\\HTTP\\Response"
            }
          ]
        ]
      },
      {
        "file": "/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 253,
        "function": "invokeMethod",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": [
          {
            "__class__": "Sabre\\HTTP\\Request"
          },
          {
            "__class__": "Sabre\\HTTP\\Response"
          }
        ]
      },
      {
        "file": "/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 321,
        "function": "start",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": []
      },
      {
        "file": "/apps/dav/lib/Server.php",
        "line": 333,
        "function": "exec",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": []
      },
      {
        "file": "/apps/dav/appinfo/v2/remote.php",
        "line": 35,
        "function": "exec",
        "class": "OCA\\DAV\\Server",
        "type": "->",
        "args": []
      },
      {
        "file": "/remote.php",
        "line": 166,
        "args": [
          "/apps/dav/appinfo/v2/remote.php"
        ],
        "function": "require_once"
      }
    ],
    "File": "/lib/private/Files/Cache/Cache.php",
    "Line": 340,
    "CustomMessage": "--"
  },
  "id": "61b351a67d87a"
}

cartman29 commented 2 years ago

maybe it's unique to postgresSQL ?

solracsf commented 2 years ago

@cartman29 No, I'm using MariaDB.

cartman29 commented 2 years ago

@acsfer same log from MariaDb ?

la valeur d'une clé dupliquée rompt la contrainte unique « fs_storage_path_hash » (duplicate key violates unique constraint « fs_storage_path_hash ») ?

For me it's on /var/log/postgresql/postgresql-13-main.log

solracsf commented 2 years ago

@cartman29 nothing like that on MariaDB. There are other issues mentioning that DK, maybe they can help you.

szaimen commented 1 year ago

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

nextcloud / server