Goldinteractive / craft3-sitecopy

With this plugin you can copy the content of an entry to another site.
MIT License
22 stars 6 forks source link

SiteCopy breaks Queue on large multi-site #34

Closed sandrodunkel closed 1 year ago

sandrodunkel commented 1 year ago

I have a craft instance with 20+ sites for multilingual/multinational use. To distribute content across these sites we use Site Copy. Since using it, the queue often has failed jobs, blocks out the control panel with an internal server error and backs up the queue to up to 800k jobs. We are 95% sure that the problem is Site Copy since we only have this problem since using Site Copy. Plus, most of of the stack traces clearly hint towards sitecopy being the trigger.

Stack trace example: This hints to a duplicated entry id and a duplicated slug. Is it possible that site copy somehow mixed up entry ids? I can't find duplicated entry ids or slugs in my control panel or my database.

2023-03-07 06:16:48 [queue.ERROR] [craft\queue\QueueLogBehavior::afterError]  [1455257] Syncing element contents (attempt: 1, pid: 32704) - Error (time: 8.961s): SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '716774-107' for key 'revisions_sourceId_num_unq_idx'
The SQL being executed was: INSERT INTO `revisions` (`canonicalId`, `creatorId`, `num`, `notes`) VALUES (716774, 5002, 107, NULL) {"memory":12716912} 
2023-03-07 06:16:48 [queue.ERROR] [yii\db\IntegrityException] PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '716774-107' for key 'revisions_sourceId_num_unq_idx' in /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/db/Command.php:1302
Stack trace:
#0 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/db/Command.php(1302): PDOStatement->execute()
#1 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/db/Command.php(1102): yii\db\Command->internalExecute('INSERT INTO `re...')
#2 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/helpers/Db.php(953): yii\db\Command->execute()
#3 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/services/Revisions.php(145): craft\helpers\Db::insert('{{%revisions}}', Array)
#4 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/elements/Entry.php(2007): craft\services\Revisions->createRevision(Object(craft\elements\Entry), 5002, NULL)
#5 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/services/Elements.php(3040): craft\elements\Entry->afterPropagate(false)
#6 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/services/Elements.php(1039): craft\services\Elements->_saveElementInternal(Object(craft\elements\Entry), true, true, true)
#7 /home/httpd/vhosts/domain/root/vendor/goldinteractive/craft-sitecopy/src/jobs/SyncElementContent.php(103): craft\services\Elements->saveElement(Object(craft\elements\Entry))
#8 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2-queue/src/Queue.php(246): goldinteractive\sitecopy\jobs\SyncElementContent->execute(Object(craft\queue\Queue))
#9 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2-queue/src/cli/Queue.php(147): yii\queue\Queue->handleMessage(1455257, 'O:48:"goldinter...', 300, 1)
#10 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/queue/Queue.php(183): yii\queue\cli\Queue->handleMessage(1455257, 'O:48:"goldinter...', 300, 1)
#11 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/queue/Queue.php(158): craft\queue\Queue->executeJob()
#12 [internal function]: craft\queue\Queue->craft\queue\{closure}(Object(Closure))
#13 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2-queue/src/cli/Queue.php(117): call_user_func(Object(Closure), Object(Closure))
#14 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/queue/Queue.php(166): yii\queue\cli\Queue->runWorker(Object(Closure))
#15 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/controllers/QueueController.php(82): craft\queue\Queue->run()
#16 [internal function]: craft\controllers\QueueController->actionRun()
#17 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#18 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams(Array)
#19 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/base/Module.php(552): yii\base\Controller->runAction('run', Array)
#20 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/web/Application.php(301): yii\base\Module->runAction('queue/run', Array)
#21 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/web/Application.php(625): craft\web\Application->runAction('queue/run', Array)
#22 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/web/Application.php(280): craft\web\Application->_processActionRequest(Object(craft\web\Request))
#23 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/base/Application.php(384): craft\web\Application->handleRequest(Object(craft\web\Request))
#24 /home/httpd/vhosts/domain/root/web/index.php(21): yii\base\Application->run()
#25 {main}

Next yii\db\IntegrityException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '716774-107' for key 'revisions_sourceId_num_unq_idx'
The SQL being executed was: INSERT INTO `revisions` (`canonicalId`, `creatorId`, `num`, `notes`) VALUES (716774, 5002, 107, NULL) in /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/db/Schema.php:676
Stack trace:
#0 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/db/Command.php(1307): yii\db\Schema->convertException(Object(PDOException), 'INSERT INTO `re...')
#1 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/db/Command.php(1102): yii\db\Command->internalExecute('INSERT INTO `re...')
#2 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/helpers/Db.php(953): yii\db\Command->execute()
#3 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/services/Revisions.php(145): craft\helpers\Db::insert('{{%revisions}}', Array)
#4 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/elements/Entry.php(2007): craft\services\Revisions->createRevision(Object(craft\elements\Entry), 5002, NULL)
#5 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/services/Elements.php(3040): craft\elements\Entry->afterPropagate(false)
#6 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/services/Elements.php(1039): craft\services\Elements->_saveElementInternal(Object(craft\elements\Entry), true, true, true)
#7 /home/httpd/vhosts/domain/root/vendor/goldinteractive/craft-sitecopy/src/jobs/SyncElementContent.php(103): craft\services\Elements->saveElement(Object(craft\elements\Entry))
#8 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2-queue/src/Queue.php(246): goldinteractive\sitecopy\jobs\SyncElementContent->execute(Object(craft\queue\Queue))
#9 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2-queue/src/cli/Queue.php(147): yii\queue\Queue->handleMessage(1455257, 'O:48:"goldinter...', 300, 1)
#10 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/queue/Queue.php(183): yii\queue\cli\Queue->handleMessage(1455257, 'O:48:"goldinter...', 300, 1)
#11 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/queue/Queue.php(158): craft\queue\Queue->executeJob()
#12 [internal function]: craft\queue\Queue->craft\queue\{closure}(Object(Closure))
#13 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2-queue/src/cli/Queue.php(117): call_user_func(Object(Closure), Object(Closure))
#14 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/queue/Queue.php(166): yii\queue\cli\Queue->runWorker(Object(Closure))
#15 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/controllers/QueueController.php(82): craft\queue\Queue->run()
#16 [internal function]: craft\controllers\QueueController->actionRun()
#17 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#18 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams(Array)
#19 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/base/Module.php(552): yii\base\Controller->runAction('run', Array)
#20 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/web/Application.php(301): yii\base\Module->runAction('queue/run', Array)
#21 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/web/Application.php(625): craft\web\Application->runAction('queue/run', Array)
#22 /home/httpd/vhosts/domain/root/vendor/craftcms/cms/src/web/Application.php(280): craft\web\Application->_processActionRequest(Object(craft\web\Request))
#23 /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/base/Application.php(384): craft\web\Application->handleRequest(Object(craft\web\Request))
#24 /home/httpd/vhosts/domain/root/web/index.php(21): yii\base\Application->run()
#25 {main}
Additional Information:
Array
(
    [0] => 23000
    [1] => 1062
    [2] => Duplicate entry '716774-107' for key 'revisions_sourceId_num_unq_idx'
)
 {"memory":12717288,"exception":"[object] (yii\\db\\IntegrityException(code: 23000): SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '716774-107' for key 'revisions_sourceId_num_unq_idx'
The SQL being executed was: INSERT INTO `revisions` (`canonicalId`, `creatorId`, `num`, `notes`) VALUES (716774, 5002, 107, NULL) at /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/db/Schema.php:676)
[previous exception] [object] (PDOException(code: 23000): SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '716774-107' for key 'revisions_sourceId_num_unq_idx' at /home/httpd/vhosts/domain/root/vendor/yiisoft/yii2/db/Command.php:1302)"} 
lukasNo1 commented 1 year ago

This is a craft problem, probably related to https://github.com/craftcms/cms/issues/11220

I had this "duplicate entry" error before too when I was editing an entry while a queue job was saving an element for another plugin.

What craft version are you running? Craft made a fix related to that issue in 3.7.43

sandrodunkel commented 1 year ago

I'm using Craft 4.0.3. As you said, the error discussed in your linked issue already got fixed in 3.7.43.

Let me make sure I understand what you say. SiteCopy uses the queue to process all the copying. If I start the queue job of copying entry.id 123 and then—whilst SiteCopy is handling the copy job in the queue—I edit the very same entry, then I trigger this error since the entry is basically being used twice? This would explain another error we sometimes used to get, which said that the mutex lock for the queue failed.

My project.yaml:

{
  "require": {
    "carlcs/craft-redactorcustomstyles": "4.0.3",
    "craftcms/cms": "4.3.10",
    "craftcms/feed-me": "5.0.5",
    "craftcms/redactor": "3.0.3",
    "goldinteractive/craft-sitecopy": "1.0.4",
    "nystudio107/craft-cookies": "4.0.0",
    "nystudio107/craft-seomatic": "4.0.20",
    "putyourlightson/craft-blitz": "4.4.0",
    "putyourlightson/craft-dashboard-begone": "2.0.0",
    "putyourlightson/craft-elements-panel": "2.0.0",
    "putyourlightson/craft-sprig": "2.5.1",
    "vaersaagod/geomate": "v2.1.0",
    "vaersaagod/matrixmate": "2.1.2",
    "vlucas/phpdotenv": "^3.4.0"
  },
  "autoload": {
    "psr-4": {
      "modules\\": "modules/"
    }
  },
  "config": {
    "sort-packages": true,
    "optimize-autoloader": true,
    "platform": {
      "php": "8.0.2"
    },
    "allow-plugins": {
      "craftcms/plugin-installer": true,
      "yiisoft/yii2-composer": true
    }
  },
  "scripts": {
    "post-root-package-install": [
      "@php -r \"file_exists('.env') || copy('.env.example', '.env');\""
    ]
  }
}
lukasNo1 commented 1 year ago

Thats correct yes. Under the hood, sitecopy calls the standard saveElement() function from craft for each site you select it to copy to. If you edit the entry at the same time this can cause problems.

A Mutex lock is a bit different, that error just says that the job wants to write something to the db, but another process is already writing something (has locked the db). This can happen if multiple queue jobs are running at the same time. I also had it happen some times when multiple users were editing entries together.

sandrodunkel commented 1 year ago

Alrighty, thanks for your quick help and your explanations!