search

farcasterxyz / hub-monorepo

Implementation of the Farcaster Hub specification and supporting libraries for building applications on Farcaster
https://www.thehubble.xyz
MIT License
700 stars 392 forks source link

bug(replicator): Replicator stops updating with many "foreign key constraint" errors #1556

Closed cameronvoell closed 3 months ago

cameronvoell commented 11 months ago

What is the bug?

Replicator stopped updating last night (querying db missing data for last 15hours or so). Checking on the server the replicator process was logging many errors: error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign"

How can it be reproduced? (optional)

Hub and Replicator both launched using the install scripts on the same Digital Ocean ubuntu 22.04 server about 3 days ago, so should have been version Hub Release 1.7.0 for both.

Hubble Dashboard wasn't showing anything unusual. Replicator redis WebUI (script defaults to port 9000) was showing ~300+ delayed jobs that all showed the same "foreign key constraint" error.

Not an urgent blocker as I was able to get it syncing again. => To attempt to clear up the issue I clicked "clean all" in the Delayed section of the Replicator Web UI. This didnt seem to fix it. I then brought hubble and replicator down via their scripts, and then brought them back up and at that point the replicator started syncing and appears to up to date again. Though the Replicator Web UI does still show ~40 of the "foreign key constraint" errors in the "Delayed" column.

Additional context (optional)

See Warpcast thread of myself(@cyrcus) and @haole discussing as we both ran into the error => https://warpcast.com/haole/0x89b9f44b

Screenshot of Replicator Redis UI:

image

See delayed job error logs below:

ProcessHubEvent Error Data: ```json { "data": { "hubEventJsonStr": "{\"type\":\"HUB_EVENT_TYPE_MERGE_MESSAGE\",\"id\":366630771179520,\"mergeMessageBody\":{\"message\":{\"data\":{\"type\":\"MESSAGE_TYPE_REACTION_ADD\",\"fid\":680,\"timestamp\":89486665,\"network\":\"FARCASTER_NETWORK_MAINNET\",\"reactionBody\":{\"type\":\"REACTION_TYPE_LIKE\",\"targetCastId\":{\"fid\":4163,\"hash\":\"VC+AAXFsvIxZVNzuq8l2vEIk6uE=\"}}},\"hash\":\"rQn98rxbgWI8JTK0fw+WvKoZLUE=\",\"hashScheme\":\"HASH_SCHEME_BLAKE3\",\"signature\":\"1yIVNQPNrgg4WEP+s6tvkXS3od8Aur80PngXZk7H4SdCWemSlKFo4SmFZoTJZe5JeUaAHMMHkhkFBRtC77D2Cg==\",\"signatureScheme\":\"SIGNATURE_SCHEME_ED25519\",\"signer\":\"sLPssdYagY3ui5gxFBDrfUr0nO3/Vay7mTrSXeESM4Q=\"},\"deletedMessages\":[]}}" }, "returnValue": null } ``` Error: ``` error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:40:40.208Z', '2023-11-02T23:40:40.208Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:40:40.208Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:40:41.278Z', '2023-11-02T23:40:41.278Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:40:41.278Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:40:43.399Z', '2023-11-02T23:40:43.399Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:40:43.399Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:40:47.447Z', '2023-11-02T23:40:47.447Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:40:47.447Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:40:55.512Z', '2023-11-02T23:40:55.512Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:40:55.512Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:41:11.563Z', '2023-11-02T23:41:11.563Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:41:11.563Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:41:43.622Z', '2023-11-02T23:41:43.622Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:41:43.622Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:42:47.667Z', '2023-11-02T23:42:47.667Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:42:47.667Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:44:55.708Z', '2023-11-02T23:44:55.708Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:44:55.708Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:49:11.983Z', '2023-11-02T23:49:11.983Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:49:11.983Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign" at /home/node/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:72:28) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at PostgresConnection.executeQuery (/home/node/app/node_modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:91:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/node/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:37:28 at async #run (/home/node/app/node_modules/kysely/dist/cjs/driver/single-connection-provider.js:27:16) at execute (/home/node/app/apps/replicator/build/db.js:129:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async storeMessage (/home/node/app/apps/replicator/build/processors/index.js:197:5) at async processMessage (/home/node/app/apps/replicator/build/processors/index.js:70:5) at async mergeMessage (/home/node/app/apps/replicator/build/processors/index.js:52:5) at async /home/node/app/apps/replicator/build/processors/index.js:289:9 at async /home/node/app/apps/replicator/build/db.js:156:20 at async /home/node/app/node_modules/kysely/dist/cjs/kysely.js:418:32 at async DefaultConnectionProvider.provideConnection (/home/node/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:12:20) at async executeTx (/home/node/app/apps/replicator/build/db.js:155:16) insert into "messages" ( "created_at", "updated_at", "fid", "type", "timestamp", "hash", "hash_scheme", "signature", "signature_scheme", "signer", "raw", "deleted_at", "pruned_at", "revoked_at", "body" ) values ( '2023-11-02T23:57:44.049Z', '2023-11-02T23:57:44.049Z', 680, 1, '2023-11-02T19:34:53.000Z', '\x8f85ffd908b9dc8eb5b13751761f03ae858e257f', 1, '\x826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a02', 1, '\xb0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', '\x0a2d080110a80518dda7d62a20012a1f12001a1908bf01121428368a8e258bb9062dbea5009d1ea97cdbf04e522a0012148f85ffd908b9dc8eb5b13751761f03ae858e257f18012240826f27434b70f2a5e45c9a8eceee5fbe19f7b8ce797bc83604a759373c4bded5987fbddaa3ba6141a3b6c5a1909de1f6f5acd9082b1b85eb39f2451a1b017a0228013220b0b3ecb1d61a818dee8b98311410eb7d4af49cedff55acbb993ad25de1123384', null, null, null, '{"embeds":[],"mentions":[],"mentionsPositions":[],"text":"","parentCastId":{"fid":191,"hash":"0x28368a8e258bb9062dbea5009d1ea97cdbf04e52"}}' ) on conflict ("hash") do update set "updated_at" = '2023-11-02T23:57:44.049Z', "signature" = "excluded"."signature", "signature_scheme" = "excluded"."signature_scheme", "signer" = "excluded"."signer", "deleted_at" = null, "pruned_at" = null, "revoked_at" = null where ( "excluded"."signature" != "messages"."signature" or "excluded"."signature_scheme" != "messages"."signature_scheme" or "excluded"."signer" != "messages"."signer" or "excluded"."deleted_at" is distinct from "messages"."deleted_at" or "excluded"."pruned_at" is distinct from "messages"."pruned_at" or "excluded"."revoked_at" is distinct from "messages"."revoked_at" ) returning "hash", "updated_at", "created_at" at executeTx (/home/node/app/apps/replicator/build/db.js:161:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async processMergeMessageHubEvent (/home/node/app/apps/replicator/build/processors/index.js:288:5) at async processHubEvent (/home/node/app/apps/replicator/build/processors/index.js:247:13) at async Object.run (/home/node/app/apps/replicator/build/jobs/processHubEvent.js:11:9) at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33 ```
ethisscam commented 11 months ago

You can remove the foreign key constraint from the messages table and check whether it starts syncing. ALTER TABLE messages DROP CONSTRAINT messages_signer_fid_foreign;

christopherwxyz commented 11 months ago

+1, i'm also having this same issue.

sanjayprabhu commented 11 months ago

This is likely due to the replicator missing certain hub events. https://github.com/farcasterxyz/hub-monorepo/pull/1553 should reduce the likelihood of this happening, but real fix is to retry the missing on chain event if this is detected. Will fix.

Bijan-Massoumi commented 11 months ago

+1, i'm also having this same issue. Any workaround you recommend until this gets fixed?

I've tried pinning both my replicator and hub to 1.6.6 and syncing a fresh DB blasts my console output with these types of Errors

edit 1: I flushed my local redis and restarted and now it seems to be syncing. This bug could be caused by the fact that i switched DBs and hub instances without clearing my redis?

edit 2 ok now im having the same issue as OP

Bijan-Massoumi commented 8 months ago

any update on this? @sanjayprabhu

sanjayprabhu commented 8 months ago

@Bijan-Massoumi Are you still seeing this issue after upgrading to the latest version?

obo20 commented 7 months ago

@sanjayprabhu We're still seeing this issue (heavily) as of yesterday. Running hubble 1.9.8.

namanjain commented 7 months ago

@sanjayprabhu I'm also seeing this error pretty heavily at this point. Backfill is pretty much stuck at Backfilled events for 333607 of 333666 FIDs and the dashboard show 226 delayed jobs. The delayed jobs don't clear up after promoting and go back to delayed, which I assume is because of the foreign key errors.

obo20 commented 7 months ago

An update on the pinata side, we did a fresh start of a new hub / replicator and immediately removed the foreign key constraints. From what we're seeing:

@sanjayprabhu do you see any issues with this approach?

sds commented 6 months ago

Hey all, wanted to explain what we currently believe is happening here.

A core assumption we made with the Replicator's design was that it should not process messages which refer to other messages/data that are missing. Hubs don't strictly follow this rule—a message can be valid even if it refers to another message that doesn't yet exist on the hub.

Two simple examples of this are:

Similar situations apply for reactions, links, etc. Behavior is documented here, but we will call this out more clearly in the README: https://github.com/farcasterxyz/hub-monorepo/blob/0662959e95b27b92f9315e58c5a26f6f668f8371/apps/replicator/src/migrations/001_initial_migration.ts#L38-L43

The reason enforcing foreign key constraints are helpful in this context is that it can help find legitimate bugs (whether its due to missing FIDs from a lost event or something else)! However, for some purposes disabling them can be helpful in the short term, but I would not consider it a long-term solution—it's just masking the problem.

We're giving a lot of thought to the current Replicator design. We hope to share more on the path forward soon.

Another advantage to using foreign keys is that when you want to delete pruned/revoked/deleted messages, the constraints will automatically cascade the deletes with no additional application logic required by you.

sds commented 3 months ago

We now recommend you use the Shuttle package instead of the old replicator.