search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.9k stars 1.06k forks source link

[DocDB][xCluster DR] Unplanned DR scenario with non-transactional workload fails during Failback #15464

Closed Arjun-yb closed 1 year ago

Arjun-yb commented 1 year ago

Jira Link: DB-4763

Description

Steps:

  1. Create 2 universes with enable_replicate_transaction_status_table=true and disable automatic tablet splitting
  2. Create table and indexes at source and target
  3. Setup unidirectional replication from source to target.
  4. Change target’s xcluster role to **STANDBY**
  5. Create snapshot schedule at target
  6. Load data at Source(350K+)
  7. Validate data at both sides - works fine
  8. Stop applications and pause the replication from source to target
  9. At target: Get xcluster safe time and restore to the safe time
  10. Promote Target to Active role and switch applications to target and resume applications.
  11. Delete replication from source to target
  12. Drop table and database at source.
  13. Wait for some time and follow bootstrap flow from target to source(Create same database, table and indexes at source and create replication from target to source)

Observations

  1. There is huge lag.
  2. YSQLSH connectivity issue at source side
  3. 1 Node is in unreachable state and 1 node's master is down.
  4. Observed >50% CPU Usage at Source side even though there are no running applications at source
  5. Found some errors at both source and target

Source: 

E1229 13:22:45.353241 32740 twodc_output_client.cc:714] Error while applying replicated record: Timed out (yb/rpc/rpc.cc:222): CDCWriteRpc: tablet_id: "b13f18c638984f0dac73da279343e932" write_batch { write_pairs { key: "GYsH\200\000\000/!SG\010gSused-id-34-96827\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808863302017024 transaction { transaction_id: "\302\231\r\245\242\204L[\272.X\272\323\260\032q" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYUH\200\000\0000!SGk\301Sused-id-34-96828\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808863317233664 transaction { transaction_id: "\2303\277\320\364\267JW\265\226\013\271M\271a\223" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYsH\200\000\000/!SG\261\320Sused-id-34-96887\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808864265035776 transaction { transaction_id: "A\345\350\026|\363O\320\260K0\222]V\024\246" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYUH\200\000\0000!SG8\374Sused-id-34-96888\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808864280018944 transaction { transaction_id: "\273k,\207\014\210Lg\271\020\024\373*j\026\305" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYsH\200\000\000/!SG\320\355Sused-id-34-96947\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808865216151552 transaction { transaction_id: "+Qu\021\204\331K9\214\220\321\367\212\256\227\210" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYUH\200\000\0000!SG\231 Sused-id-34-96948\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808865232613376 transaction { transaction_id: "u\022^\300\341!H\316\263\333\nK\361\305\267\331" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYsH\200\000\000/!SG1\342Sused-id-34-97007\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808866160971776 transaction { transaction_id: "\252<\266B\346\006D\335\214\226\265x\327QS\216" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYUH\200\000\0000!SGv\003Sused-id-34-97008\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808866177540096 transaction { transaction_id: "\265\236\336\250\213zL\212\201\316\235R\250O\207\223" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYsH\200\000\000/!SG\266FSused-id-34-97067\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808867123052544 transaction { transaction_id: "AX\215\276\375vMz\222^I\330J|\306&" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYUH\200\000\0000!SG\356>Sused-id-34-97068\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808867141070848 transaction { transaction_id: "\021]j\r8\240N\261\205G^+\343\277_9" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYsH\200\000\000/!SGh\367Sused-id-34-97127\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808868087779328 transaction { transaction_id: "\341\022\270]<\354M\242\265\271\'\244\351\334\301\"" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYUH\200\000\0000!SGr\367Sused-id-34-97128\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808868105408512 transaction { transaction_id: "\344c_\272\362xO\006\212\320t\330\017K\\," isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYsH\200\000\000/!SG\224]Sused-id-34-97187\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808869035593728 transaction { transaction_id: "\214\343\305d\367\224A\210\245\350\010INX\3114" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYUH\200\000\0000!SG\374\217Sused-id-34-97188\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808869051846656 transaction { transaction_id: "u\324\034\202o\325O\177\205\202M4\253\373;\331" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYsH\200\000\000/!SG\004kSused-id-34-97247\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808869981728768 transaction { transaction_id: "\337\314\334\232C\235I8\271\326\274\020\323\n;*" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYUH\200\000\0000!SGq\310Sused-id-34-97248\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808869996449792 transaction { transaction_id: "\007\035\235\202\324\213C\206\213\217\330\201\361\302\024*" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYsH\200\000\000/!SG\022zSused-id-34-97307\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808870938972160 transaction { transaction_id: "\026#\032\301_EN\200\221\204iT\257\273\201\230" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYUH\200\000\0000!SGU\336Sused-id-34-97308\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808870954864640 transaction { transaction_id: "\325/X,\307\240N\007\244\361\273\363\326\347\271\177" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYsH\200\000\000/!SG\221\311Sused-id-34-97367\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808871891226624 transaction { transaction_id: "\274q\313\341\352=G5\2534\204H\276\246\357\261" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYUH\200\000\0000!SG\0079Sused-id-34-97368\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808871907729408 transaction { transaction_id: "\036G^\301\2158Gq\264\006\307\322;7\327\247" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYsH\200\000\000/!SGQ&Sused-id-34-97427\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808872862224384 transaction { transaction_id: "\207\341\327\017\275\213As\256S\341\366lI\205&" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYUH\200\000\0000!SG#\"Sused-id-34-97428\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808872878931968 transaction { transaction_id: "\371\351\231\346~6BZ\226\322\253-\nQ\264^" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYsH\200\000\000/!SG\316\306Sused-id-34-97487\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808873831133184 transaction { transaction_id: "\335p\354\257s\307J`\2217\3612\237\336Q\203" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYUH\200\000\0000!SG\201\254Sused-id-34-97488\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808873848950784 transaction { transaction_id: "6\364/\273\001{B\317\2142w\334\016J5b" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYsH\200\000\000/!SG\307\375Sused-id-34-97547\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808874789527552 transaction { transaction_id: "\206\354\241Pyl@\230\232\222\217z4*\344\007" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYUH\200\000\0000!SG[\261Sused-id-34-97548\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808874807508992 transaction { transaction_id: "HUx%Z9Ey\220A\210T\010\331[#" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYsH\200\000\000/!SG\017\240Sused-id-34-97607\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808875770294272 transaction { transaction_id: "oJ\237,j\205@\206\200t\235$F\275\331L" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYUH\200\000\0000!SG\250\314Sused-id-34-97608\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808875786678272 transaction { transaction_id: "\375\277j\317\200\nEq\227]\305K\305\221w\351" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYsH\200\000\000/!SG\330\215Sused-id-34-97667\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808876746289152 transaction { transaction_id: "\030\237PG\033?C\332\214\316j\034\201\035\237\277" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYUH\200\000\0000!SGW\017Sused-id-34-97668\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808876763926528 transaction { transaction_id: "(\361\345\031p\243M/\246.\r\360\371e\3351" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYsH\200\000\000/!SG\266\362Sused-id-34-97727\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808877711478784 transaction { transaction_id: "\n\342\200\023g\246@\311\261\271W\370+\342\234a" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYUH\200\000\0000!SG\221\214Sused-id-34-97728\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808877732159488 transaction { transaction_id: "\257@O?\"\271G%\207\370\037\033\341\351\340v" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYsH\200\000\000/!SG\210\252Sused-id-34-97787\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808878683340800 transaction { transaction_id: "\2401\nj#\235O\336\243\374\177\357\330\346\277\010" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYUH\200\000\0000!SG\316~Sused-id-34-97788\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808878699642880 transaction { transaction_id: ")\367\367K\304\201H\200\277VnME2\227\376" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYsH\200\000\000/!SG\256HSused-id-34-97847\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808879645569024 transaction { transaction_id: "q`\234\2656\243O\237\263\3031\022@\031\351\004" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYUH\200\000\0000!SG\307\351Sused-id-34-97848\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808879661670400 transaction { transaction_id: "\326U$\242w EA\266\255\345\2021|D\354" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYsH\200\000\000/!SG&\303Sused-id-34-97907\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808880600711168 transaction { transaction_id: "7\020\207\"\321%G/\262K=n\221l,\351" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYUH\200\000\0000!SG\0354Sused-id-34-97908\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808880618115072 transaction { transaction_id: "k\r\356~\014kHr\226\036\206V\260e\251\305" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYsH\200\000\000/!SG\362\324Sused-id-34-97967\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808881557450752 transaction { transaction_id: "\331\234G\2131\340I\312\205\203U\022nC\215#" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYUH\200\000\0000!SG\277\306Sused-id-34-97968\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808881572659200 transaction { transaction_id: "j\214o\323h\367Nm\205\375\002s}\304\026-" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYsH\200\000\000/!SG\361\013Sused-id-34-98027\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808882501640192 transaction { transaction_id: "Z\243\274\272\230\340E\247\227_#]\316\330\330\250" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYUH\200\000\0000!SG>\262Sused-id-34-98028\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808882517848064 transaction { transaction_id: "\3776\357\374\217\272A-\206FL\347\366mu\232" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYsH\200\000\000/!SG-\200Sused-id-34-98087\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808883468263424 transaction { transaction_id: "\n\252\263i+\346E\345\233e\335\306R\370K`" isolation: SNAPSHOT_ISOLATION status_tablet: "1c16ced2d99c46b5a798fcd4aa49eeb3" } } write_pairs { key: "GYUH\200\000\0000!SG\355xSused-id-34-98088\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808883484180480 transaction { transaction_id: "\337c\3138OcN|\255H\022(\n\033\261\276" isolation: SNAPSHOT_ISOLATION status_tablet: "edba634a48e046319b14d8e2c09152db" } } write_pairs { key: "GYsH\200\000\000/!SG\261\302Sused-id-34-98147\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808884417073152 transaction { transaction_id: "\3715R\257\266\311F\006\262\022!\211\203\310S\276" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYUH\200\000\0000!SG\021\357Sused-id-34-98148\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808884432080896 transaction { transaction_id: "d}\256G_lM1\240\237c\2455+%\355" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYsH\200\000\000/!SG9\024Sused-id-34-98207\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808885374816256 transaction { transaction_id: "\024\341\"\'\177mH\343\2147\340\n&CO\262" isolation: SNAPSHOT_ISOLATION status_tablet: "edba634a48e046319b14d8e2c09152db" } } write_pairs { key: "GYUH\200\000\0000!SGY\317Sused-id-34-98208\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808885391003648 transaction { transaction_id: "o}\347=cRB\325\263\035p\033\342w!6" isolation: SNAPSHOT_ISOLATION status_tablet: "edba634a48e046319b14d8e2c09152db" } } write_pairs { key: "GYsH\200\000\000/!SG\320aSused-id-34-98267\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808886329085952 transaction { transaction_id: "U0\372\235{\003A\344\216\377\275e7\0338\034" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYUH\200\000\0000!SGR\374Sused-id-34-98268\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808886345863168 transaction { transaction_id: "\001\274\\\263\241rJ\377\217d\276\361\216\030\226\255" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYsH\200\000\000/!SG\250\301Sused-id-34-98327\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808887306940416 transaction { transaction_id: "\227c2\225\366SG\r\212\353\276\220\221B\267n" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYUH\200\000\0000!SGkvSused-id-34-98328\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808887322988544 transaction { transaction_id: "\246\022c\242\205VB\257\237-\343\207\330g\014\225" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYsH\200\000\000/!SGIcSused-id-34-98387\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808888257732608 transaction { transaction_id: "AQl\202\0168L2\270\374\230\230\230\357o\336" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYUH\200\000\0000!SG<\276Sused-id-34-98388\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808888274419712 transaction { transaction_id: ":\230\024C\343\000G`\232Pmd\307\377\035\001" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYsH\200\000\000/!SG\314\241Sused-id-34-98447\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808889222942720 transaction { transaction_id: "\324~\351\323KWH\242\233OU\322\273\351\000\316" isolation: SNAPSHOT_ISOLATION status_tablet: "1c16ced2d99c46b5a798fcd4aa49eeb3" } } write_pairs { key: "GYUH\200\000\0000!SG\275\363Sused-id-34-98448\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808889240117248 transaction { transaction_id: "\025\260p\342\376\231C\270\24773\211\354\231\323\263" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYsH\200\000\000/!SGY\213Sused-id-34-98507\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808890186465280 transaction { transaction_id: "\323\304\274\316\361#O\252\265<\225n\001\300E\273" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYUH\200\000\0000!SGnvSused-id-34-98508\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808890202669056 transaction { transaction_id: "\216\205`\256:rG\256\203\"\330\332\307Sk#" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYsH\200\000\000/!SG8NSused-id-34-98567\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808891142664192 transaction { transaction_id: "\217\347v)u\252L\304\260)t\272\323\033>\321" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYUH\200\000\0000!SG\"sSused-id-34-98568\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808891158589440 transaction { transaction_id: "\305\321\257\275\211\021AC\221y\006\211n\237\364\221" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYsH\200\000\000/!SGO\023Sused-id-34-98627\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808892123107328 transaction { transaction_id: "\0036\252\320Q\373F\373\250e\017\266`\333\276\215" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYUH\200\000\0000!SG\360\213Sused-id-34-98628\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808892140359680 transaction { transaction_id: "\205\201\370\311+iA\016\223\004\026x\325(\005H" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYsH\200\000\000/!SG^\363Sused-id-34-98687\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808893079560192 transaction { transaction_id: "@\214\207\351\354{N\017\224<\242W\251\245z\250" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYUH\200\000\0000!SGI\300Sused-id-34-98688\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808893094326272 transaction { transaction_id: "\303S\014\n\251,K\323\214_\247\357\366b\262\346" isolation: SNAPSHOT_ISOLATION status_tablet: "edba634a48e046319b14d8e2c09152db" } } write_pairs { key: "GYsH\200\000\000/!SG\227\372Sused-id-34-98747\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808894027481088 transaction { transaction_id: "Vf\226\314\007\255D\211\251\017\260>\302\216\206\006" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYUH\200\000\0000!SG\033dSused-id-34-98748\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808894042951680 transaction { transaction_id: "\036\235\355FD\333K\240\211\2367!\020s2J" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYsH\200\000\000/!SG\264\206Sused-id-34-98807\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808894987128832 transaction { transaction_id: "l\242\010Hd\216O\017\277_\205[\312Pao" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYUH\200\000\0000!SGSbSused-id-34-98808\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808895001886720 transaction { transaction_id: "\007\204M\357\367\211C\377\236\037\242\317\315\357A\242" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYsH\200\000\000/!SG(ZSused-id-34-98867\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808895945162752 transaction { transaction_id: "\310\355Rv\034ZB\\\272\361H\346,\010\254\010" isolation: SNAPSHOT_ISOLATION status_tablet: "edba634a48e046319b14d8e2c09152db" } } write_pairs { key: "GYUH\200\000\0000!SG\236\213Sused-id-34-98868\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808895961194496 transaction { transaction_id: "\014I\207_A\255J\370\237i\361\304\3116\362\272" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYsH\200\000\000/!SG\341\267Sused-id-34-98927\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808896899637248 transaction { transaction_id: "\025\303\344\002)\340NO\277\023\277\343\230tz6" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYUH\200\000\0000!SGI\365Sused-id-34-98928\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808896914440192 transaction { transaction_id: "\3126\031#\000\016J\t\216\323\261=\032a^Y" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYsH\200\000\000/!SG\342oSused-id-34-98987\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808897849683968 transaction { transaction_id: "p@t4\261tI\272\247\305\220v\264\230\247T" isolation: SNAPSHOT_ISOLATION status_tablet: "edba634a48e046319b14d8e2c09152db" } } write_pairs { key: "GYUH\200\000\0000!SG\246sSused-id-34-98988\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808897865633792 transaction { transaction_id: "\364^c\250\3642C\265\2154\217c\354\351o\306" isolation: SNAPSHOT_ISOLATION status_tablet: "edba634a48e046319b14d8e2c09152db" } } write_pairs { key: "GYsH\200\000\000/!SGp\315Sused-id-34-99047\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808898818453504 transaction { transaction_id: "3\025\224\\\245\252A\203\271$\373\0021\327\243\352" isolation: SNAPSHOT_ISOLATION status_tablet: "edba634a48e046319b14d8e2c09152db" } } write_pairs { key: "GYUH\200\000\0000!SG\215`Sused-id-34-99048\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808898835120128 transaction { transaction_id: "c\207\325|wf@!\236Cz\343]\tk\336" isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYsH\200\000\000/!SG\263\271Sused-id-34-99107\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808899802812416 transaction { transaction_id: "\302\365\233S6\347@\340\256\311\3632 \312\354\341" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYUH\200\000\0000!SG\252\225Sused-id-34-99108\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808899819470848 transaction { transaction_id: "\253S\030\343N\215G\025\272\"wd\230\370PE" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYsH\200\000\000/!SG\326hSused-id-34-99167\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808900762456064 transaction { transaction_id: "\366\024\237|]\207Km\223>O\337\224\035\030\300" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYUH\200\000\0000!SGycSused-id-34-99168\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808900778172416 transaction { transaction_id: "?[\037E7\000CZ\270\0343w!\266\0316" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYsH\200\000\000/!SGU\211Sused-id-34-99227\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808901711683584 transaction { transaction_id: "\020$\324i\242\030K\334\252\255W\361\007\322\323\222" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYUH\200\000\0000!SG?!Sused-id-34-99228\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808901726355456 transaction { transaction_id: "!9\362B>\274BY\241\313U\203\025\372\257|" isolation: SNAPSHOT_ISOLATION status_tablet: "561b1146c1ee4f8e8912b31b0e646cf1" } } write_pairs { key: "GYsH\200\000\000/!SG\333\257Sused-id-34-99287\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808902671208448 transaction { transaction_id: "\316\275\340\004\374\245O\221\247\\\326\223\235\320\221\351" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYUH\200\000\0000!SG\2618Sused-id-34-99288\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808902686253056 transaction { transaction_id: "-,\371\220\367\247C\317\210\227tm\230\337F5" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYsH\200\000\000/!SG\003\003Sused-id-34-99347\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808903616598016 transaction { transaction_id: "\262\262\337\271|\231H\367\241D\217\331\263\321Bn" isolation: SNAPSHOT_ISOLATION status_tablet: "edba634a48e046319b14d8e2c09152db" } } write_pairs { key: "GYUH\200\000\0000!SG16Sused-id-34-99348\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808903632273408 transaction { transaction_id: "a\3175R3\266G\034\261\364ka\336ogF" isolation: SNAPSHOT_ISOLATION status_tablet: "d7993b2f1118453bb69d5b3c2e7c9bf7" } } write_pairs { key: "GYsH\200\000\000/!SGk\232Sused-id-34-99407\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808904586928128 transaction { transaction_id: "\263\035\275\236\245\331C&\216U\241\351a\323\030l" isolation: SNAPSHOT_ISOLATION status_tablet: "1c16ced2d99c46b5a798fcd4aa49eeb3" } } write_pairs { key: "GYUH\200\000\0000!SG\372bSused-id-34-99408\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808904602918912 transaction { transaction_id: "\374T\003r\324;C\310\202\000\264\223\304F\202," isolation: SNAPSHOT_ISOLATION status_tablet: "820aa1c819d84a49ae28abbc98a4ed6d" } } write_pairs { key: "GYsH\200\000\000/!SG\334\027Sused-id-34-99467\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808905547157504 transaction { transaction_id: "\241YT\352 UA\376\234g\032\033\273r\267\001" isolation: SNAPSHOT_ISOLATION status_tablet: "3fa67e8132b645f2bfb5e76278eb6e34" } } write_pairs { key: "GYUH\200\000\0000!SG\021\367Sused-id-34-99468\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808905561980928 transaction { transaction_id: "\226\3743.\000zOX\203\300\370\216i\315p\001" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYsH\200\000\000/!SG\313\312Sused-id-34-99527\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808906495696896 transaction { transaction_id: "x\373k5\210\311M\212\261\254\332L\004\306\204h" isolation: SNAPSHOT_ISOLATION status_tablet: "edba634a48e046319b14d8e2c09152db" } } write_pairs { key: "GYUH\200\000\0000!SG\2442Sused-id-34-99528\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808906512855040 transaction { transaction_id: "?\260\255Y\367\256G\340\264\023\262@\224Q\343\330" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYsH\200\000\000/!SGA\021Sused-id-34-99587\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808907455356928 transaction { transaction_id: "fG\021\377D\312B4\234\375\222zv\272\3442" isolation: SNAPSHOT_ISOLATION status_tablet: "acb350f503c7499d9803dc875617877b" } } write_pairs { key: "GYUH\200\000\0000!SGqiSused-id-34-99588\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808907471941632 transaction { transaction_id: "+j\342f\341CO,\2559,4\032\273s\215" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYsH\200\000\000/!SG\251MSused-id-34-99647\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808908416864256 transaction { transaction_id: "\307\037>\240B\322B\t\261\274\203\316\005\232\'\203" isolation: SNAPSHOT_ISOLATION status_tablet: "cdeba40ac97f4471a01059a2b2d7f59c" } } write_pairs { key: "GYUH\200\000\0000!SGz\352Sused-id-34-99648\000\001\000\001!!\000\000!J\200" value: "$" external_hybrid_time: 6849808908432248832 transaction { transaction_id: "\261P\220\3715%F\213\267\217\303,\243\233\032\010" isolation: SNAPSHOT_ISOLATION s
E1229 13:22:50.859436 19095 twodc_output_client.cc:714] Error while applying replicated record: Timed out (yb/rpc/outbound_call.cc:488): UpdateTransaction RPC (request call id 40786319) to 172.151.30.48:9100 timed out after 7.303s, consumer tablet: 847c4ff24da84b91b5e2218f2060a885

At Target:

E1229 09:23:50.661798 10204 docdb_compaction_filter_intents.cc:180] T ab64476ef3c14b59944b71c8f3733529: Not found (yb/tablet/transaction_participant.cc:1117): Unknown transaction f4f1aaa3-fcc5-4303-beef-748db1bf6ed1
E1229 09:23:50.661861 10204 docdb_compaction_filter_intents.cc:180] T ab64476ef3c14b59944b71c8f3733529: Not found (yb/tablet/transaction_participant.cc:1117): Unknown transaction f9a00b35-f809-4a0b-9f38-82db18cb4602
E1229 09:23:50.661883 10204 docdb_compaction_filter_intents.cc:180] T ab64476ef3c14b59944b71c8f3733529: Not found (yb/tablet/transaction_participant.cc:1117): Unknown transaction faea383e-2e22-4d4e-a5c2-eae2d2b4da28
lingamsandeep commented 1 year ago

This should be fixed by ef90b7bbc37cfd45befe55617d9886fcb51cee09.