mongodb cluster files corruption on snapshots

[bhzhu203]

The issue of mongodb cluster files corruption on BcacheFS snapshots is still not solved.

1) bcachefs format --compression=lz4 /dev/vdb

2) mount -t bcachefs -o verbose /dev/vdb /data

3) su -c "/home/mongo/start.sh" mongo & (/home/mongo/mongodb/bin/mongod --replSet repset --port 27026 --bind_ip 0.0.0.0 --dbpath /home/mongo/data/ --oplogSize 2048 --logpath /home/mongo/data/mongod.log --fork --auth --keyFile /home/mongo/keyfile --wiredTigerCacheSizeGB 4)

4) making snapshots every 10mins :

 cat /root/snapshot-cron.sh
#! /bin/bash
Time=`date +%s`
echo ''
Time=`date +%s`
date
echo Time=$Time
cd /data/snapshots

#killall -19 mongod
sleep 1
sync
sleep 1
sync

/usr/local/sbin/bcachefs subvolume snapshot /data $Time

#killall -18 mongod

ls -l /data/snapshots
echo ''
echo ''

After runing for 10 hours Mongodb panic for files corruption :

{"t":{"$date":"2024-01-11T09:17:34.144+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"ReplWriterWorker-11","msg":"WiredTiger error","attr":{"error":0,"message":"[1704935854:144361][5977:0x7f4e5ce83640], file:collection-140-1053591952236326365.wt, WT_CURSOR.search: __wt_bm_corrupt_dump, 135: {308092928, 12288, 0x9647b460}: (chunk 12 of 12): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "}}
{"t":{"$date":"2024-01-11T09:17:34.144+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"ReplWriterWorker-11","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1704935854:144404][5977:0x7f4e5ce83640], file:collection-140-1053591952236326365.wt, WT_CURSOR.search: __wt_block_read_off, 292: collection-140-1053591952236326365.wt: fatal read error: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-01-11T09:17:34.144+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"ReplWriterWorker-11","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1704935854:144418][5977:0x7f4e5ce83640], file:collection-140-1053591952236326365.wt, WT_CURSOR.search: __wt_block_read_off, 292: the process must exit and restart: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2024-01-11T09:17:34.144+08:00"},"s":"F",  "c":"-",        "id":23089,   "ctx":"ReplWriterWorker-11","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":446}}
{"t":{"$date":"2024-01-11T09:17:34.144+08:00"},"s":"F",  "c":"-",        "id":23090,   "ctx":"ReplWriterWorker-11","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":{"$date":"2024-01-11T09:17:34.144+08:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"ReplWriterWorker-11","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2024-01-11T09:17:34.441+08:00"},"s":"I",  "c":"CONTROL",  "id":31431,   "ctx":"ReplWriterWorker-11","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"55B6803A3D11","b":"55B67D6DE000","o":"2CC5D11","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"},{"a":"55B6803A5349","b":"55B67D6DE000","o":"2CC7349","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"55B6803A2BA6","b":"55B67D6DE000","o":"2CC4BA6","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7F4E72A42520","b":"7F4E72A00000","o":"42520","s":"__sigaction","s+":"50"},{"a":"7F4E72A969FC","b":"7F4E72A00000","o":"969FC","s":"pthread_kill","s+":"12C"},{"a":"7F4E72A42476","b":"7F4E72A00000","o":"42476","s":"raise","s+":"16"},{"a":"7F4E72A287F3","b":"7F4E72A00000","o":"287F3","s":"abort","s+":"D3"},{"a":"55B67E5E582A","b":"55B67D6DE000","o":"F0782A","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12B"},{"a":"55B67E2D38D1","b":"55B67D6DE000","o":"BF58D1","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1026","s+":"16"},{"a":"55B67E7B4BEB","b":"55B67D6DE000","o":"10D6BEB","s":"__eventv","s+":"3FB"},{"a":"55B67E2E4867","b":"55B67D6DE000","o":"C06867","s":"__wt_panic_func","s+":"10B"},{"a":"55B67E2F0C0C","b":"55B67D6DE000","o":"C12C0C","s":"__wt_block_read_off.cold.5","s+":"25"},{"a":"55B67E8BC58F","b":"55B67D6DE000","o":"11DE58F","s":"__wt_bm_read","s+":"13F"},{"a":"55B67E804922","b":"55B67D6DE000","o":"1126922","s":"__wt_bt_read","s+":"92"},{"a":"55B67E811885","b":"55B67D6DE000","o":"1133885","s":"__page_read","s+":"165"},{"a":"55B67E812E7E","b":"55B67D6DE000","o":"1134E7E","s":"__wt_page_in_func","s+":"42E"},{"a":"55B67E85195D","b":"55B67D6DE000","o":"117395D","s":"__wt_row_search","s+":"92D"},{"a":"55B67E7F570F","b":"55B67D6DE000","o":"111770F","s":"__wt_btcur_search","s+":"94F"},{"a":"55B67E743AF8","b":"55B67D6DE000","o":"1065AF8","s":"__curfile_search","s+":"228"},{"a":"55B67E6FF409","b":"55B67D6DE000","o":"1021409","s":"_ZN5mongo30wiredTigerPrepareConflictRetryIZNS_31WiredTigerRecordStoreCursorBase9seekExactERKNS_8RecordIdEEUlvE_EEiPNS_16OperationContextEOT_","s+":"89"},{"a":"55B67E6FF781","b":"55B67D6DE000","o":"1021781","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase9seekExactERKNS_8RecordIdE","s+":"B1"},{"a":"55B67F0FF975","b":"55B67D6DE000","o":"1A21975","s":"_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmNS_11unowned_ptrINS_20SeekableRecordCursorEEERKNS_15NamespaceStringE","s+":"65"},{"a":"55B67F0D3C80","b":"55B67D6DE000","o":"19F5C80","s":"_ZN5mongo11IDHackStage6doWorkEPm","s+":"100"},{"a":"55B67F0E0828","b":"55B67D6DE000","o":"1A02828","s":"_ZN5mongo9PlanStage4workEPm","s+":"68"},{"a":"55B67F0FBDC3","b":"55B67D6DE000","o":"1A1DDC3","s":"_ZN5mongo11UpdateStage6doWorkEPm","s+":"493"},{"a":"55B67F0FDE3B","b":"55B67D6DE000","o":"1A1FE3B","s":"_ZN5mongo11UpsertStage6doWorkEPm","s+":"7B"},{"a":"55B67F0E0828","b":"55B67D6DE000","o":"1A02828","s":"_ZN5mongo9PlanStage4workEPm","s+":"68"},{"a":"55B67F124F92","b":"55B67D6DE000","o":"1A46F92","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_8DocumentEEEPNS_8RecordIdE","s+":"222"},{"a":"55B67F1259DB","b":"55B67D6DE000","o":"1A479DB","s":"_ZN5mongo16PlanExecutorImpl7getNextEPNS_8DocumentEPNS_8RecordIdE","s+":"4B"},{"a":"55B67F125B1D","b":"55B67D6DE000","o":"1A47B1D","s":"_ZN5mongo16PlanExecutorImpl11executePlanEv","s+":"4D"},{"a":"55B67F0AB632","b":"55B67D6DE000","o":"19CD632","s":"_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestE","s+":"1C2"},{"a":"55B67EFE02A1","b":"55B67D6DE000","o":"19022A1","s":"_ZZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS0_26OplogEntryOrGroupedInsertsEbNS0_16OplogApplication4ModeESt8functionIFvvEEENKUlvE10_clEv","s+":"91"},{"a":"55B67EFE5818","b":"55B67D6DE000","o":"1907818","s":"_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS0_26OplogEntryOrGroupedInsertsEbNS0_16OplogApplication4ModeESt8functionIFvvEE","s+":"1418"},{"a":"55B67EA22E7F","b":"55B67D6DE000","o":"1344E7F","s":"_ZZN5mongo4repl31applyOplogEntryOrGroupedInsertsEPNS_16OperationContextERKNS0_26OplogEntryOrGroupedInsertsENS0_16OplogApplication4ModeEENKUlvE1_clEv","s+":"2FF"},{"a":"55B67EA2609D","b":"55B67D6DE000","o":"134809D","s":"_ZN5mongo4repl31applyOplogEntryOrGroupedInsertsEPNS_16OperationContextERKNS0_26OplogEntryOrGroupedInsertsENS0_16OplogApplication4ModeE","s+":"9DD"},{"a":"55B67EA26C09","b":"55B67D6DE000","o":"1348C09","s":"_ZN5mongo4repl16OplogApplierImpl24applyOplogBatchPerWorkerEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS7_EEPS4_INS_16MultikeyPathInfoESaISB_EE","s+":"209"},{"a":"55B67EA22641","b":"55B67D6DE000","o":"1344641","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_4repl16OplogApplierImpl16_applyOplogBatchEPNS_16OperationContextESt6vectorINS5_10OplogEntryESaISA_EEEUlT_E3_EEDaOSD_EN12SpecificImpl4callEOS1_","s+":"111"},{"a":"55B67FEDC932","b":"55B67D6DE000","o":"27FE932","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockINS_12latch_detail5LatchEE","s+":"132"},{"a":"55B67FEDF156","b":"55B67D6DE000","o":"2801156","s":"_ZN5mongo10ThreadPool13_consumeTasksEv","s+":"86"},{"a":"55B67FEDFF01","b":"55B67D6DE000","o":"2801F01","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE","s+":"E1"},{"a":"55B67FEE0230","b":"55B67D6DE000","o":"2802230","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool25_startWorkerThread_inlockEvEUlvE2_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"60"},{"a":"55B68054FE7F","b":"55B67D6DE000","o":"2E71E7F","s":"execute_native_thread_routine","s+":"F"},{"a":"7F4E72A94AC3","b":"7F4E72A00000","o":"94AC3","s":"pthread_condattr_setpshared","s+":"513"},{"a":"7F4E72B26850","b":"7F4E72A00000","o":"126850","s":"__xmknodat","s+":"230"}],"processInfo":{"mongodbVersion":"4.4.1","gitVersion":"ad91a93a5a31e175f5cbf8c69561e788bbc55ce1","compiledModules":[],"uname":{"sysname":"Linux","release":"6.7.0-uksm+","version":"#13 SMP PREEMPT_DYNAMIC Wed Jan 10 11:03:19 CST 2024","machine":"x86_64"},"somap":[{"b":"55B67D6DE000","elfType":3,"buildId":"6AF036CAFAF898729FE132E06C67BAC328F17731"},{"b":"7F4E72A00000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"C289DA5071A3399DE893D2AF81D6A30C62646E1E"}]}}}}
{"t":{"$date":"2024-01-11T09:17:34.441+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B6803A3D11","b":"55B67D6DE000","o":"2CC5D11","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"}}}
{"t":{"$date":"2024-01-11T09:17:34.441+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B6803A5349","b":"55B67D6DE000","o":"2CC7349","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":{"$date":"2024-01-11T09:17:34.441+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B6803A2BA6","b":"55B67D6DE000","o":"2CC4BA6","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F4E72A42520","b":"7F4E72A00000","o":"42520","s":"__sigaction","s+":"50"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F4E72A969FC","b":"7F4E72A00000","o":"969FC","s":"pthread_kill","s+":"12C"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F4E72A42476","b":"7F4E72A00000","o":"42476","s":"raise","s+":"16"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F4E72A287F3","b":"7F4E72A00000","o":"287F3","s":"abort","s+":"D3"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E5E582A","b":"55B67D6DE000","o":"F0782A","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12B"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E2D38D1","b":"55B67D6DE000","o":"BF58D1","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1026","s+":"16"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E7B4BEB","b":"55B67D6DE000","o":"10D6BEB","s":"__eventv","s+":"3FB"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E2E4867","b":"55B67D6DE000","o":"C06867","s":"__wt_panic_func","s+":"10B"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E2F0C0C","b":"55B67D6DE000","o":"C12C0C","s":"__wt_block_read_off.cold.5","s+":"25"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E8BC58F","b":"55B67D6DE000","o":"11DE58F","s":"__wt_bm_read","s+":"13F"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E804922","b":"55B67D6DE000","o":"1126922","s":"__wt_bt_read","s+":"92"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E811885","b":"55B67D6DE000","o":"1133885","s":"__page_read","s+":"165"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E812E7E","b":"55B67D6DE000","o":"1134E7E","s":"__wt_page_in_func","s+":"42E"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E85195D","b":"55B67D6DE000","o":"117395D","s":"__wt_row_search","s+":"92D"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E7F570F","b":"55B67D6DE000","o":"111770F","s":"__wt_btcur_search","s+":"94F"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E743AF8","b":"55B67D6DE000","o":"1065AF8","s":"__curfile_search","s+":"228"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E6FF409","b":"55B67D6DE000","o":"1021409","s":"_ZN5mongo30wiredTigerPrepareConflictRetryIZNS_31WiredTigerRecordStoreCursorBase9seekExactERKNS_8RecordIdEEUlvE_EEiPNS_16OperationContextEOT_","s+":"89"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67E6FF781","b":"55B67D6DE000","o":"1021781","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase9seekExactERKNS_8RecordIdE","s+":"B1"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67F0FF975","b":"55B67D6DE000","o":"1A21975","s":"_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmNS_11unowned_ptrINS_20SeekableRecordCursorEEERKNS_15NamespaceStringE","s+":"65"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67F0D3C80","b":"55B67D6DE000","o":"19F5C80","s":"_ZN5mongo11IDHackStage6doWorkEPm","s+":"100"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67F0E0828","b":"55B67D6DE000","o":"1A02828","s":"_ZN5mongo9PlanStage4workEPm","s+":"68"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67F0FBDC3","b":"55B67D6DE000","o":"1A1DDC3","s":"_ZN5mongo11UpdateStage6doWorkEPm","s+":"493"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67F0FDE3B","b":"55B67D6DE000","o":"1A1FE3B","s":"_ZN5mongo11UpsertStage6doWorkEPm","s+":"7B"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67F0E0828","b":"55B67D6DE000","o":"1A02828","s":"_ZN5mongo9PlanStage4workEPm","s+":"68"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67F124F92","b":"55B67D6DE000","o":"1A46F92","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_8DocumentEEEPNS_8RecordIdE","s+":"222"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67F1259DB","b":"55B67D6DE000","o":"1A479DB","s":"_ZN5mongo16PlanExecutorImpl7getNextEPNS_8DocumentEPNS_8RecordIdE","s+":"4B"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67F125B1D","b":"55B67D6DE000","o":"1A47B1D","s":"_ZN5mongo16PlanExecutorImpl11executePlanEv","s+":"4D"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67F0AB632","b":"55B67D6DE000","o":"19CD632","s":"_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestE","s+":"1C2"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67EFE02A1","b":"55B67D6DE000","o":"19022A1","s":"_ZZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS0_26OplogEntryOrGroupedInsertsEbNS0_16OplogApplication4ModeESt8functionIFvvEEENKUlvE10_clEv","s+":"91"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67EFE5818","b":"55B67D6DE000","o":"1907818","s":"_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS0_26OplogEntryOrGroupedInsertsEbNS0_16OplogApplication4ModeESt8functionIFvvEE","s+":"1418"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67EA22E7F","b":"55B67D6DE000","o":"1344E7F","s":"_ZZN5mongo4repl31applyOplogEntryOrGroupedInsertsEPNS_16OperationContextERKNS0_26OplogEntryOrGroupedInsertsENS0_16OplogApplication4ModeEENKUlvE1_clEv","s+":"2FF"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67EA2609D","b":"55B67D6DE000","o":"134809D","s":"_ZN5mongo4repl31applyOplogEntryOrGroupedInsertsEPNS_16OperationContextERKNS0_26OplogEntryOrGroupedInsertsENS0_16OplogApplication4ModeE","s+":"9DD"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67EA26C09","b":"55B67D6DE000","o":"1348C09","s":"_ZN5mongo4repl16OplogApplierImpl24applyOplogBatchPerWorkerEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS7_EEPS4_INS_16MultikeyPathInfoESaISB_EE","s+":"209"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67EA22641","b":"55B67D6DE000","o":"1344641","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_4repl16OplogApplierImpl16_applyOplogBatchEPNS_16OperationContextESt6vectorINS5_10OplogEntryESaISA_EEEUlT_E3_EEDaOSD_EN12SpecificImpl4callEOS1_","s+":"111"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67FEDC932","b":"55B67D6DE000","o":"27FE932","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockINS_12latch_detail5LatchEE","s+":"132"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67FEDF156","b":"55B67D6DE000","o":"2801156","s":"_ZN5mongo10ThreadPool13_consumeTasksEv","s+":"86"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67FEDFF01","b":"55B67D6DE000","o":"2801F01","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE","s+":"E1"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B67FEE0230","b":"55B67D6DE000","o":"2802230","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool25_startWorkerThread_inlockEvEUlvE2_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"60"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"55B68054FE7F","b":"55B67D6DE000","o":"2E71E7F","s":"execute_native_thread_routine","s+":"F"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F4E72A94AC3","b":"7F4E72A00000","o":"94AC3","s":"pthread_condattr_setpshared","s+":"513"}}}
{"t":{"$date":"2024-01-11T09:17:34.442+08:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"ReplWriterWorker-11","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F4E72B26850","b":"7F4E72A00000","o":"126850","s":"__xmknodat","s+":"230"}}}
Aborted (core dumped)

bcachefs show-super /dev/vdb
External UUID:                              5b47f962-2599-420c-a914-64caaa695988
Internal UUID:                              ffc3adc1-6287-47bb-91f8-e2531c8c9240
Magic number:                               c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index:                               0
Label:                                      
Version:                                    1.4: member_seq
Version upgrade complete:                   1.4: member_seq
Oldest version on disk:                     1.4: member_seq
Created:                                    Wed Jan 10 16:21:24 2024
Sequence number:                            14
Time of last write:                         Thu Jan 11 09:18:44 2024
Superblock size:                            4544
Clean:                                      1
Devices:                                    1
Sections:                                   members_v1,replicas_v0,clean,journal_v2,counters,members_v2,errors,ext,downgrade
Features:                                   lz4,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               512 B
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic 
  metadata_replicas:                        1
  data_replicas:                            1
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash 
  data_checksum:                            none [crc32c] crc64 xxhash 
  compression:                              lz4
  background_compression:                   none
  str_hash:                                 crc32c crc64 [siphash] 
  metadata_target:                          none
  foreground_target:                        none
  background_target:                        none
  promote_target:                           none
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  version_upgrade:                          [compatible] incompatible none 
  nocow:                                    0

members_v2 (size 144):
Device:                                     0
  Label:                                    (none)
  UUID:                                     518ead16-bf1f-46e0-8399-6680d16fe60b
  Size:                                     250 GiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              256 KiB
  First bucket:                             0
  Buckets:                                  1024000
  Last mount:                               Thu Jan 11 09:16:36 2024
  Last superblock write:                    14
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 journal,btree,user
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1

errors (size 8):

Here is the FS dump https://1drv.ms/u/s!Ao8p2C5olADnhk84GjtzoDECgBDJ?e=XsBHq3

Kernel version 045d7ff42fd8fcdc08ddee367347e895c4a09ade Tool version 9a9af6e9e6560407326b37345ebe434a0839420f

[Damenly]

Related to https://github.com/koverstreet/bcachefs/issues/619. Seems not ztsd only.

[Damenly]

I tried to attach the qcow2 image after uncompressing: Image does not contain a reference count table It looks like a QEMU QCOW Image (v2):

$ file mongo-0111.qcow2
mongo-0111.qcow2: QEMU QCOW Image (v2), 268435456000 bytes (v2), 268435456000 bytes

[bhzhu203]

I tried to attach the qcow2 image after uncompressing: Image does not contain a reference count table It looks like a QEMU QCOW Image (v2):

$ file mongo-0111.qcow2
mongo-0111.qcow2: QEMU QCOW Image (v2), 268435456000 bytes (v2), 268435456000 bytes

It is just a FS metadata dump , not a disk image . I think