search

bepaald / signalbackup-tools

Tool to work with Signal Backup files.
GNU General Public License v3.0
755 stars 36 forks source link

bad alloc when deleting attachments #190

Closed hrehfeld closed 5 months ago

hrehfeld commented 6 months ago 
 *** Starting log: 2024-02-24 01:36:24 *** 
signalbackup-tools (signalbackup-tools) source version 20240222.164410
BACKUPFILE VERSION: 1
BACKUPFILE SIZE: 9435042329
COUNTER: 1412936055
Reading backup file...
FRAME 279459 (100.0%)... 
Read entire backup file...
done!
Database version: 219
Deleted: 1416 'part'-entries.
  Deleting unused attachments...
cleanDatabaseByMessages
  Deleting attachment entries from 'attachment' not belonging to remaining message entries
  Deleting other threads from 'thread'...
updateThreadsEntries
  Dealing with thread id: 4, 9, 8, 10, 11, 63, 41, 20, 24, 96, 137, 49, 43, 42, 73, 107, 55, 60, 36, 201, 12, 14, 23, 16, 67, 13, 84, 30, 46, 54, 136, 219, 173, 3, 57, 37, 76, 34, 15, 66, 78, 25, 17, 90, 59, 62, 97, 222, 29, 117, 147, 83, 56, 141, 52, 58, 53, 45, 32, 134, 449, 138, 208, 2, 139, 101, 22, 51, 135, 588, 27, 33, 131, 75, 44, 35, 111, 61, 19, 21, 103, 48, 31, 18, 50, 70, 47, 26, 39, 92, 82, 194, 140, 28, 40, 38, 104, 95, 110, 65, 68, 69, 64, 7, 71, 72, 98, 77, 74, 79, 80, 106, 86, 166, 127, 85, 6, 5, 81, 87, 91, 88, 89, 256, 115, 93, 94, 247, 102, 105, 109, 299, 112, 113, 114, 118, 262, 119, 120, 121, 122, 123, 124, 125, 126, 128, 129, 130, 133, 142, 143, 144, 154, 145, 198, 153, 146, 160, 170, 568, 149, 150, 148, 174, 155, 161, 171, 570, 310, 157, 396, 159, 151, 152, 205, 202, 297, 179, 156, 158, 499, 353, 497, 162, 505, 360, 163, 164, 216, 165, 167, 168, 309, 500, 257, 504, 169, 236, 172, 192, 492, 254, 559, 585, 384, 214, 512, 178, 359, 175, 191, 176, 177, 508, 231, 506, 211, 185, 361, 184, 511, 204, 180, 308, 181, 182, 183, 225, 253, 186, 187, 188, 292, 189, 190, 245, 507, 540, 195, 193, 267, 196, 215, 203, 306, 206, 307, 207, 209, 316, 212, 210, 226, 213, 483, 217, 395, 218, 220, 221, 510, 581, 223, 241, 224, 469, 533, 227, 228, 229, 230, 279, 255, 232, 233, 234, 235, 237, 238, 239, 240, 282, 242, 244, 243, 259, 246, 277, 445, 248, 252, 249, 250, 251, 258, 260, 261, 263, 265, 311, 364, 337, 535, 264, 409, 372, 354, 335, 298, 562, 453, 355, 296, 446, 560, 405, 356, 320, 427, 575, 351, 266, 268, 269, 325, 281, 294, 495, 289, 318, 276, 481, 334, 270, 295, 275, 271, 273, 556, 321, 278, 280, 367, 514, 283, 293, 286, 288, 285, 326, 287, 290, 537, 352, 305, 300, 301, 324, 302, 522, 323, 567, 342, 347, 358, 460, 447, 430, 369, 343, 435, 349, 385, 345, 388, 376, 303, 315, 402, 392, 549, 312, 313, 314, 458, 420, 317, 319, 558, 322, 582, 327, 328, 344, 329, 330, 331, 332, 538, 482, 370, 341, 440, 375, 491, 441, 336, 488, 566, 489, 443, 454, 371, 333, 338, 339, 368, 340, 555, 346, 436, 348, 365, 546, 350, 565, 397, 373, 527, 501, 433, 386, 571, 399, 462, 551, 455, 400, 378, 519, 456, 357, 580, 410, 457, 520, 415, 550, 362, 553, 503, 363, 377, 493, 485, 366, 496, 401, 434, 416, 417, 383, 374, 391, 379, 521, 380, 382, 389, 525, 513, 539, 472, 437, 544, 543, 394, 407, 593, 591, 387, 465, 536, 471, 390, 526, 418, 408, 439, 515, 393, 545, 403, 404, 428, 425, 423, 490, 406, 518, 586, 411, 426, 429, 424, 502, 498, 459, 448, 413, 422, 414, 431, 468, 421, 470, 461, 432, 578, 524, 494, 509, 451, 438, 442, 557, 592, 450, 452, 466, 463, 474, 477, 473, 480, 532, 476, 484, 464, 475, 590, 478, 554, 479, 486, 552, 487, 564, 517, 531, 528, 523, 530, 561, 529, 534, 541, 542, 587, 584, 548, 563, 597, 569, 572, 573, 574, 576, 579, 583, 589, 594, 596, 598
  Deleting entries from 'mention' not belonging to remaining mms entries
  Deleting removed groups...
  Deleting unneeded MessageSendLog entries... (0, 0, 0)
  Deleting reactions to non-existing messages... (0)
  Deleting call details from non-existing messages... (0)
  Deleting unreferenced recipient entries...
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
zsh: IOT instruction (core dumped)  signalbackup-tools --output signal-2024-02-14-01-37-39_noattachments.backup -

This is on a 8.8G backup file.

bepaald commented 6 months ago

Thanks for reporting, that does not look good.

Just very quickly looking at this before work: first of all, I can not reproduce it, and even in full debugging mode, running through valgrind I get no problems reported. Usually a bad_alloc would mean trying to allocate more memory than available, is there any chance your machine was out of free RAM while running? It would likely be from other running processes since this tool (especially during this operation) shouldn't use much more than maybe 50MB. Is this bad_alloc reproducible for you, does it happen at the same moment each time?

I will try to investigate more when I get back from work. I do not see a lot of interesting things happening in the code between Deleting unreferenced recipient entries... and the next output (which isn't reached in your run). I've just added a few extra output messages around that part of the code. Could you please update and run again, with --verbose added? It will produce a lot of output while reading the backup file, but only the part after Read entire backup file... could be of any interest.

Thanks!

edit Also, are you using any more options besides --deleteattachments?

bepaald commented 6 months ago

I've been going over this code a bit more, but can't figure it out right now. So I'd really like to know if you can reproduce this, and if so, see the output when running with --verbose. Thanks!

hrehfeld commented 6 months ago

Thanks so much for investigating!

with the old version

new version pulled 15mins ago

...
Input opened successfully
Deleted: 1416 'part'-entries.
  Deleting unused attachments...
cleanDatabaseByMessages
  Deleting attachment entries from 'attachment' not belonging to remaining message entries
  Deleting other threads from 'thread'...
updateThreadsEntries
  Dealing with thread id: 4, 9, 8, 10, 11, 63, 41, 20, 24, 96, 137, 49, 43, 42, 73, 107, 55, 60, 36, 201, 12, 14, 23, 16, 67, 13, 84, 30, 46, 54, 136, 219, 173, 3, 57, 37, 76, 34, 15, 66, 78, 25, 17, 90, 59, 62, 97, 222, 29, 117, 147, 83, 56, 141, 52, 58, 53, 45, 32, 134, 449, 138, 208, 2, 139, 101, 22, 51, 135, 588, 27, 33, 131, 75, 44, 35, 111, 61, 19, 21, 103, 48, 31, 18, 50, 70, 47, 26, 39, 92, 82, 194, 140, 28, 40, 38, 104, 95, 110, 65, 68, 69, 64, 7, 71, 72, 98, 77, 74, 79, 80, 106, 86, 166, 127, 85, 6, 5, 81, 87, 91, 88, 89, 256, 115, 93, 94, 247, 102, 105, 109, 299, 112, 113, 114, 118, 262, 119, 120, 121, 122, 123, 124, 125, 126, 128, 129, 130, 133, 142, 143, 144, 154, 145, 198, 153, 146, 160, 170, 568, 149, 150, 148, 174, 155, 161, 171, 570, 310, 157, 396, 159, 151, 152, 205, 202, 297, 179, 156, 158, 499, 353, 497, 162, 505, 360, 163, 164, 216, 165, 167, 168, 309, 500, 257, 504, 169, 236, 172, 192, 492, 254, 559, 585, 384, 214, 512, 178, 359, 175, 191, 176, 177, 508, 231, 506, 211, 185, 361, 184, 511, 204, 180, 308, 181, 182, 183, 225, 253, 186, 187, 188, 292, 189, 190, 245, 507, 540, 195, 193, 267, 196, 215, 203, 306, 206, 307, 207, 209, 316, 212, 210, 226, 213, 483, 217, 395, 218, 220, 221, 510, 581, 223, 241, 224, 469, 533, 227, 228, 229, 230, 279, 255, 232, 233, 234, 235, 237, 238, 239, 240, 282, 242, 244, 243, 259, 246, 277, 445, 248, 252, 249, 250, 251, 258, 260, 261, 263, 265, 311, 364, 337, 535, 264, 409, 372, 354, 335, 298, 562, 453, 355, 296, 446, 560, 405, 356, 320, 427, 575, 351, 266, 268, 269, 325, 281, 294, 495, 289, 318, 276, 481, 334, 270, 295, 275, 271, 273, 556, 321, 278, 280, 367, 514, 283, 293, 286, 288, 285, 326, 287, 290, 537, 352, 305, 300, 301, 324, 302, 522, 323, 567, 342, 347, 358, 460, 447, 430, 369, 343, 435, 349, 385, 345, 388, 376, 303, 315, 402, 392, 549, 312, 313, 314, 458, 420, 317, 319, 558, 322, 582, 327, 328, 344, 329, 330, 331, 332, 538, 482, 370, 341, 440, 375, 491, 441, 336, 488, 566, 489, 443, 454, 371, 333, 338, 339, 368, 340, 555, 346, 436, 348, 365, 546, 350, 565, 397, 373, 527, 501, 433, 386, 571, 399, 462, 551, 455, 400, 378, 519, 456, 357, 580, 410, 457, 520, 415, 550, 362, 553, 503, 363, 377, 493, 485, 366, 496, 401, 434, 416, 417, 383, 374, 391, 379, 521, 380, 382, 389, 525, 513, 539, 472, 437, 544, 543, 394, 407, 593, 591, 387, 465, 536, 471, 390, 526, 418, 408, 439, 515, 393, 545, 403, 404, 428, 425, 423, 490, 406, 518, 586, 411, 426, 429, 424, 502, 498, 459, 448, 413, 422, 414, 431, 468, 421, 470, 461, 432, 578, 524, 494, 509, 451, 438, 442, 557, 592, 450, 452, 466, 463, 474, 477, 473, 480, 532, 476, 484, 464, 475, 590, 478, 554, 479, 486, 552, 487, 564, 517, 531, 528, 523, 530, 561, 529, 534, 541, 542, 587, 584, 548, 563, 597, 569, 572, 573, 574, 576, 579, 583, 589, 594, 596, 598
  Deleting entries from 'mention' not belonging to remaining mms entries
  Deleting removed groups...
  Deleting unneeded MessageSendLog entries... (0, 0, 0)
  Deleting reactions to non-existing messages... (0)
  Deleting call details from non-existing messages... (0)
  Deleting unreferenced recipient entries...
    Got recipient from GV1_MIGRATION: 205
    Got recipient from GV1_MIGRATION: 140
Got recipients from gv1migration. List now: 140,205
Got recipients from groupmemberships. List now: 2,4,6,19,39,59,90,119,120,140,183,185,192,205,237,254,256,263,289,304,318,382,389,396,397,398,399,400,401,402,403,421,422,479,481,489,491,492,493,494,495,496,497,503,505,506,507,509,510,513,514,516,517,518,521,526,537,541,544,547,548,549,552,554,560,561,562,566,567,575,579,586,597,603,604,605,610,619,621,623,627,630,631,632,640,643,645,646,648,654,656,666,668,671,676,677,680,687,688,689,691,693,695,715,732,736,749,755,756,757,758,759,760,762,763,764,765,767,768,769,770,771,772,773,774,775,776,777,778,779,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,799,800,801,802,803,804,806,807,808,809,810,816,817,818,819,820,822,823,824,825,826,827,829,830,832,834,835,840,841,842,843,844,845,846,848,849,851,855,856,857,858,859,860,863,864,865,866,867,868,869,873,875,876,879,882,883,884,885,888,890,891,892,893,894,895,896,898,899,900,901,902,903,904,907,909,910,911,912,913,914,915,916,917,919,920,922,923,924,926,928,932,933,934,940,941,944,948,949,950,951,953,959,960,963,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,988,989,990,991,992,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1015,1016,1017,1018,1021,1022,1023,1024,1025,1026,1027,1028,1034,1037,1038,1040,1041,1042,1044,1046,1047,1049,1050,1052,1053,1054,1055,1056,1057,1058,1059,1060,1061,1062,1063,1064,1065,1066,1067,1068,1069,1070,1071,1072,1073,1074,1075,1076,1078,1079,1080,1081,1082,1083,1084,1085,1086,1087,1088,1089,1090,1091,1092,1093,1094,1095,1097,1098,1099,1100,1101,1102,1103,1104,1105,1106,1107,1108,1109,1110,1111,1112,1113,1114,1115,1116,1117,1118,1119,1120,1121,1122,1124,1125,1126,1127,1128,1132,1133,1135,1136,1137,1138,1140,1141,1142,1143,1144,1146,1148,1150,1151,1152,1153,1154,1155,1156,1157,1159,1162,1164,1165,1166,1167,1168,1169,1170,1171,1174,1176,1177,1178,1179,1180,1181,1182,1183,1185,1186,1187,1188,1191,1192,1193,1194,1195,1196,1197,1198,1199,1200,1202,1203,1204,1205,1206,1207,1208,1209,1210,1211,1212,1213,1214,1215,1216,1217,1218,1219,1220,1221,1222,1223,1224,1225,1226,1227,1228,1229,1230,1232,1233,1234,1235,1237,1238,1239,1240,1241,1242,1245,1246,1247,1248,1249,1250,1251,1252,1253,1256,1257,1258,1259,1260,1261,1262,1263,1264,1265,1266,1267,1268,1269,1270,1271,1272,1273,1276,1277,1278,1279,1283,1284,1285,1286,1287,1288,1289,1290,1291,1292,1293,1294,1295,1296,1297,1298,1299,1300,1301,1303,1306,1307,1309,1310,1311,1312,1313,1314,1318,1319,1320,1325,1329,1330,1331,1332,1333,1336,1337,1341,1342,1343,1345,1346,1347,1348,1349,1350,1351,1352,1353,1354,1355,1356,1357,1358,1359,1360,1361,1362,1363,1364,1370,1371,1372,1373,1374,1375,1376,1377,1379,1384,1386,1387,1394,1396,1397,1398,1399,1401,1407,1409,1410,1411,1412,1413,1414,1415,1416,1417,1418,1419,1420,1421,1422,1423,1424,1425,1426,1427,1428,1431,1432,1433,1434,1435,1436,1437,1438,1439,1440,1441,1445,1446,1447,1448,1450,1454,1455,1456,1457,1458,1460,1461,1462,1463,1464,1465,1466,1467,1468,1469,1470,1471,1472,1473,1475,1476,1477,1478,1479,1485,1487,1488,1489,1493,1494,1495,1496,1497,1498,1501,1502,1503,1506,1507,1509,1511,1512,1513,1514,1515,1516,1517,1518,1520,1527,1528,1529,1530,1531,1537,1541,1543,1544,1547
Got recipients from mentions. List now: 2,4,6,19,39,59,86,90,119,120,140,183,185,192,205,237,254,256,263,289,304,318,382,389,396,397,398,399,400,401,402,403,421,422,479,481,489,491,492,493,494,495,496,497,499,503,504,505,506,507,508,509,510,511,512,513,514,516,517,518,519,520,521,522,525,526,527,528,530,531,532,537,538,541,544,547,548,549,550,551,552,553,554,557,560,561,562,564,565,566,567,569,570,572,573,575,578,579,583,586,593,594,597,599,600,601,603,604,605,606,607,608,610,612,613,615,616,619,620,621,623,627,629,630,631,632,633,634,635,637,640,643,644,645,646,648,650,651,653,654,656,657,665,666,668,671,672,676,677,680,682,686,687,688,689,691,693,695,715,732,736,747,749,755,756,757,758,759,760,761,762,763,764,765,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,806,807,808,809,810,812,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,834,835,840,841,842,843,844,845,846,848,849,851,854,855,856,857,858,859,860,863,864,865,866,867,868,869,873,874,875,876,879,882,883,884,885,888,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,928,932,933,934,935,936,940,941,944,947,948,949,950,951,953,954,959,960,962,963,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1021,1022,1023,1024,1025,1026,1027,1028,1030,1034,1035,1036,1037,1038,1040,1041,1042,1044,1045,1046,1047,1049,1050,1051,1052,1053,1054,1055,1056,1057,1058,1059,1060,1061,1062,1063,1064,1065,1066,1067,1068,1069,1070,1071,1072,1073,1074,1075,1076,1077,1078,1079,1080,1081,1082,1083,1084,1085,1086,1087,1088,1089,1090,1091,1092,1093,1094,1095,1096,1097,1098,1099,1100,1101,1102,1103,1104,1105,1106,1107,1108,1109,1110,1111,1112,1113,1114,1115,1116,1117,1118,1119,1120,1121,1122,1124,1125,1126,1127,1128,1132,1133,1134,1135,1136,1137,1138,1140,1141,1142,1143,1144,1145,1146,1148,1149,1150,1151,1152,1153,1154,1155,1156,1157,1158,1159,1160,1161,1162,1163,1164,1165,1166,1167,1168,1169,1170,1171,1174,1176,1177,1178,1179,1180,1181,1182,1183,1185,1186,1187,1188,1191,1192,1193,1194,1195,1196,1197,1198,1199,1200,1201,1202,1203,1204,1205,1206,1207,1208,1209,1210,1211,1212,1213,1214,1215,1216,1217,1218,1219,1220,1221,1222,1223,1224,1225,1226,1227,1228,1229,1230,1232,1233,1234,1235,1236,1237,1238,1239,1240,1241,1242,1245,1246,1247,1248,1249,1250,1251,1252,1253,1256,1257,1258,1259,1260,1261,1262,1263,1264,1265,1266,1267,1268,1269,1270,1271,1272,1273,1276,1277,1278,1279,1280,1283,1284,1285,1286,1287,1288,1289,1290,1291,1292,1293,1294,1295,1296,1297,1298,1299,1300,1301,1303,1306,1307,1309,1310,1311,1312,1313,1314,1315,1316,1317,1318,1319,1320,1325,1329,1330,1331,1332,1333,1336,1337,1341,1342,1343,1345,1346,1347,1348,1349,1350,1351,1352,1353,1354,1355,1356,1357,1358,1359,1360,1361,1362,1363,1364,1365,1370,1371,1372,1373,1374,1375,1376,1377,1379,1382,1383,1384,1386,1387,1388,1394,1395,1396,1397,1398,1399,1400,1401,1402,1407,1409,1410,1411,1412,1413,1414,1415,1416,1417,1418,1419,1420,1421,1422,1423,1424,1425,1426,1427,1428,1431,1432,1433,1434,1435,1436,1437,1438,1439,1440,1441,1445,1446,1447,1448,1450,1454,1455,1456,1457,1458,1459,1460,1461,1462,1463,1464,1465,1466,1467,1468,1469,1470,1471,1472,1473,1474,1475,1476,1477,1478,1479,1485,1487,1488,1489,1490,1493,1494,1495,1496,1497,1498,1501,1502,1503,1505,1506,1507,1509,1511,1512,1513,1514,1515,1516,1517,1518,1520,1527,1528,1529,1530,1531,1537,1541,1543,1544,1547
Got recipients from releasechannel. List now: 2,4,6,19,39,59,86,90,119,120,140,183,185,192,205,237,254,256,263,289,304,318,382,389,396,397,398,399,400,401,402,403,421,422,479,481,489,491,492,493,494,495,496,497,499,503,504,505,506,507,508,509,510,511,512,513,514,516,517,518,519,520,521,522,525,526,527,528,530,531,532,537,538,541,544,547,548,549,550,551,552,553,554,557,560,561,562,564,565,566,567,569,570,572,573,575,578,579,583,586,593,594,597,599,600,601,603,604,605,606,607,608,610,612,613,615,616,619,620,621,623,627,629,630,631,632,633,634,635,637,640,643,644,645,646,648,650,651,653,654,656,657,665,666,668,671,672,676,677,680,682,686,687,688,689,691,693,695,715,732,736,747,749,755,756,757,758,759,760,761,762,763,764,765,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,806,807,808,809,810,812,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,834,835,840,841,842,843,844,845,846,848,849,851,854,855,856,857,858,859,860,863,864,865,866,867,868,869,873,874,875,876,879,882,883,884,885,888,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,928,932,933,934,935,936,940,941,944,947,948,949,950,951,953,954,959,960,962,963,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1021,1022,1023,1024,1025,1026,1027,1028,1030,1034,1035,1036,1037,1038,1040,1041,1042,1044,1045,1046,1047,1049,1050,1051,1052,1053,1054,1055,1056,1057,1058,1059,1060,1061,1062,1063,1064,1065,1066,1067,1068,1069,1070,1071,1072,1073,1074,1075,1076,1077,1078,1079,1080,1081,1082,1083,1084,1085,1086,1087,1088,1089,1090,1091,1092,1093,1094,1095,1096,1097,1098,1099,1100,1101,1102,1103,1104,1105,1106,1107,1108,1109,1110,1111,1112,1113,1114,1115,1116,1117,1118,1119,1120,1121,1122,1124,1125,1126,1127,1128,1132,1133,1134,1135,1136,1137,1138,1140,1141,1142,1143,1144,1145,1146,1148,1149,1150,1151,1152,1153,1154,1155,1156,1157,1158,1159,1160,1161,1162,1163,1164,1165,1166,1167,1168,1169,1170,1171,1174,1176,1177,1178,1179,1180,1181,1182,1183,1185,1186,1187,1188,1191,1192,1193,1194,1195,1196,1197,1198,1199,1200,1201,1202,1203,1204,1205,1206,1207,1208,1209,1210,1211,1212,1213,1214,1215,1216,1217,1218,1219,1220,1221,1222,1223,1224,1225,1226,1227,1228,1229,1230,1232,1233,1234,1235,1236,1237,1238,1239,1240,1241,1242,1245,1246,1247,1248,1249,1250,1251,1252,1253,1256,1257,1258,1259,1260,1261,1262,1263,1264,1265,1266,1267,1268,1269,1270,1271,1272,1273,1276,1277,1278,1279,1280,1283,1284,1285,1286,1287,1288,1289,1290,1291,1292,1293,1294,1295,1296,1297,1298,1299,1300,1301,1303,1306,1307,1309,1310,1311,1312,1313,1314,1315,1316,1317,1318,1319,1320,1325,1329,1330,1331,1332,1333,1336,1337,1341,1342,1343,1345,1346,1347,1348,1349,1350,1351,1352,1353,1354,1355,1356,1357,1358,1359,1360,1361,1362,1363,1364,1365,1370,1371,1372,1373,1374,1375,1376,1377,1379,1382,1383,1384,1386,1387,1388,1394,1395,1396,1397,1398,1399,1400,1401,1402,1407,1409,1410,1411,1412,1413,1414,1415,1416,1417,1418,1419,1420,1421,1422,1423,1424,1425,1426,1427,1428,1431,1432,1433,1434,1435,1436,1437,1438,1439,1440,1441,1445,1446,1447,1448,1450,1454,1455,1456,1457,1458,1459,1460,1461,1462,1463,1464,1465,1466,1467,1468,1469,1470,1471,1472,1473,1474,1475,1476,1477,1478,1479,1485,1487,1488,1489,1490,1493,1494,1495,1496,1497,1498,1501,1502,1503,1505,1506,1507,1509,1511,1512,1513,1514,1515,1516,1517,1518,1520,1527,1528,1529,1530,1531,1537,1541,1543,1544,1547
Running query: "DELETE FROM recipient WHERE _id NOT IN (SELECT DISTINCT from_recipient_id FROM message UNION SELECT DISTINCT quote_author FROM message WHERE quote_author IS NOT NULL UNION SELECT DISTINCT to_recipient_id FROM message UNION SELECT DISTINCT recipient_id FROM mention UNION SELECT DISTINCT author_id FROM reaction UNION SELECT DISTINCT recipient_id FROM story_sends UNION SELECT * FROM (VALUES(2),(4),(6),(19),(39),(59),(86),(90),(119),(120),(140),(183),(185),(192),(205),(237),(254),(256),(263),(289),(304),(318),(382),(389),(396),(397),(398),(399),(400),(401),(402),(403),(421),(422),(479),(481),(489),(491),(492),(493),(494),(495),(496),(497),(499),(503),(504),(505),(506),(507),(508),(509),(510),(511),(512),(513),(514),(516),(517),(518),(519),(520),(521),(522),(525),(526),(527),(528),(530),(531),(532),(537),(538),(541),(544),(547),(548),(549),(550),(551),(552),(553),(554),(557),(560),(561),(562),(564),(565),(566),(567),(569),(570),(572),(573),(575),(578),(579),(583),(586),(593),(594),(597),(599),(600),(601),(603),(604),(605),(606),(607),(608),(610),(612),(613),(615),(616),(619),(620),(621),(623),(627),(629),(630),(631),(632),(633),(634),(635),(637),(640),(643),(644),(645),(646),(648),(650),(651),(653),(654),(656),(657),(665),(666),(668),(671),(672),(676),(677),(680),(682),(686),(687),(688),(689),(691),(693),(695),(715),(732),(736),(747),(749),(755),(756),(757),(758),(759),(760),(761),(762),(763),(764),(765),(767),(768),(769),(770),(771),(772),(773),(774),(775),(776),(777),(778),(779),(780),(781),(782),(783),(784),(785),(786),(787),(788),(789),(790),(791),(792),(793),(794),(795),(796),(797),(798),(799),(800),(801),(802),(803),(804),(806),(807),(808),(809),(810),(812),(816),(817),(818),(819),(820),(821),(822),(823),(824),(825),(826),(827),(828),(829),(830),(831),(832),(834),(835),(840),(841),(842),(843),(844),(845),(846),(848),(849),(851),(854),(855),(856),(857),(858),(859),(860),(863),(864),(865),(866),(867),(868),(869),(873),(874),(875),(876),(879),(882),(883),(884),(885),(888),(890),(891),(892),(893),(894),(895),(896),(897),(898),(899),(900),(901),(902),(903),(904),(905),(906),(907),(908),(909),(910),(911),(912),(913),(914),(915),(916),(917),(918),(919),(920),(921),(922),(923),(924),(925),(926),(928),(932),(933),(934),(935),(936),(940),(941),(944),(947),(948),(949),(950),(951),(953),(954),(959),(960),(962),(963),(965),(966),(967),(968),(969),(970),(971),(972),(973),(974),(975),(976),(977),(978),(979),(980),(981),(982),(983),(984),(985),(986),(987),(988),(989),(990),(991),(992),(993),(994),(995),(996),(997),(998),(999),(1000),(1001),(1002),(1003),(1004),(1005),(1006),(1007),(1008),(1009),(1010),(1011),(1012),(1013),(1014),(1015),(1016),(1017),(1018),(1021),(1022),(1023),(1024),(1025),(1026),(1027),(1028),(1030),(1034),(1035),(1036),(1037),(1038),(1040),(1041),(1042),(1044),(1045),(1046),(1047),(1049),(1050),(1051),(1052),(1053),(1054),(1055),(1056),(1057),(1058),(1059),(1060),(1061),(1062),(1063),(1064),(1065),(1066),(1067),(1068),(1069),(1070),(1071),(1072),(1073),(1074),(1075),(1076),(1077),(1078),(1079),(1080),(1081),(1082),(1083),(1084),(1085),(1086),(1087),(1088),(1089),(1090),(1091),(1092),(1093),(1094),(1095),(1096),(1097),(1098),(1099),(1100),(1101),(1102),(1103),(1104),(1105),(1106),(1107),(1108),(1109),(1110),(1111),(1112),(1113),(1114),(1115),(1116),(1117),(1118),(1119),(1120),(1121),(1122),(1124),(1125),(1126),(1127),(1128),(1132),(1133),(1134),(1135),(1136),(1137),(1138),(1140),(1141),(1142),(1143),(1144),(1145),(1146),(1148),(1149),(1150),(1151),(1152),(1153),(1154),(1155),(1156),(1157),(1158),(1159),(1160),(1161),(1162),(1163),(1164),(1165),(1166),(1167),(1168),(1169),(1170),(1171),(1174),(1176),(1177),(1178),(1179),(1180),(1181),(1182),(1183),(1185),(1186),(1187),(1188),(1191),(1192),(1193),(1194),(1195),(1196),(1197),(1198),(1199),(1200),(1201),(1202),(1203),(1204),(1205),(1206),(1207),(1208),(1209),(1210),(1211),(1212),(1213),(1214),(1215),(1216),(1217),(1218),(1219),(1220),(1221),(1222),(1223),(1224),(1225),(1226),(1227),(1228),(1229),(1230),(1232),(1233),(1234),(1235),(1236),(1237),(1238),(1239),(1240),(1241),(1242),(1245),(1246),(1247),(1248),(1249),(1250),(1251),(1252),(1253),(1256),(1257),(1258),(1259),(1260),(1261),(1262),(1263),(1264),(1265),(1266),(1267),(1268),(1269),(1270),(1271),(1272),(1273),(1276),(1277),(1278),(1279),(1280),(1283),(1284),(1285),(1286),(1287),(1288),(1289),(1290),(1291),(1292),(1293),(1294),(1295),(1296),(1297),(1298),(1299),(1300),(1301),(1303),(1306),(1307),(1309),(1310),(1311),(1312),(1313),(1314),(1315),(1316),(1317),(1318),(1319),(1320),(1325),(1329),(1330),(1331),(1332),(1333),(1336),(1337),(1341),(1342),(1343),(1345),(1346),(1347),(1348),(1349),(1350),(1351),(1352),(1353),(1354),(1355),(1356),(1357),(1358),(1359),(1360),(1361),(1362),(1363),(1364),(1365),(1370),(1371),(1372),(1373),(1374),(1375),(1376),(1377),(1379),(1382),(1383),(1384),(1386),(1387),(1388),(1394),(1395),(1396),(1397),(1398),(1399),(1400),(1401),(1402),(1407),(1409),(1410),(1411),(1412),(1413),(1414),(1415),(1416),(1417),(1418),(1419),(1420),(1421),(1422),(1423),(1424),(1425),(1426),(1427),(1428),(1431),(1432),(1433),(1434),(1435),(1436),(1437),(1438),(1439),(1440),(1441),(1445),(1446),(1447),(1448),(1450),(1454),(1455),(1456),(1457),(1458),(1459),(1460),(1461),(1462),(1463),(1464),(1465),(1466),(1467),(1468),(1469),(1470),(1471),(1472),(1473),(1474),(1475),(1476),(1477),(1478),(1479),(1485),(1487),(1488),(1489),(1490),(1493),(1494),(1495),(1496),(1497),(1498),(1501),(1502),(1503),(1505),(1506),(1507),(1509),(1511),(1512),(1513),(1514),(1515),(1516),(1517),(1518),(1520),(1527),(1528),(1529),(1530),(1531),(1537),(1541),(1543),(1544),(1547)) UNION SELECT DISTINCT recipient_id FROM thread) RETURNING _id,distribution_list_id"
  Deleted 457 unreferenced recipients
  Deleting unneeded notification profiles entries...
  Deleting unused avatars...
  Deleting unused attachments...
  Delete others from 'identities'
  Deleting group_receipts entries from deleted messages...
  Deleting group_receipts from non-existing recipients
  Deleting drafts from deleted threads...
  Deleting remapped recipients for non existing recipients
  Vacuuming database

Exporting backup to 'signal-2024-02-14-01-37-39_noattachments.backup'
Writing HeaderFrame...
Writing DatabaseVersionFrame...
Writing SqlStatementFrame(s)...
  Dealing with table 'attachment'... 487/19225 entries...
[Warning]: Attachment data not found (rowid: 600, uniqueid: -1)
  Dealing with table 'attachment'... 521/19225 entries...
[Warning]: Attachment data not found (rowid: 647, uniqueid: -1)
  Dealing with table 'attachment'... 595/19225 entries...
[Warning]: Attachment data not found (rowid: 777, uniqueid: -1)
  Dealing with table 'attachment'... 596/19225 entries...
[Warning]: Attachment data not found (rowid: 778, uniqueid: -1)
  Dealing with table 'attachment'... 628/19225 entries...
[Warning]: Attachment data not found (rowid: 817, uniqueid: -1)
  Dealing with table 'attachment'... 848/19225 entries...
[Warning]: Attachment data not found (rowid: 1060, uniqueid: -1)
  Dealing with table 'attachment'... 867/19225 entries...
[Warning]: Attachment data not found (rowid: 1082, uniqueid: -1)
  Dealing with table 'attachment'... 881/19225 entries...
[Warning]: Attachment data not found (rowid: 1099, uniqueid: -1)
  Dealing with table 'attachment'... 1305/19225 entries...
[Warning]: Attachment data not found (rowid: 1625, uniqueid: -1)
  Dealing with table 'attachment'... 1331/19225 entries...
[Warning]: Attachment data not found (rowid: 1653, uniqueid: -1)
  Dealing with table 'attachment'... 1332/19225 entries...
[Warning]: Attachment data not found (rowid: 1654, uniqueid: -1)
  Dealing with table 'attachment'... 1333/19225 entries...
[Warning]: Attachment data not found (rowid: 1655, uniqueid: -1)
  Dealing with table 'attachment'... 1351/19225 entries...
[Warning]: Attachment data not found (rowid: 1676, uniqueid: -1)
  Dealing with table 'attachment'... 1460/19225 entries...
[Warning]: Attachment data not found (rowid: 1817, uniqueid: -1)
  Dealing with table 'attachment'... 5627/19225 entries...
[Warning]: Attachment data not found (rowid: 7108, uniqueid: -1)
  Dealing with table 'attachment'... 6053/19225 entries...
[Warning]: Attachment data not found (rowid: 7594, uniqueid: -1)
  Dealing with table 'attachment'... 6063/19225 entries...
[Warning]: Attachment data not found (rowid: 7604, uniqueid: -1)
  Dealing with table 'attachment'... 6082/19225 entries...
[Warning]: Attachment data not found (rowid: 7629, uniqueid: -1)
  Dealing with table 'attachment'... 6378/19225 entries...
[Warning]: Attachment data not found (rowid: 7987, uniqueid: -1)
  Dealing with table 'attachment'... 6702/19225 entries...
[Warning]: Attachment data not found (rowid: 8384, uniqueid: -1)
  Dealing with table 'attachment'... 11604/19225 entries...
[Warning]: Attachment data not found (rowid: 13746, uniqueid: -1)
  Dealing with table 'attachment'... 11791/19225 entries...
[Warning]: Attachment data not found (rowid: 13939, uniqueid: -1)
  Dealing with table 'attachment'... 16434/19225 entries...
[Warning]: Attachment data not found (rowid: 18878, uniqueid: -1)
  Dealing with table 'attachment'... 16946/19225 entries...
[Warning]: Attachment data not found (rowid: 19413, uniqueid: -1)
  Dealing with table 'attachment'... 17628/19225 entries...
[Warning]: Attachment data not found (rowid: 20129, uniqueid: -1)
  Dealing with table 'attachment'... 17958/19225 entries...
[Warning]: Attachment data not found (rowid: 20467, uniqueid: -1)
  Dealing with table 'attachment'... 18185/19225 entries...
[Warning]: Attachment data not found (rowid: 20702, uniqueid: -1)
  Dealing with table 'attachment'... 18690/19225 entries...
[Warning]: Attachment data not found (rowid: 21243, uniqueid: -1)
  Dealing with table 'attachment'... 18857/19225 entries...
[Warning]: Attachment data not found (rowid: 21418, uniqueid: -1)
  Dealing with table 'attachment'... 18890/19225 entries...
[Warning]: Attachment data not found (rowid: 21451, uniqueid: -1)
  Dealing with table 'attachment'... 18996/19225 entries...
[Warning]: Attachment data not found (rowid: 21561, uniqueid: -1)
  Dealing with table 'attachment'... 19075/19225 entries...
[Warning]: Attachment data not found (rowid: 21646, uniqueid: -1)
  Dealing with table 'attachment'... 19123/19225 entries...
[Warning]: Attachment data not found (rowid: 21694, uniqueid: -1)
  Dealing with table 'attachment'... 19225/19225 entries...done
  Dealing with table 'avatar_picker'... 0/0 entries...done
  Dealing with table 'recipient'... 1079/1079 entries...done
  Dealing with table 'thread'... 574/574 entries...done
  Dealing with table 'message'... 123492/123492 entries...done
  Dealing with table 'call'... 827/827 entries...done
  Dealing with table 'call_link'... 0/0 entries...done
  Dealing with table 'cds'... 1376/1376 entries...done
  Dealing with table 'chat_colors'... 1/1 entries...done
  Dealing with table 'distribution_list'... 1/1 entries...done
  Dealing with table 'distribution_list_member'... 1/1 entries...done
  Dealing with table 'donation_receipt'... 0/0 entries...done
  Dealing with table 'drafts'... 16/16 entries...done
  Dealing with table 'emoji_search'... 0/0 entries...done
  Dealing with table 'groups'... 91/91 entries...done
  Dealing with table 'group_membership'... 1278/1278 entries...done
  Dealing with table 'group_receipts'... 82417/82417 entries...done
  Dealing with table 'identities'... 0/0 entries...done
  Dealing with table 'kyber_prekey'... 540/540 entries...done
  Dealing with table 'mention'... 790/790 entries...done
  Dealing with table 'msl_payload'... 2383/2383 entries...done
  Dealing with table 'msl_message'... 2383/2383 entries...done
  Dealing with table 'msl_recipient'... 5123/5123 entries...done
  Dealing with table 'notification_profile'... 0/0 entries...done
  Dealing with table 'notification_profile_allowed_members'... 0/0 entries...done
  Dealing with table 'notification_profile_schedule'... 0/0 entries...done
  Dealing with table 'payments'... 0/0 entries...done
  Dealing with table 'pending_pni_signature_message'... 0/0 entries...done
  Dealing with table 'pending_retry_receipts'... 0/0 entries...done
  Dealing with table 'reaction'... 14466/14466 entries...done
  Dealing with table 'remapped_recipients'... 11/11 entries...done
  Dealing with table 'remapped_threads'... 0/0 entries...done
  Dealing with table 'remote_megaphone'... 11/11 entries...done
  Dealing with table 'sender_key_shared'... 0/0 entries...done
  Dealing with table 'sender_keys'... 0/0 entries...done
  Dealing with table 'sticker'... 103/103 entries...done
  Dealing with table 'storage_key'... 4/4 entries...done
  Dealing with table 'story_sends'... 1/1 entries...done
Writing SharedPrefFrame(s)...
Writing KeyValueFrame(s)...
Writing Avatars...
Writing EndFrame...
Done! Wrote 5191004796 bytes.
hrehfeld commented 6 months ago

Oh, and sorry, I accidentally deleted the cmdline from the original post:

$ signalbackup-tools --output signal-2024-02-14-01-37-39_noattachments.backup -i signal-2024-02-14-01-37-39.backup -p 'foo' --deleteattachments  --opassphrase 'foo' --onlylargerthan 1000000 --onlyolderthan '2023-01-01 00:00:00' --prependbody "(One or more media attachments for this message were deleted)"
bepaald commented 6 months ago

Thanks for reporting back.

This is such a weird thing to me. Looking just at the output between Deleting unreferenced recipient entries... and Deleted 457 unreferenced recipients, this is where it crashed in the old version (it never reached that second statement). All this code lives in cleandatabasebymessagse.cc, between line 144 and 259. And — as still can be seen in the commit history (https://github.com/bepaald/signalbackup-tools/commits/master/signalbackup/cleandatabasebymessages.cc) — I did not change anything in that part of the code, or any function called by that code, except some output messages (and white space changes).

So, it's a complete and utter mystery what's going on here. I'm very happy the new version worked for you, but it does make it that much harder to figure out what was wrong in the previous version.

I would love to know if the resulting backup was successfully restored and fully working as expected (assuming you plan on restoring it to phone). If so, I think I'm going to be closing this issue since I won't have anything to go on anymore (and the problem may even have been solved somehow).

But please do let me know if there is anything wrong or incorrect.

Thanks again!

hrehfeld commented 6 months ago

Stack corruption?

25 Feb 2024 21:03:31 bepaald @.***>:

Thanks for reporting back.

This is such a weird thing to me. Looking just at the output between Deleting unreferenced recipient entries... and Deleted 457 unreferenced recipients, this is where it crashed in the old version (it never reached that second statement). All this code lives in cleandatabasebymessagse.cc[https://github.com/bepaald/signalbackup-tools/blob/master/signalbackup/cleandatabasebymessages.cc], between line 144 and 259. And — as still can be seen in the commit history (https://github.com/bepaald/signalbackup-tools/commits/master/signalbackup/cleandatabasebymessages.cc) — I did not change /anything/ in that part of the code, or any function called by that code, except some output messages (and white space changes).

So, it's a complete and utter mystery what's going on here. I'm very happy the new version worked for you, but it does make it that much harder to figure out what was wrong in the previous version.

I would love to know if the resulting backup was successfully restored and fully working as expected (assuming you plan on restoring it to phone). If so, I think I'm going to be closing this issue since I won't have anything to go on anymore (and the problem may even have been solved somehow).

But please do let me know if there is anything wrong or incorrect.

Thanks again!

— Reply to this email directly, view it on GitHub[https://github.com/bepaald/signalbackup-tools/issues/190#issuecomment-1963045451], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AAATX5ZO2NSADIV2IAJCZOLYVOKIPAVCNFSM6AAAAABDXRPQ5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRTGA2DKNBVGE]. You are receiving this because you authored the thread. [Tracking image][https://github.com/notifications/beacon/AAATX5YDOJJUXZTFZFVAKGTYVOKIPA5CNFSM6AAAAABDXRPQ5CWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTVAGZEW.gif]

bepaald commented 6 months ago

Stack corruption?

Given how little I know of this, I'd suppose it's possible. But wouldn't that more likely simply result in a segmentation fault? As far as I know a std::bad_alloc is usually just trying to allocate more memory than is available. Either because the system is actually low on resources or (more often) a programming bug where on tries to allocate hundreds of GB of memory (for example, accidentally requesting a negative number of bytes interpreted as positive).

If you are able and willing (please do not feel obligated), it would be great if you could do a little investigating for me. This would mean getting the old code (if you don't have it anymore), compiling with some special flags, and running the same command through a tool like valgrind. That would very likely give a lot of information about the crash (probably even an exact line number). If you want to try this (and again, please don't feel like you have to), let me know and I can try to write up some instructions if you need them. If you already know all about that stuff or have another favorite debugging tool (like gdb, which I know nothing about) that's of course also fine.

Thanks!

hrehfeld commented 6 months ago

Yeah, sure, I can debug a little if you tell me what to do.

bepaald commented 6 months ago

Thanks, that is very kind of you. Some instructions:

  1. Get the old, crashing, version of this tool, if you don't have it anymore: 
    [~] $ mkdir tmpdebug
[~] $ cd tmpdebug/
total 0
[~/tmpdebug] $ git clone https://github.com/bepaald/signalbackup-tools
Cloning into 'signalbackup-tools'...
remote: Enumerating objects: 6764, done.
remote: Counting objects: 100% (626/626), done.
remote: Compressing objects: 100% (136/136), done.
remote: Total 6764 (delta 506), reused 586 (delta 488), pack-reused 6138
Receiving objects: 100% (6764/6764), 2.16 MiB | 10.10 MiB/s, done.
Resolving deltas: 100% (5367/5367), done.
[~/tmpdebug] $ cd signalbackup-tools/
[~/tmpdebug/signalbackup-tools] $ git reset --hard 7ec05ae
HEAD is now at 7ec05ae Fix build: forgot to remove function argument
[~/tmpdebug/signalbackup-tools] $
  2. Build it with debugging symbols present. You could do this simply by running g++ -Wall -Wextra -pedantic -std=c++2a -O1 -g */*.cc *.cc -o signalbackup-tools -lcrypto -lsqlite3. Do note that compiling this way takes a bit of time (not hours, but certainly minutes). If you were using the one of the provided buildscripts before, or the cmake file, you could adjust those instead: the idea is to remove the -O3, -s, -march=native, and -flto options (if present), and add -g and -O1.
  3. At this point, you could check to make sure the crash still happens by running the original command. Output should be identical to your first message (including the version source version 20240222.164410).
  4. If you do not have valgrind installed, install it. instructions will vary with OS, I assume you know how it works for your system (but let me know if not).
  5. Run the exact same command that causes the crash but prepend valgrind --leak-check=full. So: 
    [~/tmpdebug/signalbackup-tools] $ valgrind --leak-check=full ./signalbackup-tools --output signal-2024-02-14-01-37-39_noattachments.backup -i signal-2024-02-14-01-37-39.backup -p 'foo' --deleteattachments  --opassphrase 'foo' --onlylargerthan 1000000 --onlyolderthan '2023-01-01 00:00:00' --prependbody "(One or more media attachments for this message were deleted)"

The tool will run quite a bit slower through valgrind, but hopefully when the crash happens it will spit out a lot of useful information. Again I would like all output after the input is fully read. Note valgrind will print its output to stderr so if you'd want redirect output to a file, you need to do [command] > OUTPUT.txt 2>&1.

Let me know if anythings not clear. If any of the above instructions seem wrong and you feel you know better, adjust accordingly. No hurry with any of this, whenever you have time.

Thank you so much.

hrehfeld commented 6 months ago

no need to thank me, you're the one creating awesome OS software ;-)

Sadly, I can't reproduce now, everything just works. I was using the AUR package before, which produced this binary (which still reproduces last time i checked):

signalbackup-tools-git-20240222.3.r0.g7ec05ae-1-x86_64.pkg.zip

I'm pretty sure it was on the revision you mention, because before rebuilding that's what the pull command showed.

Also, I had to do a system update inbetween, because valgrind couldn't find glibc debug symbols. So maybe something got updated that was causing the bad alloc?

I also tried building with BUILDSCRIPT_MULTI*, adding -g at first, and then without -g, both ALSO just work and are what the AUR package uses.

 *** Starting log: 2024-02-26 18:58:30 *** 
signalbackup-tools (./signalbackup-tools) source version 20240222.164410
BACKUPFILE VERSION: 1
BACKUPFILE SIZE: 9435042329
COUNTER: 1412936055
Reading backup file...
FRAME 279459 (100.0%)... 
Read entire backup file...
done!
Database version: 219
Deleted: 1416 'part'-entries.
  Deleting unused attachments...
cleanDatabaseByMessages
  Deleting attachment entries from 'attachment' not belonging to remaining message entries
  Deleting other threads from 'thread'...
updateThreadsEntries
  Dealing with thread id: 4, 9, 8, 10, 11, 63, 41, 20, 24, 96, 137, 49, 43, 42, 73, 107, 55, 60, 36, 201, 12, 14, 23, 16, 67, 13, 84, 30, 46, 54, 136, 219, 173, 3, 57, 37, 76, 34, 15, 66, 78, 25, 17, 90, 59, 62, 97, 222, 29, 117, 147, 83, 56, 141, 52, 58, 53, 45, 32, 134, 449, 138, 208, 2, 139, 101, 22, 51, 135, 588, 27, 33, 131, 75, 44, 35, 111, 61, 19, 21, 103, 48, 31, 18, 50, 70, 47, 26, 39, 92, 82, 194, 140, 28, 40, 38, 104, 95, 110, 65, 68, 69, 64, 7, 71, 72, 98, 77, 74, 79, 80, 106, 86, 166, 127, 85, 6, 5, 81, 87, 91, 88, 89, 256, 115, 93, 94, 247, 102, 105, 109, 299, 112, 113, 114, 118, 262, 119, 120, 121, 122, 123, 124, 125, 126, 128, 129, 130, 133, 142, 143, 144, 154, 145, 198, 153, 146, 160, 170, 568, 149, 150, 148, 174, 155, 161, 171, 570, 310, 157, 396, 159, 151, 152, 205, 202, 297, 179, 156, 158, 499, 353, 497, 162, 505, 360, 163, 164, 216, 165, 167, 168, 309, 500, 257, 504, 169, 236, 172, 192, 492, 254, 559, 585, 384, 214, 512, 178, 359, 175, 191, 176, 177, 508, 231, 506, 211, 185, 361, 184, 511, 204, 180, 308, 181, 182, 183, 225, 253, 186, 187, 188, 292, 189, 190, 245, 507, 540, 195, 193, 267, 196, 215, 203, 306, 206, 307, 207, 209, 316, 212, 210, 226, 213, 483, 217, 395, 218, 220, 221, 510, 581, 223, 241, 224, 469, 533, 227, 228, 229, 230, 279, 255, 232, 233, 234, 235, 237, 238, 239, 240, 282, 242, 244, 243, 259, 246, 277, 445, 248, 252, 249, 250, 251, 258, 260, 261, 263, 265, 311, 364, 337, 535, 264, 409, 372, 354, 335, 298, 562, 453, 355, 296, 446, 560, 405, 356, 320, 427, 575, 351, 266, 268, 269, 325, 281, 294, 495, 289, 318, 276, 481, 334, 270, 295, 275, 271, 273, 556, 321, 278, 280, 367, 514, 283, 293, 286, 288, 285, 326, 287, 290, 537, 352, 305, 300, 301, 324, 302, 522, 323, 567, 342, 347, 358, 460, 447, 430, 369, 343, 435, 349, 385, 345, 388, 376, 303, 315, 402, 392, 549, 312, 313, 314, 458, 420, 317, 319, 558, 322, 582, 327, 328, 344, 329, 330, 331, 332, 538, 482, 370, 341, 440, 375, 491, 441, 336, 488, 566, 489, 443, 454, 371, 333, 338, 339, 368, 340, 555, 346, 436, 348, 365, 546, 350, 565, 397, 373, 527, 501, 433, 386, 571, 399, 462, 551, 455, 400, 378, 519, 456, 357, 580, 410, 457, 520, 415, 550, 362, 553, 503, 363, 377, 493, 485, 366, 496, 401, 434, 416, 417, 383, 374, 391, 379, 521, 380, 382, 389, 525, 513, 539, 472, 437, 544, 543, 394, 407, 593, 591, 387, 465, 536, 471, 390, 526, 418, 408, 439, 515, 393, 545, 403, 404, 428, 425, 423, 490, 406, 518, 586, 411, 426, 429, 424, 502, 498, 459, 448, 413, 422, 414, 431, 468, 421, 470, 461, 432, 578, 524, 494, 509, 451, 438, 442, 557, 592, 450, 452, 466, 463, 474, 477, 473, 480, 532, 476, 484, 464, 475, 590, 478, 554, 479, 486, 552, 487, 564, 517, 531, 528, 523, 530, 561, 529, 534, 541, 542, 587, 584, 548, 563, 597, 569, 572, 573, 574, 576, 579, 583, 589, 594, 596, 598
  Deleting entries from 'mention' not belonging to remaining mms entries
  Deleting removed groups...
  Deleting unneeded MessageSendLog entries... (0, 0, 0)
  Deleting reactions to non-existing messages... (0)
  Deleting call details from non-existing messages... (0)
  Deleting unreferenced recipient entries...
  Deleted 457 unreferenced recipients
  Deleting unneeded notification profiles entries...
  Deleting unused avatars...
  Deleting unused attachments...
  Delete others from 'identities'
  Deleting group_receipts entries from deleted messages...
  Deleting group_receipts from non-existing recipients
  Deleting drafts from deleted threads...
  Deleting remapped recipients for non existing recipients
  Vacuuming database

Exporting backup to 'signal-2024-02-14-01-37-39_noattachments.backup'
Writing HeaderFrame...
Writing DatabaseVersionFrame...
Writing SqlStatementFrame(s)...
  Dealing with table 'attachment'... 487/19225 entries...
[Warning]: Attachment data not found (rowid: 600, uniqueid: -1)
  Dealing with table 'attachment'... 521/19225 entries...
[Warning]: Attachment data not found (rowid: 647, uniqueid: -1)
  Dealing with table 'attachment'... 595/19225 entries...
[Warning]: Attachment data not found (rowid: 777, uniqueid: -1)
  Dealing with table 'attachment'... 596/19225 entries...
[Warning]: Attachment data not found (rowid: 778, uniqueid: -1)
  Dealing with table 'attachment'... 628/19225 entries...
[Warning]: Attachment data not found (rowid: 817, uniqueid: -1)
  Dealing with table 'attachment'... 848/19225 entries...
[Warning]: Attachment data not found (rowid: 1060, uniqueid: -1)
  Dealing with table 'attachment'... 867/19225 entries...
[Warning]: Attachment data not found (rowid: 1082, uniqueid: -1)
  Dealing with table 'attachment'... 881/19225 entries...
[Warning]: Attachment data not found (rowid: 1099, uniqueid: -1)
  Dealing with table 'attachment'... 1305/19225 entries...
[Warning]: Attachment data not found (rowid: 1625, uniqueid: -1)
  Dealing with table 'attachment'... 1331/19225 entries...
[Warning]: Attachment data not found (rowid: 1653, uniqueid: -1)
  Dealing with table 'attachment'... 1332/19225 entries...
[Warning]: Attachment data not found (rowid: 1654, uniqueid: -1)
  Dealing with table 'attachment'... 1333/19225 entries...
[Warning]: Attachment data not found (rowid: 1655, uniqueid: -1)
  Dealing with table 'attachment'... 1351/19225 entries...
[Warning]: Attachment data not found (rowid: 1676, uniqueid: -1)
  Dealing with table 'attachment'... 1460/19225 entries...
[Warning]: Attachment data not found (rowid: 1817, uniqueid: -1)
  Dealing with table 'attachment'... 5627/19225 entries...
[Warning]: Attachment data not found (rowid: 7108, uniqueid: -1)
  Dealing with table 'attachment'... 6053/19225 entries...
[Warning]: Attachment data not found (rowid: 7594, uniqueid: -1)
  Dealing with table 'attachment'... 6063/19225 entries...
[Warning]: Attachment data not found (rowid: 7604, uniqueid: -1)
  Dealing with table 'attachment'... 6082/19225 entries...
[Warning]: Attachment data not found (rowid: 7629, uniqueid: -1)
  Dealing with table 'attachment'... 6378/19225 entries...
[Warning]: Attachment data not found (rowid: 7987, uniqueid: -1)
  Dealing with table 'attachment'... 6702/19225 entries...
[Warning]: Attachment data not found (rowid: 8384, uniqueid: -1)
  Dealing with table 'attachment'... 11604/19225 entries...
[Warning]: Attachment data not found (rowid: 13746, uniqueid: -1)
  Dealing with table 'attachment'... 11791/19225 entries...
[Warning]: Attachment data not found (rowid: 13939, uniqueid: -1)
  Dealing with table 'attachment'... 16434/19225 entries...
[Warning]: Attachment data not found (rowid: 18878, uniqueid: -1)
  Dealing with table 'attachment'... 16946/19225 entries...
[Warning]: Attachment data not found (rowid: 19413, uniqueid: -1)
  Dealing with table 'attachment'... 17628/19225 entries...
[Warning]: Attachment data not found (rowid: 20129, uniqueid: -1)
  Dealing with table 'attachment'... 17958/19225 entries...
[Warning]: Attachment data not found (rowid: 20467, uniqueid: -1)
  Dealing with table 'attachment'... 18185/19225 entries...
[Warning]: Attachment data not found (rowid: 20702, uniqueid: -1)
  Dealing with table 'attachment'... 18690/19225 entries...
[Warning]: Attachment data not found (rowid: 21243, uniqueid: -1)
  Dealing with table 'attachment'... 18857/19225 entries...
[Warning]: Attachment data not found (rowid: 21418, uniqueid: -1)
  Dealing with table 'attachment'... 18890/19225 entries...
[Warning]: Attachment data not found (rowid: 21451, uniqueid: -1)
  Dealing with table 'attachment'... 18996/19225 entries...
[Warning]: Attachment data not found (rowid: 21561, uniqueid: -1)
  Dealing with table 'attachment'... 19075/19225 entries...
[Warning]: Attachment data not found (rowid: 21646, uniqueid: -1)
  Dealing with table 'attachment'... 19123/19225 entries...
[Warning]: Attachment data not found (rowid: 21694, uniqueid: -1)
  Dealing with table 'story_sends'... 1/1 entries...done..doneeonee...done...done
Writing SharedPrefFrame(s)...
Writing KeyValueFrame(s)...
Writing Avatars...
Writing EndFrame...
Done! Wrote 5191007606 bytes.

Any further ideas how to proceed?

bepaald commented 6 months ago

Thanks for going through all that trouble.

It's strange how manually compiling (the same sources) the problem doesn't show, but when building through aur it does. I have installed the package you attached (in a VM) and with that, I can indeed also reproduce.

Any further ideas how to proceed?

Honestly, not really. I'm not really sure what to do with the debugging symbols stripped from the package. Maybe you could share some flags from your /etc/makepkg.conf? Interesting ones could be C(XX)FLAGS, LDFLAGS, BUILDENV and OPTIONS. I believe those are used when compiling the aur package, maybe one of those will allow me to reproduce on a fresh build. That should get me started on finding the problem.

EDIT maybe when you build the package from aur (using some aur-helper I presume) the compile commands are even printed to your terminal? For me it looks like this:

[...]
==> Starting build()...
g++ -c -march=native -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS  -march=native -std=c++2a keyvalueframe/statics.cc -o keyvalueframe/o/statics.o
g++ -c -march=native -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS  -march=native -std=c++2a signalbackup/tgmapcontacts.cc -o signalbackup/o/tgmapcontacts.o
g++ -c -march=native -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS  -march=native -std=c++2a signalbackup/tgbuildbody.cc -o signalbackup/o/tgbuildbody.o
g++ -c -march=native -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS  -march=native -std=c++2a signalbackup/checkdbintegrity.cc -o signalbackup/o/checkdbintegrity.o
[...]
g++ -c -march=native -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS  -march=native -std=c++2a cryptbase/getcipherandmac.cc -o cryptbase/o/getcipherandmac.o
g++ keyvalueframe/o/statics.o signalbackup/o/tgmapcontacts.o signalbackup/o/tgbuildbody.o signalbackup/o/checkdbintegrity.o signalbackup/o/mergegroups.o signalbackup/o/writeencryptedframe.o signalbackup/o/scanself.o signalbackup/o/applyranges.o signalbackup/o/prepareoutputdirectory.o signalbackup/o/scramble.o signalbackup/o/htmlescapestring.o signalbackup/o/cleandatabasebymessages.o signalbackup/o/insertattachments.o signalbackup/o/setminimumid.o signalbackup/o/htmlprepbody.o signalbackup/o/croptodates.o signalbackup/o/datetomsecssinceepoch.o signalbackup/o/updategroupmembers.o signalbackup/o/exporttofile.o signalbackup/o/setcolumnnames.o signalbackup/o/handledtgroupchangemessage.o signalbackup/o/tgimportmessages.o signalbackup/o/dtupdateprofile.o signalbackup/o/getrecipientinfofrommap.o signalbackup/o/insertrow.o signalbackup/o/getgroupv1migrationrecipients.o signalbackup/o/htmlwriteblockedlist.o signalbackup/o/listrecipients.o signalbackup/o/importwachat.o signalbackup/o/htmlgetmessageinfo.o signalbackup/o/dumpmedia.o signalbackup/o/getattachmentmetadata.o signalbackup/o/makeidsunique.o signalbackup/o/dtimportstickerpacks.o signalbackup/o/getallthreadrecipients.o signalbackup/o/croptothread.o signalbackup/o/initfromdir.o signalbackup/o/fillthreadtablefrommessages.o signalbackup/o/dtsetavatar.o signalbackup/o/getrecipientidfrom.o signalbackup/o/compactids.o signalbackup/o/importfromdesktop.o signalbackup/o/htmlwritecalllog.o signalbackup/o/exporttxt.o signalbackup/o/getdtreactions.o signalbackup/o/decodestatusmessage.o signalbackup/o/htmlwriteavatar.o signalbackup/o/statics_html.o signalbackup/o/exporttodir.o signalbackup/o/getfreedateformessage.o signalbackup/o/setfiletimestamp.o signalbackup/o/exportcsv.o signalbackup/o/handledtgroupv1migration.o signalbackup/o/migratedatabase.o signalbackup/o/listthreads.o signalbackup/o/htmlgetemojipos.o signalbackup/o/updatethreadsentries.o signalbackup/o/getcustomcolor.o signalbackup/o/updaterecipientid.o signalbackup/o/getgroupinfo.o signalbackup/o/deleteattachments.o signalbackup/o/getgroupmembers.o signalbackup/o/customs.o signalbackup/o/getminmaxusedid.o signalbackup/o/reordermmssmsids.o signalbackup/o/importcsv.o signalbackup/o/handledtcalltypemessage.o signalbackup/o/statics.o signalbackup/o/getnamefromrecipientid.o signalbackup/o/importthread.o signalbackup/o/getrecipientidfromname.o signalbackup/o/htmlwritesettings.o signalbackup/o/decodeprofilechangemessage.o signalbackup/o/remaprecipients.o signalbackup/o/htmlwriteindex.o signalbackup/o/dumpavatars.o signalbackup/o/summarize.o signalbackup/o/removedoubles.o signalbackup/o/getthreadidfromrecipient.o signalbackup/o/cleanattachments.o signalbackup/o/exporthtml.o signalbackup/o/buildsqlstatementframe.o signalbackup/o/htmlescapeurl.o signalbackup/o/gettranslatedname.o signalbackup/o/importtelegramjson.o signalbackup/o/escapexmlstring.o signalbackup/o/tgsetquote.o signalbackup/o/handlewamessage.o signalbackup/o/addsmsmessage.o signalbackup/o/tgsetbodyranges.o signalbackup/o/handledtexpirationchangemessage.o signalbackup/o/htmlwritestickerpacks.o signalbackup/o/dtsetsharedcontactsjsonstring.o signalbackup/o/htmlwrite.o signalbackup/o/initfromfile.o signalbackup/o/makefilenameunique.o signalbackup/o/sanitizefilename.o signalbackup/o/mergerecipients.o signalbackup/o/htmlwritesearchpage.o signalbackup/o/missingattachmentexpected.o signalbackup/o/dtcreaterecipient.o signalbackup/o/getgroupupdaterecipients.o signalbackup/o/insertreactions.o signalbackup/o/dumpinfoonbadframe.o signalbackup/o/dropbadframes.o signalbackup/o/dtsetcolumnnames.o signalbackup/o/tgsetattachment.o signalbackup/o/exportxml.o signalbackup/o/statics_emoji.o signalbackup/o/updategv1migrationmessage.o signalbackup/o/updatereactionauthors.o signalbackup/o/setrecipientinfo.o signalbackup/o/findrecipient.o signalbackup/o/updateavatars.o signalbackup/o/htmlwriteattachment.o signalbackup/o/dtsetmessagedeliveryreceipts.o attachmentframe/o/statics.o logger/o/isterminal.o logger/o/supportsansi.o logger/o/statics.o logger/o/outputhead.o databaseversionframe/o/statics.o endframe/o/statics.o sqlcipherdecryptor/o/getkey.o sqlcipherdecryptor/o/destructor.o sqlcipherdecryptor/o/gethmackey.o sqlcipherdecryptor/o/sqlcipherdecryptor.o sqlcipherdecryptor/o/decryptdata.o framewithattachment/o/setattachmentdata.o sharedprefframe/o/statics.o avatarframe/o/statics.o sqlstatementframe/o/statics.o sqlstatementframe/o/buildstatement.o backupframe/o/init.o memfiledb/o/statics.o sqlitedb/o/valueasstring.o sqlitedb/o/prettyprint.o sqlitedb/o/renamecolumn.o sqlitedb/o/availablewidth.o sqlitedb/o/removecolumn.o sqlitedb/o/valueasint.o sqlitedb/o/copydb.o sqlitedb/o/print.o sqlitedb/o/printlinemode.o stickerframe/o/statics.o csvreader/o/readrow.o csvreader/o/read.o o/main.o headerframe/o/statics.o basedecryptor/o/getattachment.o desktopdatabase/o/init.o reactionlist/o/setauthor.o jsondatabase/o/jsondatabase.o fileencryptor/o/init.o fileencryptor/o/encryptframe.o fileencryptor/o/fileencryptor.o fileencryptor/o/encryptattachment.o filedecryptor/o/getframe.o filedecryptor/o/getframebrute.o filedecryptor/o/filedecryptor.o filedecryptor/o/customs.o filedecryptor/o/initbackupframe.o arg/o/usage.o arg/o/arg.o cryptbase/o/getbackupkey.o cryptbase/o/getcipherandmac.o -Wl,-O2,--sort-common,--as-needed,-z,relro,-z,now -o signalbackup-tools -lcrypto -lsqlite3
==> Entering fakeroot environment...
[...]

With that, and the exact same source files, I should really be able to build a binary showing the same bad_alloc.

bepaald commented 6 months ago

Ok, just a little update:

I've ran the binary from your package through valgrind, and at the moment of the exception, it says: 

  Deleting unreferenced recipient entries...
==6688== Conditional jump or move depends on uninitialized value(s)
==6688==   at 0x484FDEE: memmove (vg_replace_strmem.c:1410)
==6688==   by 0x16A115: std::__cxx11::basic_string(char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) (in /usr/bin/signalbackup-tools)
==6688==   by 0x395251: ??? (in /usr/bin/signalbackup-tools)
==6688==   by 0x1B00B9: ??? (in /usr/bin/signalbackup-tools)
==6688==   by 0x2DFEFC: ??? (in /usr/bin/signalbackup-tools)
==6688==   by 0x16820F: ??? (in /usr/bin/signalbackup-tools)
[...]

So, the use of an uninitialized value does to me suggest a bug in the code. Unfortunately, because the binary does not have debugging symbols, the location where this happens is just ??? (as expected).

I've since tried to hunt this possible uninitialized value down in the part of the code where it must happen, looking specifically at strings, but I've not been successful. I've also attempted to use static code analysis tools (both from gcc and clang), but those have not turned anything up either (nothing related to this at least). Lastly I've again tried to build a binary exhibiting this crash myself by adding the git reset --hard 7ec05ae line to the PKGBUILD from aur and building on an otherwise completely default system. Unfortunately this also produced a working binary without any crashes.

So, since this is driving me slightly nuts, one last attempt. If willing, would you be able to reproduce a crashing binary? Assuming you initially simply installed with pikaur as such: pikaur -S signalbackup-tools-git, could try the exact same procedure again, but this time when asked if you want to edit the PKGBUILD, respond Y and add git reset --hard 7ec05ae to the build() function after the cd and right before the buildscript is run?

As far as I can tell this should produce the same binary as before, assuming you haven't touched your makepkg.conf since then*. Do you still see the crash with this version? If so, could you copy-paste the exact build commands echoed by the buildscript?

Thanks!

* (unless maybe other system updates influence the build somehow, though I'm still seeing the bad_alloc from your package on a fully updated system)

bepaald commented 5 months ago

I will keep this in mind if something similar is ever reported, for know just closing to clean up. Thanks for your efforts!