kogeler commented 10 months ago

Needs

Syncing archive nodes from scratch takes a lot of time. Also, we sometimes have sync issues like the incoming network slots issue before or the paritydb issue now. We already have the --sync warp feature, but it can't be used for archive nodes and it doesn't help to resolve all sync issues (like network issues, disaster recovery when a network doesn't work, etc.).

At the same time, networks need archive nodes. It means that users need the possibility to spin up their nodes in a reasonable time. It's why binary db snapshots are still very important for now.

How it works now

rocksdb uses a lot of files that have small sizes but paritydb uses smaller amounts of really big files. For the archive paritydb nodes, they can be table files that are more than 160-200 GB.

Big files lead to some issues:

it is not a trivial task to move really big files to or from the backup storage. any network interruptions can increase the number of retries. any retries with 200Gb files take a lot of time and expensive traffic
all files are changed very often, it doesn't allow us to use diff backups based on the modification time of separate files
any CDN cache services don't allow cache files with sizes more than 500-1000MB. it makes providing public snapshots more expensive

These things lead to the use of some additional backup systems that can split files into chunks, have their own DB of chunks, and sync them based on checksum. It takes a lot of additional CPU time. Also, it makes using snapshots more difficult for users.

The request

Is it possible to add a new CLI flag that allows restricting the size of table paritydb files for newly synchronized nodes?

BulatSaif commented 10 months ago

if we are planning to modify the db parameters it also makes sense to add a flag to enable/disable db compression, for now, for test purposes. It may be more efficient to disable compression on the DB level and use hardware compression on the disk level. Also, it can be useful for backup purposes to implement incremental backups.

hitchhooker commented 9 months ago

feels like some disclaimer in README about not to be used with --archive could be reasonable addition.

been now 3 weeks syncing ksm and getting constantly stuck on ZFS(5x2TB striped pool samsung980) / 128GB DDR5 ECC / 7950x setup inside lxcontainer

ksm paritydb archive at blockheight 15067273

-rw-r--r-- 1 polkadot polkadot    100239020 Sep 24 04:45 table_05_d0
-rw-r--r-- 1 polkadot polkadot    101064604 Sep 24 05:33 table_01_77
-rw-r--r-- 1 polkadot polkadot    101552022 Sep 24 05:33 table_01_79
-rw-r--r-- 1 polkadot polkadot    103013052 Sep 24 05:33 table_01_6e
-rw-r--r-- 1 polkadot polkadot    103275280 Sep  6 07:44 table_05_08
-rw-r--r-- 1 polkadot polkadot    104888832 Sep 24 05:32 table_01_92
-rw-r--r-- 1 polkadot polkadot    105682267 Sep 24 05:32 table_05_d1
-rw-r--r-- 1 polkadot polkadot    113358956 Sep 24 05:32 table_01_85
-rw-r--r-- 1 polkadot polkadot    113872568 Sep 24 05:32 table_01_9a
-rw-r--r-- 1 polkadot polkadot    115133760 Sep 24 05:32 table_01_96
-rw-r--r-- 1 polkadot polkadot    115980400 Sep 24 05:32 table_01_91
-rw-r--r-- 1 polkadot polkadot    117440128 Sep 24 05:09 table_05_d2
-rw-r--r-- 1 polkadot polkadot    118444120 Sep 24 05:33 table_01_73
-rw-r--r-- 1 polkadot polkadot    128520812 Sep 24 05:32 table_01_89
-rw-r--r-- 1 polkadot polkadot    128959407 Sep 24 05:32 table_05_d3
-rw-r--r-- 1 polkadot polkadot    130965972 Sep 24 05:32 table_01_87
-rw-r--r-- 1 polkadot polkadot    136836112 Sep 24 05:32 table_01_8b
-rw-r--r-- 1 polkadot polkadot    141509063 Sep 24 05:24 table_05_d4
-rw-r--r-- 1 polkadot polkadot    143115336 Sep  4 12:39 table_05_13
-rw-r--r-- 1 polkadot polkadot    147369554 Sep 24 05:33 table_01_9e
-rw-r--r-- 1 polkadot polkadot    159600624 Sep 24 05:32 table_05_d5
-rw-r--r-- 1 polkadot polkadot    168077664 Sep 24 05:32 table_01_95
-rw-r--r-- 1 polkadot polkadot    170614080 Sep 11 10:38 table_04_fe
-rw-r--r-- 1 polkadot polkadot    171363200 Sep 24 05:32 table_01_94
-rw-r--r-- 1 polkadot polkadot    177727648 Sep 24 05:25 table_05_d6
-rw-r--r-- 1 polkadot polkadot    182684608 Sep 24 05:33 table_01_45
-rw-r--r-- 1 polkadot polkadot    184711348 Sep 24 05:33 table_01_6f
-rw-r--r-- 1 polkadot polkadot    185087907 Sep 24 05:32 table_01_98
-rw-r--r-- 1 polkadot polkadot    185803296 Sep 24 05:32 table_01_9c
-rw-r--r-- 1 polkadot polkadot    188604532 Sep 24 05:32 table_01_93
-rw-r--r-- 1 polkadot polkadot    189781504 Sep 24 05:33 table_01_37
-rw-r--r-- 1 polkadot polkadot    192631578 Sep 24 05:32 table_01_99
-rw-r--r-- 1 polkadot polkadot    201425632 Sep 24 05:00 table_05_d7
-rw-r--r-- 1 polkadot polkadot    205499656 Sep 24 05:33 table_01_38
-rw-r--r-- 1 polkadot polkadot    205638948 Sep 24 05:32 table_01_9b
-rw-r--r-- 1 polkadot polkadot    215481697 Sep 24 05:32 table_01_97
-rw-r--r-- 1 polkadot polkadot    221764200 Sep 24 05:33 table_01_9f
-rw-r--r-- 1 polkadot polkadot    228739164 Sep 24 05:16 table_05_d8
-rw-r--r-- 1 polkadot polkadot    246152277 Sep 24 02:44 table_01_1e
-rw-r--r-- 1 polkadot polkadot    254752640 Sep 24 05:33 table_01_a0
-rw-r--r-- 1 polkadot polkadot    264998912 Sep 24 05:32 table_01_9d
-rw-r--r-- 1 polkadot polkadot    268451840 Sep 24 05:33 index_00_19
-rw-r--r-- 1 polkadot polkadot    268451840 Sep 24 05:33 index_04_19
-rw-r--r-- 1 polkadot polkadot    268451840 Sep 24 05:33 index_05_19
-rw-r--r-- 1 polkadot polkadot    270890016 Sep 24 05:08 table_05_d9
-rw-r--r-- 1 polkadot polkadot    284102159 Sep 24 05:33 table_01_a1
-rw-r--r-- 1 polkadot polkadot    297408816 Sep 24 05:33 table_01_a5
-rw-r--r-- 1 polkadot polkadot    311370352 Sep 24 05:33 table_01_a7
-rw-r--r-- 1 polkadot polkadot    316111168 Sep 24 05:33 table_01_3f
-rw-r--r-- 1 polkadot polkadot    317078660 Sep 24 05:32 table_05_da
-rw-r--r-- 1 polkadot polkadot    340252139 Sep 24 05:33 table_01_1f
-rw-r--r-- 1 polkadot polkadot    352810624 Sep 24 05:33 table_01_31
-rw-r--r-- 1 polkadot polkadot    353893200 Sep 24 05:33 table_01_41
-rw-r--r-- 1 polkadot polkadot    372358080 Sep 24 05:33 table_01_a6
-rw-r--r-- 1 polkadot polkadot    373266912 Sep 24 05:33 table_01_32
-rw-r--r-- 1 polkadot polkadot    379393544 Sep 24 05:17 table_05_db
-rw-r--r-- 1 polkadot polkadot    380310026 Sep 24 05:33 table_01_6c
-rw-r--r-- 1 polkadot polkadot    394520984 Sep 24 05:33 table_01_2c
-rw-r--r-- 1 polkadot polkadot    423785391 Sep 24 05:33 table_01_a9
-rw-r--r-- 1 polkadot polkadot    440887464 Sep 24 05:33 table_01_69
-rw-r--r-- 1 polkadot polkadot    449632472 Sep 24 05:32 table_05_dc
-rw-r--r-- 1 polkadot polkadot    473064024 Sep 24 05:33 table_01_43
-rw-r--r-- 1 polkadot polkadot    476559084 Sep 24 05:33 table_01_ae
-rw-r--r-- 1 polkadot polkadot    482313140 Sep 24 05:33 table_01_48
-rw-r--r-- 1 polkadot polkadot    504321648 Sep 24 05:33 table_01_5f
-rw-r--r-- 1 polkadot polkadot    508347600 Sep 24 05:33 table_01_ad
-rw-r--r-- 1 polkadot polkadot    511808126 Sep 24 05:33 table_01_30
-rw-r--r-- 1 polkadot polkadot    524567122 Sep 24 05:33 table_01_a8
-rw-r--r-- 1 polkadot polkadot    536608768 Sep 24 05:33 table_04_ff
-rw-r--r-- 1 polkadot polkadot    536887296 Sep 24 05:33 index_03_20
-rw-r--r-- 1 polkadot polkadot    537383936 Sep 24 05:16 table_05_dd
-rw-r--r-- 1 polkadot polkadot    557542231 Sep 24 05:33 table_01_71
-rw-r--r-- 1 polkadot polkadot    560580904 Sep 24 05:33 table_01_40
-rw-r--r-- 1 polkadot polkadot    567689878 Sep 24 05:33 table_01_af
-rw-r--r-- 1 polkadot polkadot    569634244 Sep 24 05:33 table_01_a4
-rw-r--r-- 1 polkadot polkadot    611801592 Sep 24 05:33 table_01_b3
-rw-r--r-- 1 polkadot polkadot    625971424 Sep 24 05:33 table_01_ac
-rw-r--r-- 1 polkadot polkadot    645076293 Sep 24 05:32 table_05_de
-rw-r--r-- 1 polkadot polkadot    685025328 Sep 24 05:33 table_01_b7
-rw-r--r-- 1 polkadot polkadot    706031200 Sep 24 05:33 table_01_ab
-rw-r--r-- 1 polkadot polkadot    716659896 Sep 24 05:33 table_01_21
-rw-r--r-- 1 polkadot polkadot    731205760 Sep 24 05:33 table_01_3a
-rw-r--r-- 1 polkadot polkadot    763543552 Sep 24 05:33 table_01_5e
-rw-r--r-- 1 polkadot polkadot    767672620 Sep 24 05:33 table_01_aa
-rw-r--r-- 1 polkadot polkadot    789105314 Sep 24 05:32 table_05_df
-rw-r--r-- 1 polkadot polkadot    789989082 Sep 24 05:33 table_01_b0
-rw-r--r-- 1 polkadot polkadot    802666872 Sep 24 05:33 table_01_b4
-rw-r--r-- 1 polkadot polkadot    865048620 Sep 24 05:33 table_01_50
-rw-r--r-- 1 polkadot polkadot    899987032 Sep 24 05:33 table_01_29
-rw-r--r-- 1 polkadot polkadot    924157904 Sep 24 05:33 table_01_b8
-rw-r--r-- 1 polkadot polkadot    934448704 Sep 24 05:33 table_01_26
-rw-r--r-- 1 polkadot polkadot    934520032 Sep 24 05:33 table_00_18
-rw-r--r-- 1 polkadot polkadot    939439521 Sep 24 05:33 table_01_b1
-rw-r--r-- 1 polkadot polkadot    945755420 Sep 24 05:33 table_01_34
-rw-r--r-- 1 polkadot polkadot    961038106 Sep 24 05:33 table_01_58
-rw-r--r-- 1 polkadot polkadot    964959526 Sep 24 05:33 table_01_b5
-rw-r--r-- 1 polkadot polkadot    966580592 Sep 24 05:33 table_01_6d
-rw-r--r-- 1 polkadot polkadot    970001086 Sep 24 05:33 table_01_5c
-rw-r--r-- 1 polkadot polkadot    982432860 Sep 24 05:33 table_01_b2
-rw-r--r-- 1 polkadot polkadot    985181568 Sep 24 05:33 table_01_b9
-rw-r--r-- 1 polkadot polkadot   1043200404 Sep 24 05:33 table_01_b6
-rw-r--r-- 1 polkadot polkadot   1066516790 Sep 24 05:33 table_01_55
-rw-r--r-- 1 polkadot polkadot   1073643456 Sep 24 05:32 table_01_bc
-rw-r--r-- 1 polkadot polkadot   1074158862 Sep 24 05:33 table_01_42
-rw-r--r-- 1 polkadot polkadot   1084156268 Sep 24 05:33 table_01_66
-rw-r--r-- 1 polkadot polkadot   1099162688 Sep 24 05:33 table_01_2e
-rw-r--r-- 1 polkadot polkadot   1171374911 Sep 24 05:32 table_01_bd
-rw-r--r-- 1 polkadot polkadot   1204753132 Sep 24 05:33 table_01_ba
-rw-r--r-- 1 polkadot polkadot   1212968960 Sep 24 05:33 table_01_3b
-rw-r--r-- 1 polkadot polkadot   1240380600 Sep  6 07:56 table_04_47
-rw-r--r-- 1 polkadot polkadot   1266141036 Sep 24 05:33 table_01_c4
-rw-r--r-- 1 polkadot polkadot   1303069280 Sep 24 05:32 table_01_bb
-rw-r--r-- 1 polkadot polkadot   1307710496 Sep 24 05:32 table_01_c1
-rw-r--r-- 1 polkadot polkadot   1323179372 Sep 24 05:32 table_01_c0
-rw-r--r-- 1 polkadot polkadot   1332247232 Sep 24 05:33 table_01_27
-rw-r--r-- 1 polkadot polkadot   1355003676 Sep 24 05:33 table_01_1d
-rw-r--r-- 1 polkadot polkadot   1363668146 Sep 24 05:33 table_01_be
-rw-r--r-- 1 polkadot polkadot   1403104832 Sep 24 05:33 table_05_e0
-rw-r--r-- 1 polkadot polkadot   1429227200 Sep 24 05:33 table_01_cd
-rw-r--r-- 1 polkadot polkadot   1473741330 Sep 24 05:33 table_01_2b
-rw-r--r-- 1 polkadot polkadot   1612447460 Sep 24 05:33 table_01_d0
-rw-r--r-- 1 polkadot polkadot   1622955664 Sep 24 05:33 table_01_4d
-rw-r--r-- 1 polkadot polkadot   1631406488 Sep 24 05:33 table_01_20
-rw-r--r-- 1 polkadot polkadot   1641516084 Sep 24 05:33 table_01_c5
-rw-r--r-- 1 polkadot polkadot   1663085440 Sep 24 05:33 table_01_36
-rw-r--r-- 1 polkadot polkadot   1736908968 Sep 24 05:32 table_01_c2
-rw-r--r-- 1 polkadot polkadot   1745664477 Sep 24 05:33 table_01_ce
-rw-r--r-- 1 polkadot polkadot   1751638873 Sep 24 05:33 table_01_c7
-rw-r--r-- 1 polkadot polkadot   1824683320 Sep 24 05:32 table_01_bf
-rw-r--r-- 1 polkadot polkadot   1880030788 Sep 24 05:32 table_01_c6
-rw-r--r-- 1 polkadot polkadot   1905441824 Sep 24 05:33 table_01_3e
-rw-r--r-- 1 polkadot polkadot   1906201937 Sep 24 05:33 table_01_d1
-rw-r--r-- 1 polkadot polkadot   1959397248 Sep 24 05:33 table_03_1a
-rw-r--r-- 1 polkadot polkadot   1986635316 Sep 24 05:33 table_01_c8
-rw-r--r-- 1 polkadot polkadot   1998513181 Sep 24 05:33 table_01_cc
-rw-r--r-- 1 polkadot polkadot   2066927428 Sep 24 05:33 table_01_64
-rw-r--r-- 1 polkadot polkadot   2102386432 Sep 24 05:33 table_01_d2
-rw-r--r-- 1 polkadot polkadot   2168212658 Sep 24 05:33 table_01_d6
-rw-r--r-- 1 polkadot polkadot   2181528227 Sep 24 05:33 table_01_d4
-rw-r--r-- 1 polkadot polkadot   2231406980 Sep 24 05:33 table_01_53
-rw-r--r-- 1 polkadot polkadot   2256096364 Sep 24 05:33 table_01_22
-rw-r--r-- 1 polkadot polkadot   2257572864 Sep 24 05:33 table_05_fe
-rw-r--r-- 1 polkadot polkadot   2289135888 Sep 24 05:33 table_01_d5
-rw-r--r-- 1 polkadot polkadot   2299231795 Sep 24 05:33 table_01_c9
-rw-r--r-- 1 polkadot polkadot   2322586020 Sep 24 05:33 table_01_d7
-rw-r--r-- 1 polkadot polkadot   2379936324 Sep 24 05:33 table_01_4f
-rw-r--r-- 1 polkadot polkadot   2403849584 Sep 24 05:33 table_01_d8
-rw-r--r-- 1 polkadot polkadot   2458063736 Sep 24 05:33 table_01_cf
-rw-r--r-- 1 polkadot polkadot   2555789396 Sep 24 05:33 table_01_d3
-rw-r--r-- 1 polkadot polkadot   2561064016 Sep 24 05:33 table_05_fd
-rw-r--r-- 1 polkadot polkadot   2586114529 Sep 24 05:33 table_01_ca
-rw-r--r-- 1 polkadot polkadot   2593765572 Sep 24 05:33 table_01_59
-rw-r--r-- 1 polkadot polkadot   2602137056 Sep 24 05:33 table_01_d9
-rw-r--r-- 1 polkadot polkadot   2609816128 Sep 24 05:33 table_01_39
-rw-r--r-- 1 polkadot polkadot   2623629824 Sep 24 05:33 table_01_e1
-rw-r--r-- 1 polkadot polkadot   2657328528 Sep 24 05:33 table_01_4c
-rw-r--r-- 1 polkadot polkadot   2748000427 Sep 24 05:33 table_01_68
-rw-r--r-- 1 polkadot polkadot   2813289979 Sep 24 05:33 table_01_4a
-rw-r--r-- 1 polkadot polkadot   2821288416 Sep 24 05:33 table_01_e2
-rw-r--r-- 1 polkadot polkadot   2842621376 Sep 24 05:33 table_05_fc
-rw-r--r-- 1 polkadot polkadot   2998306504 Sep 24 05:33 table_04_54
-rw-r--r-- 1 polkadot polkadot   3033501434 Sep 24 05:33 table_01_da
-rw-r--r-- 1 polkadot polkadot   3062540856 Sep 24 05:33 table_05_fb
-rw-r--r-- 1 polkadot polkadot   3120097720 Sep 24 05:33 table_01_47
-rw-r--r-- 1 polkadot polkadot   3151739104 Sep 24 05:33 table_01_46
-rw-r--r-- 1 polkadot polkadot   3165868240 Sep 24 05:33 table_01_dc
-rw-r--r-- 1 polkadot polkadot   3182330184 Sep 24 05:33 table_01_5b
-rw-r--r-- 1 polkadot polkadot   3202932576 Sep 24 05:33 table_05_fa
-rw-r--r-- 1 polkadot polkadot   3209200992 Sep 24 05:33 table_01_e3
-rw-r--r-- 1 polkadot polkadot   3226326760 Sep 24 05:33 table_05_f7
-rw-r--r-- 1 polkadot polkadot   3243529044 Sep 24 05:33 table_05_f8
-rw-r--r-- 1 polkadot polkadot   3248712820 Sep 24 05:33 table_01_f0
-rw-r--r-- 1 polkadot polkadot   3251266228 Sep 24 05:33 table_05_f9
-rw-r--r-- 1 polkadot polkadot   3275450368 Sep 24 05:33 table_01_dd
-rw-r--r-- 1 polkadot polkadot   3280693165 Sep 24 05:33 table_05_f6
-rw-r--r-- 1 polkadot polkadot   3329120037 Sep 24 05:33 table_01_de
-rw-r--r-- 1 polkadot polkadot   3389421236 Sep 24 05:33 table_05_f5
-rw-r--r-- 1 polkadot polkadot   3407967928 Sep 24 05:32 table_05_e2
-rw-r--r-- 1 polkadot polkadot   3563131182 Sep 24 05:33 table_01_df
-rw-r--r-- 1 polkadot polkadot   3578547040 Sep 24 05:33 table_05_f4
-rw-r--r-- 1 polkadot polkadot   3589313536 Sep 24 05:33 table_01_25
-rw-r--r-- 1 polkadot polkadot   3640055264 Sep 24 05:33 table_01_e4
-rw-r--r-- 1 polkadot polkadot   3790445504 Sep 24 05:33 table_01_f1
-rw-r--r-- 1 polkadot polkadot   3836941206 Sep 24 05:33 table_05_f3
-rw-r--r-- 1 polkadot polkadot   3934040448 Sep 24 05:32 table_05_e4
-rw-r--r-- 1 polkadot polkadot   4075817248 Sep 24 05:33 table_01_c3
-rw-r--r-- 1 polkadot polkadot   4148923084 Sep 24 05:33 table_05_f2
-rw-r--r-- 1 polkadot polkadot   4151224640 Sep 24 05:32 table_01_e5
-rw-r--r-- 1 polkadot polkadot   4254098908 Sep 24 05:33 table_01_54
-rw-r--r-- 1 polkadot polkadot   4322541026 Sep 24 05:33 table_01_5d
-rw-r--r-- 1 polkadot polkadot   4364086208 Sep 24 05:32 table_05_f1
-rw-r--r-- 1 polkadot polkadot   4391669184 Sep 24 05:33 table_01_cb
-rw-r--r-- 1 polkadot polkadot   4434840508 Sep 24 05:33 table_01_3c
-rw-r--r-- 1 polkadot polkadot   4450037764 Sep 24 05:33 table_01_f2
-rw-r--r-- 1 polkadot polkadot   4553589564 Sep 24 05:33 table_05_f0
-rw-r--r-- 1 polkadot polkadot   4573880608 Sep 24 05:33 table_01_51
-rw-r--r-- 1 polkadot polkadot   4606069184 Sep 24 05:33 table_01_5a
-rw-r--r-- 1 polkadot polkadot   4692085071 Sep 24 05:33 table_01_60
-rw-r--r-- 1 polkadot polkadot   4724172336 Sep 24 05:33 table_05_ef
-rw-r--r-- 1 polkadot polkadot   4733239896 Sep 24 05:33 table_01_e6
-rw-r--r-- 1 polkadot polkadot   4813025472 Sep 24 05:33 table_01_ed
-rw-r--r-- 1 polkadot polkadot   4845201974 Sep 24 05:33 table_01_ec
-rw-r--r-- 1 polkadot polkadot   4888200288 Sep 24 05:33 table_01_56
-rw-r--r-- 1 polkadot polkadot   4906943816 Sep 24 05:33 table_05_ee
-rw-r--r-- 1 polkadot polkadot   4940115968 Sep 24 05:32 table_05_e1
-rw-r--r-- 1 polkadot polkadot   4987111056 Sep 24 05:32 table_05_e3
-rw-r--r-- 1 polkadot polkadot   5008187488 Sep 24 05:33 table_05_ed
-rw-r--r-- 1 polkadot polkadot   5168955392 Sep 24 05:33 table_06_ff
-rw-r--r-- 1 polkadot polkadot   5248583748 Sep 24 05:33 table_01_db
-rw-r--r-- 1 polkadot polkadot   5278282464 Sep 24 05:32 table_05_e5
-rw-r--r-- 1 polkadot polkadot   5322539394 Sep 24 05:33 table_01_e7
-rw-r--r-- 1 polkadot polkadot   5434875054 Sep 24 05:33 table_05_ec
-rw-r--r-- 1 polkadot polkadot   5664356240 Sep 24 05:33 table_01_ef
-rw-r--r-- 1 polkadot polkadot   5731961008 Sep 24 05:33 table_01_f4
-rw-r--r-- 1 polkadot polkadot   5835945216 Sep 24 05:33 table_01_f3
-rw-r--r-- 1 polkadot polkadot   5836125724 Sep 24 05:33 table_01_f5
-rw-r--r-- 1 polkadot polkadot   5899949894 Sep 24 05:33 table_01_eb
-rw-r--r-- 1 polkadot polkadot   6069183466 Sep 24 05:32 table_05_eb
-rw-r--r-- 1 polkadot polkadot   6082879516 Sep 24 05:32 table_05_e6
-rw-r--r-- 1 polkadot polkadot   6155875808 Sep 24 05:33 table_01_e0
-rw-r--r-- 1 polkadot polkadot   6166639480 Sep 24 05:32 table_05_e7
-rw-r--r-- 1 polkadot polkadot   6178273996 Sep 24 05:33 table_01_e8
-rw-r--r-- 1 polkadot polkadot   6204936096 Sep 24 05:33 table_01_24
-rw-r--r-- 1 polkadot polkadot   6382046767 Sep 24 05:32 table_05_ea
-rw-r--r-- 1 polkadot polkadot   6475164388 Sep 24 05:33 table_01_52
-rw-r--r-- 1 polkadot polkadot   6552826660 Sep 24 05:33 table_05_e9
-rw-r--r-- 1 polkadot polkadot   6701876492 Sep 24 05:33 table_01_ea
-rw-r--r-- 1 polkadot polkadot   6789912364 Sep 24 05:33 table_01_e9
-rw-r--r-- 1 polkadot polkadot   7084718144 Sep 24 05:33 table_01_65
-rw-r--r-- 1 polkadot polkadot   7132266184 Sep 24 05:33 table_01_ee
-rw-r--r-- 1 polkadot polkadot   7141972668 Sep 24 05:32 table_05_e8
-rw-r--r-- 1 polkadot polkadot   7229082577 Sep 24 05:24 table_01_f6
-rw-r--r-- 1 polkadot polkadot   7231927703 Sep 24 05:33 table_01_63
-rw-r--r-- 1 polkadot polkadot   7996406304 Sep 24 05:33 table_01_f7
-rw-r--r-- 1 polkadot polkadot   8371265312 Sep 24 05:33 table_01_44
-rw-r--r-- 1 polkadot polkadot   8520785772 Sep 24 05:33 table_01_2d
-rw-r--r-- 1 polkadot polkadot   8669174372 Sep 24 05:25 table_01_f8
-rw-r--r-- 1 polkadot polkadot   9069583416 Sep 24 05:33 table_01_4e
-rw-r--r-- 1 polkadot polkadot   9172382800 Sep 24 05:33 table_01_57
-rw-r--r-- 1 polkadot polkadot   9437403232 Sep 24 05:33 table_01_3d
-rw-r--r-- 1 polkadot polkadot   9903144752 Sep 24 05:33 table_01_f9
-rw-r--r-- 1 polkadot polkadot  10179492580 Sep 24 05:33 table_01_a2
-rw-r--r-- 1 polkadot polkadot  10188693568 Sep 24 05:33 table_01_62
-rw-r--r-- 1 polkadot polkadot  10524296864 Sep 24 05:33 table_01_fa
-rw-r--r-- 1 polkadot polkadot  11681033696 Sep 24 05:33 table_01_35
-rw-r--r-- 1 polkadot polkadot  12156665856 Sep 24 05:33 table_05_ff
-rw-r--r-- 1 polkadot polkadot  12984414072 Sep 24 05:25 table_01_fb
-rw-r--r-- 1 polkadot polkadot  13557394608 Sep 24 05:33 table_01_4b
-rw-r--r-- 1 polkadot polkadot  13657993222 Sep 24 05:33 table_01_a3
-rw-r--r-- 1 polkadot polkadot  13722660624 Sep 24 05:33 table_01_67
-rw-r--r-- 1 polkadot polkadot  15225357833 Sep 24 05:33 table_01_2a
-rw-r--r-- 1 polkadot polkadot  17206735516 Sep 24 05:33 table_01_23
-rw-r--r-- 1 polkadot polkadot  18658578192 Sep 24 05:33 table_01_49
-rw-r--r-- 1 polkadot polkadot  20196776416 Sep 24 05:25 table_01_fc
-rw-r--r-- 1 polkadot polkadot  24542647424 Sep 24 05:33 table_01_fe
-rw-r--r-- 1 polkadot polkadot  29651485216 Sep 24 05:33 table_01_fd
-rw-r--r-- 1 polkadot polkadot  32946602168 Sep 24 05:33 table_01_61
-rw-r--r-- 1 polkadot polkadot  34359754752 Sep 24 05:33 index_01_26
-rw-r--r-- 1 polkadot polkadot  34714212950 Sep 24 05:33 table_01_6b
-rw-r--r-- 1 polkadot polkadot  80399223522 Sep 24 05:33 table_01_6a
-rw-r--r-- 1 polkadot polkadot 107126194176 Sep 24 05:33 table_01_ff

kiB/s ⬆ 43.7kiB/s
Sep 24 05:38:15 ksm24 polkadot[95670]: 2023-09-24 05:38:15 ⚙️  Syncing  0.0 bps, target=#19811437 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 370.3kiB/s ⬆ 26.7kiB/s
Sep 24 05:38:20 ksm24 polkadot[95670]: 2023-09-24 05:38:20 ⚙️  Syncing  0.0 bps, target=#19811438 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 276.9kiB/s ⬆ 27.7kiB/s
Sep 24 05:38:25 ksm24 polkadot[95670]: 2023-09-24 05:38:25 ⚙️  Syncing  0.0 bps, target=#19811439 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 137.9kiB/s ⬆ 23.2kiB/s
Sep 24 05:38:30 ksm24 polkadot[95670]: 2023-09-24 05:38:30 ⚙️  Syncing  0.0 bps, target=#19811439 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 97.1kiB/s ⬆ 30.8kiB/s
Sep 24 05:38:35 ksm24 polkadot[95670]: 2023-09-24 05:38:35 ⚙️  Syncing  0.0 bps, target=#19811440 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 144.7kiB/s ⬆ 48.5kiB/s
Sep 24 05:38:40 ksm24 polkadot[95670]: 2023-09-24 05:38:40 ⚙️  Syncing  0.0 bps, target=#19811441 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 12.5kiB/s ⬆ 27.4kiB/s
Sep 24 05:38:45 ksm24 polkadot[95670]: 2023-09-24 05:38:45 ⚙️  Syncing  0.0 bps, target=#19811442 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 12.0kiB/s ⬆ 26.0kiB/s
Sep 24 05:38:50 ksm24 polkadot[95670]: 2023-09-24 05:38:50 ⚙️  Syncing  0.0 bps, target=#19811443 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 133.1kiB/s ⬆ 19.6kiB/s
Sep 24 05:38:55 ksm24 polkadot[95670]: 2023-09-24 05:38:55 ⚙️  Syncing  0.0 bps, target=#19811444 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 20.1kiB/s ⬆ 27.3kiB/s
Sep 24 05:39:00 ksm24 polkadot[95670]: 2023-09-24 05:39:00 ⚙️  Syncing  0.0 bps, target=#19811444 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 70.2kiB/s ⬆ 30.3kiB/s
Sep 24 05:39:05 ksm24 polkadot[95670]: 2023-09-24 05:39:05 ⚙️  Syncing  0.0 bps, target=#19811445 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 82.3kiB/s ⬆ 28.3kiB/s
Sep 24 05:39:10 ksm24 polkadot[95670]: 2023-09-24 05:39:10 ⚙️  Syncing  0.0 bps, target=#19811446 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 184.6kiB/s ⬆ 24.2kiB/s
Sep 24 05:39:15 ksm24 polkadot[95670]: 2023-09-24 05:39:15 ⚙️  Syncing  0.0 bps, target=#19811447 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 136.7kiB/s ⬆ 23.0kiB/s
Sep 24 05:39:20 ksm24 polkadot[95670]: 2023-09-24 05:39:20 ⚙️  Syncing  0.0 bps, target=#19811448 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 21.1kiB/s ⬆ 27.4kiB/s
Sep 24 05:39:25 ksm24 polkadot[95670]: 2023-09-24 05:39:25 ⚙️  Syncing  0.0 bps, target=#19811449 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 20.6kiB/s ⬆ 27.3kiB/s
Sep 24 05:39:30 ksm24 polkadot[95670]: 2023-09-24 05:39:30 ⚙️  Syncing  0.0 bps, target=#19811449 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 13.3kiB/s ⬆ 20.5kiB/s
Sep 24 05:39:35 ksm24 polkadot[95670]: 2023-09-24 05:39:35 ⚙️  Syncing  0.0 bps, target=#19811450 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 16.9kiB/s ⬆ 23.8kiB/s
Sep 24 05:39:40 ksm24 polkadot[95670]: 2023-09-24 05:39:40 ⚙️  Syncing  0.0 bps, target=#19811451 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 140.4kiB/s ⬆ 19.5kiB/s
Sep 24 05:39:45 ksm24 polkadot[95670]: 2023-09-24 05:39:45 ⚙️  Syncing  0.0 bps, target=#19811452 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 83.6kiB/s ⬆ 19.8kiB/s
Sep 24 05:39:50 ksm24 polkadot[95670]: 2023-09-24 05:39:50 ⚙️  Syncing  0.0 bps, target=#19811453 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 263.7kiB/s ⬆ 18.6kiB/s
Sep 24 05:39:55 ksm24 polkadot[95670]: 2023-09-24 05:39:55 ⚙️  Syncing  0.0 bps, target=#19811453 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 160.7kiB/s ⬆ 22.0kiB/s
Sep 24 05:40:00 ksm24 polkadot[95670]: 2023-09-24 05:40:00 ⚙️  Syncing  0.0 bps, target=#19811454 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 27.7kiB/s ⬆ 30.6kiB/s
Sep 24 05:40:05 ksm24 polkadot[95670]: 2023-09-24 05:40:05 ⚙️  Syncing  0.0 bps, target=#19811455 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 12.2kiB/s ⬆ 22.5kiB/s
Sep 24 05:40:10 ksm24 polkadot[95670]: 2023-09-24 05:40:10 ⚙️  Syncing  0.0 bps, target=#19811456 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 11.5kiB/s ⬆ 20.1kiB/s
Sep 24 05:40:15 ksm24 polkadot[95670]: 2023-09-24 05:40:15 ⚙️  Syncing  0.0 bps, target=#19811457 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 10.9kiB/s ⬆ 12.3kiB/s
Sep 24 05:40:20 ksm24 polkadot[95670]: 2023-09-24 05:40:20 ⚙️  Syncing  0.0 bps, target=#19811458 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 169.5kiB/s ⬆ 15.1kiB/s
Sep 24 05:40:25 ksm24 polkadot[95670]: 2023-09-24 05:40:25 ⚙️  Syncing  0.0 bps, target=#19811459 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 20.3kiB/s ⬆ 25.4kiB/s
Sep 24 05:40:30 ksm24 polkadot[95670]: 2023-09-24 05:40:30 ⚙️  Syncing  0.0 bps, target=#19811459 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 15.3kiB/s ⬆ 45.3kiB/s
Sep 24 05:40:35 ksm24 polkadot[95670]: 2023-09-24 05:40:35 ⚙️  Syncing  0.0 bps, target=#19811460 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 156.4kiB/s ⬆ 43.9kiB/s
Sep 24 05:40:40 ksm24 polkadot[95670]: 2023-09-24 05:40:40 ⚙️  Syncing  0.0 bps, target=#19811461 (40 peers), best: #15072815 (0x9523…faec), finalized #15072768 (0x70a0…b5a1), ⬇ 479.1kiB/s ⬆ 50.4kiB/s
Sep 24 05:40:45 ksm24 polkadot[95670]: 2023-09-24 05:40:45 ⚙️  Syncing  9.2 bps, target=#19811462 (40 peers), best: #15072861 (0xb977…0922), finalized #15072768 (0x70a0…b5a1), ⬇ 244.7kiB/s ⬆ 37.4kiB/s
Sep 24 05:40:50 ksm24 polkadot[95670]: 2023-09-24 05:40:50 ⚙️  Syncing 28.6 bps, target=#19811463 (40 peers), best: #15073004 (0x9bee…1a39), finalized #15072768 (0x70a0…b5a1), ⬇ 1.1MiB/s ⬆ 28.3kiB/s
Sep 24 05:40:55 ksm24 polkadot[95670]: 2023-09-24 05:40:55 ⚙️  Syncing 29.0 bps, target=#19811464 (40 peers), best: #15073149 (0xea02…2b7e), finalized #15072768 (0x70a0…b5a1), ⬇ 1.1MiB/s ⬆ 34.6kiB/s
Sep 24 05:41:00 ksm24 polkadot[95670]: 2023-09-24 05:41:00 ⚙️  Syncing 29.6 bps, target=#19811464 (40 peers), best: #15073297 (0x2928…8335), finalized #15073280 (0xc8f9…6748), ⬇ 1.1MiB/s ⬆ 33.6kiB/s
Sep 24 05:41:05 ksm24 polkadot[95670]: 2023-09-24 05:41:05 ⚙️  Syncing 26.2 bps, target=#19811465 (40 peers), best: #15073428 (0x51fd…1728), finalized #15073280 (0xc8f9…6748), ⬇ 874.6kiB/s ⬆ 19.2kiB/s
Sep 24 05:41:10 ksm24 polkadot[95670]: 2023-09-24 05:41:10 ⚙️  Syncing 28.4 bps, target=#19811466 (40 peers), best: #15073570 (0x2817…7611), finalized #15073280 (0xc8f9…6748), ⬇ 776.9kiB/s ⬆ 26.9kiB/s
Sep 24 05:41:15 ksm24 polkadot[95670]: 2023-09-24 05:41:15 ⚙️  Syncing 27.2 bps, target=#19811467 (40 peers), best: #15073706 (0xb932…eb85), finalized #15073280 (0xc8f9…6748), ⬇ 1.0MiB/s ⬆ 25.7kiB/s
Sep 24 05:41:20 ksm24 polkadot[95670]: 2023-09-24 05:41:20 ⚙️  Syncing 25.2 bps, target=#19811468 (40 peers), best: #15073832 (0xbbf2…8431), finalized #15073792 (0xd791…20b0), ⬇ 818.0kiB/s ⬆ 24.6kiB/s
Sep 24 05:41:25 ksm24 polkadot[95670]: 2023-09-24 05:41:25 ⚙️  Syncing 24.6 bps, target=#19811469 (40 peers), best: #15073955 (0x873d…0b0a), finalized #15073792 (0xd791…20b0), ⬇ 546.8kiB/s ⬆ 14.5kiB/s
Sep 24 05:41:30 ksm24 polkadot[95670]: 2023-09-24 05:41:30 ⚙️  Syncing 24.8 bps, target=#19811469 (40 peers), best: #15074079 (0xc1a9…42de), finalized #15073792 (0xd791…20b0), ⬇ 1.0MiB/s ⬆ 20.2kiB/s
Sep 24 05:41:35 ksm24 polkadot[95670]: 2023-09-24 05:41:35 ⚙️  Syncing 24.4 bps, target=#19811470 (40 peers), best: #15074201 (0x68fa…4002), finalized #15073792 (0xd791…20b0), ⬇ 780.3kiB/s ⬆ 24.7kiB/s
Sep 24 05:41:40 ksm24 polkadot[95670]: 2023-09-24 05:41:40 ⚙️  Syncing 24.2 bps, target=#19811471 (40 peers), best: #15074322 (0x3eb5…13d7), finalized #15074304 (0xd335…4471), ⬇ 624.7kiB/s ⬆ 25.8kiB/s
Sep 24 05:41:45 ksm24 polkadot[95670]: 2023-09-24 05:41:45 ⚙️  Syncing 25.8 bps, target=#19811472 (40 peers), best: #15074451 (0x00ce…584a), finalized #15074304 (0xd335…4471), ⬇ 927.4kiB/s ⬆ 22.5kiB/s
Sep 24 05:41:50 ksm24 polkadot[95670]: 2023-09-24 05:41:50 ⚙️  Syncing 23.8 bps, target=#19811473 (40 peers), best: #15074570 (0x475c…8463), finalized #15074304 (0xd335…4471), ⬇ 684.3kiB/s ⬆ 17.3kiB/s
Sep 24 05:41:55 ksm24 polkadot[95670]: 2023-09-24 05:41:55 ⚙️  Syncing 17.4 bps, target=#19811474 (40 peers), best: #15074657 (0xefff…e272), finalized #15074304 (0xd335…4471), ⬇ 830.2kiB/s ⬆ 13.8kiB/s
Sep 24 05:42:00 ksm24 polkadot[95670]: 2023-09-24 05:42:00 ⚙️  Syncing 13.2 bps, target=#19811474 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 414.8kiB/s ⬆ 19.0kiB/s
Sep 24 05:42:05 ksm24 polkadot[95670]: 2023-09-24 05:42:05 ⚙️  Syncing  0.0 bps, target=#19811475 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 14.5kiB/s ⬆ 22.7kiB/s
Sep 24 05:42:10 ksm24 polkadot[95670]: 2023-09-24 05:42:10 ⚙️  Syncing  0.0 bps, target=#19811476 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 184.4kiB/s ⬆ 47.4kiB/s
Sep 24 05:42:15 ksm24 polkadot[95670]: 2023-09-24 05:42:15 ⚙️  Syncing  0.0 bps, target=#19811477 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 143.0kiB/s ⬆ 47.0kiB/s
Sep 24 05:42:20 ksm24 polkadot[95670]: 2023-09-24 05:42:20 ⚙️  Syncing  0.0 bps, target=#19811478 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 138.3kiB/s ⬆ 29.2kiB/s
Sep 24 05:42:25 ksm24 polkadot[95670]: 2023-09-24 05:42:25 ⚙️  Syncing  0.0 bps, target=#19811479 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 165.9kiB/s ⬆ 19.5kiB/s
Sep 24 05:42:30 ksm24 polkadot[95670]: 2023-09-24 05:42:30 ⚙️  Syncing  0.0 bps, target=#19811479 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 10.4kiB/s ⬆ 15.4kiB/s
Sep 24 05:42:35 ksm24 polkadot[95670]: 2023-09-24 05:42:35 ⚙️  Syncing  0.0 bps, target=#19811480 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 10.7kiB/s ⬆ 11.5kiB/s
Sep 24 05:42:40 ksm24 polkadot[95670]: 2023-09-24 05:42:40 ⚙️  Syncing  0.0 bps, target=#19811481 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 11.2kiB/s ⬆ 14.1kiB/s
Sep 24 05:42:45 ksm24 polkadot[95670]: 2023-09-24 05:42:45 ⚙️  Syncing  0.0 bps, target=#19811482 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 16.0kiB/s ⬆ 37.9kiB/s
Sep 24 05:42:50 ksm24 polkadot[95670]: 2023-09-24 05:42:50 ⚙️  Syncing  0.0 bps, target=#19811483 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 256.9kiB/s ⬆ 18.6kiB/s
Sep 24 05:42:55 ksm24 polkadot[95670]: 2023-09-24 05:42:55 ⚙️  Syncing  0.0 bps, target=#19811484 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 131.9kiB/s ⬆ 19.5kiB/s
Sep 24 05:43:00 ksm24 polkadot[95670]: 2023-09-24 05:43:00 ⚙️  Syncing  0.0 bps, target=#19811484 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 10.4kiB/s ⬆ 16.6kiB/s
Sep 24 05:43:05 ksm24 polkadot[95670]: 2023-09-24 05:43:05 ⚙️  Syncing  0.0 bps, target=#19811485 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 9.8kiB/s ⬆ 24.1kiB/s
Sep 24 05:43:10 ksm24 polkadot[95670]: 2023-09-24 05:43:10 ⚙️  Syncing  0.0 bps, target=#19811486 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 13.8kiB/s ⬆ 33.8kiB/s
Sep 24 05:43:15 ksm24 polkadot[95670]: 2023-09-24 05:43:15 ⚙️  Syncing  0.0 bps, target=#19811487 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 13.8kiB/s ⬆ 21.1kiB/s
Sep 24 05:43:20 ksm24 polkadot[95670]: 2023-09-24 05:43:20 ⚙️  Syncing  0.0 bps, target=#19811488 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 140.0kiB/s ⬆ 27.0kiB/s
Sep 24 05:43:25 ksm24 polkadot[95670]: 2023-09-24 05:43:25 ⚙️  Syncing  0.0 bps, target=#19811489 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 17.6kiB/s ⬆ 29.9kiB/s
Sep 24 05:43:30 ksm24 polkadot[95670]: 2023-09-24 05:43:30 ⚙️  Syncing  0.0 bps, target=#19811489 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 37.6kiB/s ⬆ 28.1kiB/s
Sep 24 05:43:35 ksm24 polkadot[95670]: 2023-09-24 05:43:35 ⚙️  Syncing  0.0 bps, target=#19811490 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 173.0kiB/s ⬆ 24.0kiB/s
Sep 24 05:43:40 ksm24 polkadot[95670]: 2023-09-24 05:43:40 ⚙️  Syncing  0.0 bps, target=#19811491 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 421.4kiB/s ⬆ 31.9kiB/s
Sep 24 05:43:45 ksm24 polkadot[95670]: 2023-09-24 05:43:45 ⚙️  Syncing  0.0 bps, target=#19811492 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 79.0kiB/s ⬆ 30.5kiB/s
Sep 24 05:43:50 ksm24 polkadot[95670]: 2023-09-24 05:43:50 ⚙️  Syncing  0.0 bps, target=#19811493 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 270.3kiB/s ⬆ 14.8kiB/s
Sep 24 05:43:55 ksm24 polkadot[95670]: 2023-09-24 05:43:55 ⚙️  Syncing  0.0 bps, target=#19811494 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 15.4kiB/s ⬆ 17.9kiB/s
Sep 24 05:44:00 ksm24 polkadot[95670]: 2023-09-24 05:44:00 ⚙️  Syncing  0.0 bps, target=#19811494 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 14.3kiB/s ⬆ 27.8kiB/s
Sep 24 05:44:05 ksm24 polkadot[95670]: 2023-09-24 05:44:05 ⚙️  Syncing  0.0 bps, target=#19811495 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 12.3kiB/s ⬆ 23.2kiB/s
Sep 24 05:44:10 ksm24 polkadot[95670]: 2023-09-24 05:44:10 ⚙️  Syncing  0.0 bps, target=#19811496 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 172.6kiB/s ⬆ 20.4kiB/s
Sep 24 05:44:15 ksm24 polkadot[95670]: 2023-09-24 05:44:15 ⚙️  Syncing  0.0 bps, target=#19811497 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 48.1kiB/s ⬆ 17.8kiB/s
Sep 24 05:44:20 ksm24 polkadot[95670]: 2023-09-24 05:44:20 ⚙️  Syncing  0.0 bps, target=#19811498 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 13.7kiB/s ⬆ 24.0kiB/s
Sep 24 05:44:25 ksm24 polkadot[95670]: 2023-09-24 05:44:25 ⚙️  Syncing  0.0 bps, target=#19811499 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 143.3kiB/s ⬆ 35.8kiB/s
Sep 24 05:44:30 ksm24 polkadot[95670]: 2023-09-24 05:44:30 ⚙️  Syncing  0.0 bps, target=#19811499 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 21.3kiB/s ⬆ 37.6kiB/s
Sep 24 05:44:35 ksm24 polkadot[95670]: 2023-09-24 05:44:35 ⚙️  Syncing  0.0 bps, target=#19811500 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 15.3kiB/s ⬆ 31.0kiB/s
Sep 24 05:44:40 ksm24 polkadot[95670]: 2023-09-24 05:44:40 ⚙️  Syncing  0.0 bps, target=#19811501 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 17.4kiB/s ⬆ 26.9kiB/s
Sep 24 05:44:45 ksm24 polkadot[95670]: 2023-09-24 05:44:45 ⚙️  Syncing  0.0 bps, target=#19811502 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 244.4kiB/s ⬆ 20.8kiB/s
Sep 24 05:44:50 ksm24 polkadot[95670]: 2023-09-24 05:44:50 ⚙️  Syncing  0.0 bps, target=#19811503 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 18.8kiB/s ⬆ 25.0kiB/s
Sep 24 05:44:55 ksm24 polkadot[95670]: 2023-09-24 05:44:55 ⚙️  Syncing  0.0 bps, target=#19811504 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 13.9kiB/s ⬆ 31.6kiB/s
Sep 24 05:45:00 ksm24 polkadot[95670]: 2023-09-24 05:45:00 ⚙️  Syncing  0.0 bps, target=#19811504 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 12.9kiB/s ⬆ 44.3kiB/s
Sep 24 05:45:05 ksm24 polkadot[95670]: 2023-09-24 05:45:05 ⚙️  Syncing  0.0 bps, target=#19811505 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 304.0kiB/s ⬆ 35.7kiB/s
Sep 24 05:45:10 ksm24 polkadot[95670]: 2023-09-24 05:45:10 ⚙️  Syncing  0.0 bps, target=#19811506 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 154.0kiB/s ⬆ 28.4kiB/s
Sep 24 05:45:15 ksm24 polkadot[95670]: 2023-09-24 05:45:15 ⚙️  Syncing  0.0 bps, target=#19811507 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 10.8kiB/s ⬆ 23.3kiB/s
Sep 24 05:45:20 ksm24 polkadot[95670]: 2023-09-24 05:45:20 ⚙️  Syncing  0.0 bps, target=#19811508 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 283.0kiB/s ⬆ 18.0kiB/s
Sep 24 05:45:25 ksm24 polkadot[95670]: 2023-09-24 05:45:25 ⚙️  Syncing  0.0 bps, target=#19811509 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 18.5kiB/s ⬆ 46.0kiB/s
Sep 24 05:45:30 ksm24 polkadot[95670]: 2023-09-24 05:45:30 ⚙️  Syncing  0.0 bps, target=#19811509 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 14.5kiB/s ⬆ 40.6kiB/s
Sep 24 05:45:35 ksm24 polkadot[95670]: 2023-09-24 05:45:35 ⚙️  Syncing  0.0 bps, target=#19811510 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 13.6kiB/s ⬆ 23.3kiB/s
Sep 24 05:45:40 ksm24 polkadot[95670]: 2023-09-24 05:45:40 ⚙️  Syncing  0.0 bps, target=#19811511 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 110.0kiB/s ⬆ 16.0kiB/s
Sep 24 05:45:45 ksm24 polkadot[95670]: 2023-09-24 05:45:45 ⚙️  Syncing  0.0 bps, target=#19811512 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 13.7kiB/s ⬆ 31.3kiB/s
Sep 24 05:45:50 ksm24 polkadot[95670]: 2023-09-24 05:45:50 ⚙️  Syncing  0.0 bps, target=#19811513 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 20.4kiB/s ⬆ 24.3kiB/s
Sep 24 05:45:55 ksm24 polkadot[95670]: 2023-09-24 05:45:55 ⚙️  Syncing  0.0 bps, target=#19811514 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 12.1kiB/s ⬆ 22.5kiB/s
Sep 24 05:46:00 ksm24 polkadot[95670]: 2023-09-24 05:46:00 ⚙️  Syncing  0.0 bps, target=#19811514 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 10.1kiB/s ⬆ 23.1kiB/s
Sep 24 05:46:05 ksm24 polkadot[95670]: 2023-09-24 05:46:05 ⚙️  Syncing  0.0 bps, target=#19811515 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 14.6kiB/s ⬆ 27.4kiB/s
Sep 24 05:46:10 ksm24 polkadot[95670]: 2023-09-24 05:46:10 ⚙️  Syncing  0.0 bps, target=#19811516 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 67.7kiB/s ⬆ 30.7kiB/s
Sep 24 05:46:15 ksm24 polkadot[95670]: 2023-09-24 05:46:15 ⚙️  Syncing  0.0 bps, target=#19811517 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 13.2kiB/s ⬆ 40.5kiB/s
Sep 24 05:46:20 ksm24 polkadot[95670]: 2023-09-24 05:46:20 ⚙️  Syncing  0.0 bps, target=#19811518 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 113.0kiB/s ⬆ 50.4kiB/s
Sep 24 05:46:25 ksm24 polkadot[95670]: 2023-09-24 05:46:25 ⚙️  Syncing  0.0 bps, target=#19811519 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 130.9kiB/s ⬆ 26.5kiB/s
Sep 24 05:46:30 ksm24 polkadot[95670]: 2023-09-24 05:46:30 ⚙️  Syncing  0.0 bps, target=#19811519 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 19.8kiB/s ⬆ 48.8kiB/s
Sep 24 05:46:35 ksm24 polkadot[95670]: 2023-09-24 05:46:35 ⚙️  Syncing  0.0 bps, target=#19811520 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 154.3kiB/s ⬆ 28.0kiB/s
Sep 24 05:46:40 ksm24 polkadot[95670]: 2023-09-24 05:46:40 ⚙️  Syncing  0.0 bps, target=#19811521 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 14.0kiB/s ⬆ 13.3kiB/s
Sep 24 05:46:45 ksm24 polkadot[95670]: 2023-09-24 05:46:45 ⚙️  Syncing  0.0 bps, target=#19811522 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 16.0kiB/s ⬆ 20.3kiB/s
Sep 24 05:46:50 ksm24 polkadot[95670]: 2023-09-24 05:46:50 ⚙️  Syncing  0.0 bps, target=#19811523 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 15.5kiB/s ⬆ 22.3kiB/s
Sep 24 05:46:55 ksm24 polkadot[95670]: 2023-09-24 05:46:55 ⚙️  Syncing  0.0 bps, target=#19811524 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 14.2kiB/s ⬆ 16.3kiB/s
Sep 24 05:47:00 ksm24 polkadot[95670]: 2023-09-24 05:47:00 ⚙️  Syncing  0.0 bps, target=#19811524 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 12.0kiB/s ⬆ 20.8kiB/s
Sep 24 05:47:05 ksm24 polkadot[95670]: 2023-09-24 05:47:05 ⚙️  Syncing  0.0 bps, target=#19811525 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 14.5kiB/s ⬆ 15.4kiB/s
Sep 24 05:47:10 ksm24 polkadot[95670]: 2023-09-24 05:47:10 ⚙️  Syncing  0.0 bps, target=#19811526 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 143.9kiB/s ⬆ 19.2kiB/s
Sep 24 05:47:15 ksm24 polkadot[95670]: 2023-09-24 05:47:15 ⚙️  Syncing  0.0 bps, target=#19811527 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 12.8kiB/s ⬆ 26.4kiB/s
Sep 24 05:47:20 ksm24 polkadot[95670]: 2023-09-24 05:47:20 ⚙️  Syncing  0.0 bps, target=#19811528 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 149.1kiB/s ⬆ 25.0kiB/s
Sep 24 05:47:25 ksm24 polkadot[95670]: 2023-09-24 05:47:25 ⚙️  Syncing  0.0 bps, target=#19811529 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 46.8kiB/s ⬆ 21.1kiB/s
Sep 24 05:47:30 ksm24 polkadot[95670]: 2023-09-24 05:47:30 ⚙️  Syncing  0.0 bps, target=#19811529 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 13.1kiB/s ⬆ 20.3kiB/s
Sep 24 05:47:35 ksm24 polkadot[95670]: 2023-09-24 05:47:35 ⚙️  Syncing  0.0 bps, target=#19811530 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 153.1kiB/s ⬆ 20.3kiB/s
Sep 24 05:47:40 ksm24 polkadot[95670]: 2023-09-24 05:47:40 ⚙️  Syncing  0.0 bps, target=#19811531 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 290.3kiB/s ⬆ 23.9kiB/s
Sep 24 05:47:45 ksm24 polkadot[95670]: 2023-09-24 05:47:45 ⚙️  Syncing  0.0 bps, target=#19811532 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 300.3kiB/s ⬆ 39.6kiB/s
Sep 24 05:47:50 ksm24 polkadot[95670]: 2023-09-24 05:47:50 ⚙️  Syncing  0.0 bps, target=#19811533 (40 peers), best: #15074723 (0xf175…b1b7), finalized #15074304 (0xd335…4471), ⬇ 14.8kiB/s ⬆ 32.6kiB/s

yeah not sure how I am supposed to snapshot/backup this so will just resync using rocksdb for the dot/ksm archive nodes and leave westend with paritydb for now

arkpar commented 8 months ago

it is not a trivial task to move really big files to or from the backup storage. any network interruptions can increase the number of retries. any retries with 200Gb files take a lot of time and expensive traffic all files are changed very often, it doesn't allow us to use diff backups based on the modification time of separate files all files are changed very often, it doesn't allow us to use diff backups based on the modification time of separate files

Why not use a proper backup protocol, such as rsync for example? It solves both issues.

any CDN cache services don't allow cache files with sizes more than 500-1000MB. it makes providing public snapshots more expensive

Does anyone really provides snapshots as uncompressed raw dir dumps?

I've just googled for polkadot snapshots and the first 3 result were either lz4 or tar archives.

PierreBesson commented 8 months ago

Why not use a proper backup protocol, such as rsync for example? It solves both issues.

Rsync does not work natively with cloud storage providers. We use rclone

Does anyone really provides snapshots as uncompressed raw dir dumps?

Yes we do. See https://github.com/paritytech/helm-charts/tree/main/charts/node#public-snapshots

Polkadot-Forum commented 4 months ago

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/optimal-filesystem-for-blockchain-databases/6116/1

kogeler commented 4 months ago

@arkpar

I've just googled for polkadot snapshots and the first 3 result were either lz4 or tar archives.

We are discoursing big archive DBs.

Any compression algorithm only saves a couple of percentages of size in our case. But If you use one file it is impossible to sync it fast in a multi-thread way or use diff sync (update only changed db files), also it makes it difficult to continue downloading after an interruption. In general, it will be really difficult to upload a 2-3 terabyte archive file to cloud storage.

Also, CDN doesn't allow caching of big files. We can use an additional layer of archiving to split big paritydb files into smaller files. However, it requires more resources on the client side to unpack it (CPU, extra disk space).

paritytech / parity-db

size of table files feature request #219

Needs

How it works now

The request