s3 snapshot sources not work/incomplete for "suidb/live/store/perpetual/MANIFEST-XXXXXXX"?!

[jun0tpyrc]

Steps to Reproduce Issue

aws s3 sync s3://mysten-mainnet-snapshots/epoch_549 /data/setup/app_data/suidb/live/ --request-payer --copy-props none etc to get snapshots

Expected Result

can use

Actual Result

2024-10-14T01:37:03.910505Z  INFO sui_node: Sui Node version: 1.34.3-5d1857ef1615
2024-10-14T01:37:03.910532Z  INFO sui_node: Supported protocol versions: Some(SupportedProtocolVersions { min: ProtocolVersion(1), max: ProtocolVersion(60) })
2024-10-14T01:37:03.910553Z  INFO sui_node: Started Prometheus HTTP endpoint at 0.0.0.0:9184
2024-10-14T01:37:03.910869Z  INFO sui_node: Initializing sui-node listening on /dns/localhost/tcp/8080/http node=k#a89b12bbd29438feccaed9a6a37f433e8d51e96cf4d750f4b201044e4b9964309c0ec92b
b01fec592581d3f15242e27b04a32a20fbfd746ac72f00f0f8e9e3205dc1b17c966907321c05c8feafeb277775cf8a053cfe21e0a41d287c822ef9ca
2024-10-14T01:37:03.911346Z  INFO mysten_metrics::thread_stall_monitor: Starting thread stall monitor watchdog thread
2024-10-14T01:37:04.061721Z  INFO mysten_metrics::thread_stall_monitor: Spawning future crates/mysten-metrics/src/thread_stall_monitor.rs:ThreadStallMonitor
2024-10-14T01:37:04.061761Z  INFO mysten_metrics::thread_stall_monitor: Starting thread stall monitor update task
2024-10-14T01:37:04.082783Z ERROR typed_store::rocks: error=rocksdb error: IO error: No such file or directory: While opening a file for sequentially reading: suidb/live/store/perpetual/M
ANIFEST-3413947: No such file or directory
2024-10-14T01:37:04.082858Z ERROR telemetry_subscribers: panicked at crates/sui-core/src/authority/authority_store_tables.rs:52:10:
Cannot open DB at "suidb/live/store/perpetual": RocksDBError("IO error: No such file or directory: While opening a file for sequentially reading: suidb/live/store/perpetual/MANIFEST-34139
47: No such file or directory") panic.file="crates/sui-core/src/authority/authority_store_tables.rs" panic.line=52 panic.column=10

tried epoch 549/548 , same issue

System Information

• OS:
• Compiler:

[jun0tpyrc]

looked into this a little bit actually

[root@bprod-sui-2 setup]# aws s3 ls  s3://mysten-mainnet-snapshots/epoch_547/store/perpetual/  --request-payer
...
2024-10-11 20:20:21     481428 MANIFEST-3370387
2024-10-11 20:15:04      68081 OPTIONS-3370344
2024-10-11 20:18:21      68081 OPTIONS-3370389

up to this epoch_547 seems normal having a MANIFEST-* sorted by name but missing in recent ones

should be incomplete upload?!

epoch540 
Total Objects: 45979
   Total Size: 1.3 TiB

epoch546
Total Objects: 50298
   Total Size: 1.5 TiB

epoch547
Total Objects: 9076
   Total Size: 202.9 GiB

...

[jun0tpyrc]

so it looks broken upload since epoch 547 etc , updated some stat above

anyway verified epoch-546 snapshot is usable & working

[johnjmartin]

Thanks for the report. we're looking into the problem and removing corrupted epochs. Will report back when we have root caused the issue