search

rpcpool / yellowstone-faithful

Project Yellowstone Old Faithful is the project to make all of Solana's history accessible, content addressable and available via a variety of means.
https://old-faithful.net/
GNU Affero General Public License v3.0
54 stars 9 forks source link

CAR file for epoch #584 is potentially corrupted #117

Open EugeneButusov opened 2 days ago

EugeneButusov commented 2 days ago

Probably it's not correct place to report, since it's not code related. Downloading of CAR file for epoch 584 is dropping connection after byte 600000000 (572,20MB downloading completeness). Tried with different machines and different tools to download with the same outcome.

Lusitaniae commented 2 days ago

We've heard good feedback from customers using aria2c

example

aria2c -c -s 16 -x 16 --file-allocation=none https://files.old-faithful.net/500/epoch-500.car -d /storage

Good handling retries and connection issues

The CAR files are stored in a commercial Object Storage service with Cloudflare in front.

EugeneButusov commented 2 days ago

just tried with aria and it throws some set of errors, not 100% sure it's the same, but I assume it could be since it is reported as EOF from server (and download speed drastically degrades):

aria2c -c -s 16 -x 16 --file-allocation=none https://files.old-faithful.net/584/epoch-584.car -d ./

07/01 13:53:14 [NOTICE] Downloading 1 item(s)
 *** Download Progress Summary as of Mon Jul  1 13:54:16 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 11GiB/393GiB(2%) CN:16 DL:150MiB ETA:43m15s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

[#a4c812 13GiB/393GiB(3%) CN:16 DL:253MiB ETA:25m39s]                                                                                                          
07/01 13:54:24 [ERROR] CUID#16 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
[#a4c812 19GiB/393GiB(4%) CN:15 DL:187MiB ETA:34m10s]                                                                                                          
07/01 13:54:59 [ERROR] CUID#7 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
[#a4c812 19GiB/393GiB(4%) CN:14 DL:186MiB ETA:34m17s]                                                                                                          
07/01 13:55:01 [ERROR] CUID#21 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
 *** Download Progress Summary as of Mon Jul  1 13:55:17 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 22GiB/393GiB(5%) CN:13 DL:193MiB ETA:32m47s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

[#a4c812 24GiB/393GiB(6%) CN:13 DL:126MiB ETA:49m46s]                                                                                                          
07/01 13:55:32 [ERROR] CUID#14 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
[#a4c812 29GiB/393GiB(7%) CN:12 DL:156MiB ETA:39m47s]                                                                                                          
07/01 13:56:11 [ERROR] CUID#22 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
 *** Download Progress Summary as of Mon Jul  1 13:56:17 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 30GiB/393GiB(7%) CN:11 DL:98MiB ETA:1h3m5s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

[#a4c812 33GiB/393GiB(8%) CN:11 DL:113MiB ETA:54m11s]                                                                                                          
07/01 13:56:46 [ERROR] CUID#20 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
[#a4c812 34GiB/393GiB(8%) CN:10 DL:173MiB ETA:35m22s]                                                                                                          
07/01 13:56:54 [ERROR] CUID#12 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
 *** Download Progress Summary as of Mon Jul  1 13:57:18 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 37GiB/393GiB(9%) CN:9 DL:145MiB ETA:41m49s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Mon Jul  1 13:58:19 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 44GiB/393GiB(11%) CN:9 DL:85MiB ETA:1h9m18s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

[#a4c812 45GiB/393GiB(11%) CN:9 DL:100MiB ETA:59m10s]                                                                                                          
07/01 13:58:33 [ERROR] CUID#15 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
[#a4c812 46GiB/393GiB(11%) CN:8 DL:73MiB ETA:1h20m14s]                                                                                                         
07/01 13:58:45 [ERROR] CUID#19 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
[#a4c812 47GiB/393GiB(12%) CN:7 DL:70MiB ETA:1h23m37s]                                                                                                         
07/01 13:59:00 [ERROR] CUID#13 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
 *** Download Progress Summary as of Mon Jul  1 13:59:19 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 49GiB/393GiB(12%) CN:6 DL:61MiB ETA:1h35m2s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

[#a4c812 50GiB/393GiB(12%) CN:6 DL:50MiB ETA:1h55m5s]                                                                                                          
07/01 13:59:41 [ERROR] CUID#9 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
[#a4c812 52GiB/393GiB(13%) CN:5 DL:39MiB ETA:2h28m27s]                                                                                                         
07/01 14:00:13 [ERROR] CUID#11 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
 *** Download Progress Summary as of Mon Jul  1 14:00:20 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 52GiB/393GiB(13%) CN:4 DL:88MiB ETA:1h5m48s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Mon Jul  1 14:01:20 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 56GiB/393GiB(14%) CN:4 DL:59MiB ETA:1h36m46s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Mon Jul  1 14:02:21 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 59GiB/393GiB(15%) CN:4 DL:21MiB ETA:4h31m11s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Mon Jul  1 14:03:21 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 63GiB/393GiB(16%) CN:4 DL:46MiB ETA:2h56s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Mon Jul  1 14:04:21 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 66GiB/393GiB(16%) CN:4 DL:66MiB ETA:1h24m5s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

[#a4c812 67GiB/393GiB(17%) CN:4 DL:16MiB ETA:5h42m14s]                                                                                                         
07/01 14:04:51 [ERROR] CUID#18 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
 *** Download Progress Summary as of Mon Jul  1 14:05:22 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 67GiB/393GiB(17%) CN:3 DL:36MiB ETA:2h33m57s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

[#a4c812 68GiB/393GiB(17%) CN:3 DL:31MiB ETA:2h58m2s]                                                                                                          
07/01 14:05:40 [ERROR] CUID#17 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
 *** Download Progress Summary as of Mon Jul  1 14:06:22 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 69GiB/393GiB(17%) CN:2 DL:9.5MiB ETA:9h41m12s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

 *** Download Progress Summary as of Mon Jul  1 14:07:23 2024 ***                                                                                              
===============================================================================================================================================================
[#a4c812 71GiB/393GiB(18%) CN:2 DL:7.5KiB ETA:12425h26m17s]
FILE: .//epoch-584.car
---------------------------------------------------------------------------------------------------------------------------------------------------------------

[#a4c812 71GiB/393GiB(18%) CN:2 DL:0B]                                                                                                                         
07/01 14:08:09 [ERROR] CUID#8 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.
[#a4c812 71GiB/393GiB(18%) CN:1 DL:0B]                                                                                                                         
07/01 14:08:15 [ERROR] CUID#10 - Download aborted. URI=https://files.old-faithful.net/584/epoch-584.car
Exception: [DownloadCommand.cc:234] errorCode=1 Got EOF from the server.

07/01 14:08:15 [NOTICE] Download GID#a4c812695376230f not complete: .//epoch-584.car

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
a4c812|ERR |    81MiB/s|.//epoch-584.car

Status Legend:
(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.

it worked fine for tens of CAR epoch files I've tried with, except this one.

UPD: Attached whole output when tool exited.

Lusitaniae commented 2 days ago

The download seems to progressing even though it hits errors

But you could simply rerun the command to resume the download from the last piece

For now there's no solution for the download speed as we're dependent on the object storage service, we expect heavy users to simply do a full clone of the archive (which would take days) and handle retries as required to completely download all files.

EugeneButusov commented 2 days ago

The download seems to progressing even though it hits errors

unfortunately, it is not progressing since aria2c exited (07/01 14:08:15 [NOTICE] Download GID#a4c812695376230f not complete: .//epoch-584.car).

The issue is not in the downloading speed - initially it kicks out with very good speed (~30min to get ~400GB, which is awesome). The problem is that it hits the errors, and (I assume because of decreasing amount of download threads) it slows down. Since it gets EOF from the server, I assume this file can be corrupted. Other epochs (at least from 585 up to realtime) are downloadable with zero issues. Could you please check if this epoch file is correct? Thanks in advance.

Lusitaniae commented 2 days ago

Haven't used aria2c extensively, but from looking at the logs it looks like basically it's seems to be doing 16 attempts to download the file and after 16 errors it fails/exists.

When you run the command again, it will try to resume from the last downloaded chunk

Now the problem is that the file is stored in object storage service (Backblaze B2 - which is a bit slower/performant than alternatives like S3), and the nature of object storage is that files are split into multiple pices and distributed around multiple servers for redundancy.

Now because B2 focus on backups, rather than performance there could be problems accessing files at times.

I think for now best I can recommend is retrying at a later point