search

base-org / node

Everything required to run your own Base node
https://base.org
MIT License
68.33k stars 2.58k forks source link

geth full archive snapshots might not be working #351

Open azeonmarket opened 4 days ago

azeonmarket commented 4 days ago

The Geth full archive snapshots not working for downloading snapshot to be able to run a full archive Geth Node. https://mainnet-archive-snapshots.base.org/$(curl https://mainnet-archive-snapshots.base.org/latest)

If true of malfunctions/not working properly, changes on https://docs.base.org/tutorials/run-a-base-node/ on this ligne from the table :

Mainnet | Geth | Archive | wget https://mainnet-archive-snapshots.base.org/$(curl https://mainnet-archive-snapshots.base.org/latest) -- | -- | -- | --

would need to be implemented to allow node operators to deploy easily.

azeonmarket commented 4 days ago

### This was shared from a community member :

this is what i just tested with 2 servers (dedicated, 7tb nvme, amd 5950x , 128gb ram)

Both download stopped at the same time.

server 1 : (wget with long timeout flag) wget --timeout=60 --tries=100 -c https://mainnet-full-snapshots.base.org/base-mainnet-full-173 1839014.tar.zst --2024-11-19 10:48:04-- https://mainnet-full-snapshots.base.org/base-mainnet-full-1731839014.tar.zst Resolving mainnet-full-snapshots.base.org (mainnet-full-snapshots.base.org)... 2606:4700:4400::ac40:9367, 2606:4700:4400::6812 :2899, 104.18.40.153, ... Connecting to mainnet-full-snapshots.base.org (mainnet-full-snapshots.base.org)|2606:4700:4400::ac40:9367|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1844290065485 (1.7T) Saving to: ‘base-mainnet-full-1731839014.tar.zst’

base-mainnet-full-1731839014.tar.zst 6%[====> ] 119.68G 19.5MB/s in 71m 19s

2024-11-19 11:59:24 (28.6 MB/s) - Connection closed at byte 128505195208. Retrying.

server 2: aria with longer timeout flag + -x4 -s4 aria2c -x4 -s4 --timeout=120 --retry-wait=10 --file-allocation=none https://mainnet-full-snapshots.base.org/base-mainnet-full-1731839014.tar.zst

Download Progress Summary as of Tue Nov 19 12:59:18 2024

[#0b986e 417GiB/1,717GiB(24%) CN:1 DL:78MiB ETA:4h43m55s] FILE: /home/temp/newgeth/base-mainnet-full-1731839014.tar.zst

[#0b986e 417GiB/1,717GiB(24%) CN:1 DL:53MiB ETA:6h58m13s] 11/19 12:59:24 [ERROR] CUID#7 - Download aborted. URI=https://mainnet-full-snapshots.base.org/base-mainnet-full-1731839014.tar.zst Exception: [AbstractCommand.cc:351] errorCode=22 URI=https://mainnet-full-snapshots.base.org/base-mainnet-full-1731839014.tar.zst -> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=403

11/19 12:59:24 [NOTICE] Download GID#0b986e92a962bf6a not complete: /home/temp/newgeth/base-mainnet-full-1731839014.tar.zst

Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= 0b986e|ERR | 79MiB/s|/home/temp/newgeth/base-mainnet-full-1731839014.tar.zst

Status Legend: (ERR):error occurred.

aria2 will resume download if the transfer is restarted. If there are any errors, then see the log file. See '-l' option in help/man page for details.

0x00101010 commented 4 days ago

Hey y'all, have you tried again after the failure, we hosted our snapshots on cloudflare, and this does seem like either a cloudflare issue or network issue.

Also FYI, there's some cloudflare maintenance going on that seems relevant: https://www.cloudflarestatus.com/

liusDong commented 4 days ago

Hey y'all, have you tried again after the failure, we hosted our snapshots on cloudflare, and this does seem like either a cloudflare issue or network issue.

Also FYI, there's some cloudflare maintenance going on that seems relevant: https://www.cloudflarestatus.com/

same question. I try more than 10 times always 403 error

11/20 09:31:59 [ERROR] CUID#7 - Download aborted. URI=https://mainnet-full-snapshots.base.org/base-mainnet-full-1731839014.tar.zst Exception: [AbstractCommand.cc:351] errorCode=22 URI=https://mainnet-full-snapshots.base.org/base-mainnet-full-1731839014.tar.zst -> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=403

11/20 09:31:59 [NOTICE] Download GID#8a9a7f7227700a58 not complete: /home/snap1/base-mainnet-full-1731839014.tar.zst

Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= 8a9a7f|ERR | 41MiB/s|/home/snap1/base-mainnet-full-1731839014.tar.zst

Status Legend: (ERR):error occurred.

aria2 will resume download if the transfer is restarted. If there are any errors, then see the log file. See '-l' option in help/man page for details.

0x00101010 commented 3 days ago

Trying 2 downloads to see if I could reproduce:

  1. base-mainnet-full-1731925706.tar.zst => latest
  2. base-mainnet-full-1731839014.tar.zst => the ones you mentioned
liusDong commented 3 days ago

Trying 2 downloads to see if I could reproduce:

  1. base-mainnet-full-1731925706.tar.zst => latest
  2. base-mainnet-full-1731839014.tar.zst => the ones you mentioned

I download latest

11/21 02:53:43 [ERROR] CUID#7 - Download aborted. URI=https://mainnet-full-snapshots.base.org/base-mainnet-full-1731925706.tar.zst Exception: [AbstractCommand.cc:351] errorCode=22 URI=https://mainnet-full-snapshots.base.org/base-mainnet-full-1731925706.tar.zst -> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=403

11/21 02:53:43 [NOTICE] Download GID#a702387759a7b2b3 not complete: /home/snap1/base-mainnet-full-1731925706.tar.zst

Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= a70238|ERR | 12MiB/s|/home/snap1/base-mainnet-full-1731925706.tar.zst

Status Legend: (ERR):error occurred.

aria2 will resume download if the transfer is restarted. If there are any errors, then see the log file. See '-l' option in help/man page for details.

0x00101010 commented 1 day ago

Trying 2 downloads to see if I could reproduce:

  1. base-mainnet-full-1731925706.tar.zst => latest
  2. base-mainnet-full-1731839014.tar.zst => the ones you mentioned

I download latest

11/21 02:53:43 [ERROR] CUID#7 - Download aborted. URI=https://mainnet-full-snapshots.base.org/base-mainnet-full-1731925706.tar.zst Exception: [AbstractCommand.cc:351] errorCode=22 URI=https://mainnet-full-snapshots.base.org/base-mainnet-full-1731925706.tar.zst -> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=403

11/21 02:53:43 [NOTICE] Download GID#a702387759a7b2b3 not complete: /home/snap1/base-mainnet-full-1731925706.tar.zst

Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= a70238|ERR | 12MiB/s|/home/snap1/base-mainnet-full-1731925706.tar.zst

Status Legend: (ERR):error occurred.

aria2 will resume download if the transfer is restarted. If there are any errors, then see the log file. See '-l' option in help/man page for details.

How long before your download was aborted?

I was able to replicate this myself, digging into this

0x00101010 commented 1 day ago

submitted ticket to cloudflare (where we host the snapshts) for triage, will update here once we hear back

usoftvn commented 1 day ago

I can't download with wget, I can download for a while and then have to download again. I download it takes about 10% to 30% and then the connection is lost and it asks to re-download. This happens constantly and I can't complete the download. Is there any way to fix this problem?

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 36 100 36 0 0 115 0 --:--:-- --:--:-- --:--:-- 115 --2024-11-23 03:17:52-- https://mainnet-full-snapshots.base.org/base-mainnet-full-1732178835.tar.zst Resolving mainnet-full-snapshots.base.org (mainnet-full-snapshots.base.org)... 2606:4700:4400::6812:2899, 2606:4700:4400::ac40:9367, 104.18.40.153, ... Connecting to mainnet-full-snapshots.base.org (mainnet-full-snapshots.base.org)|2606:4700:4400::6812:2899|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1871006057191 (1.7T) Saving to: ‘base-mainnet-full-1732178835.tar.zst’

base-mainnet-full-1732178835.tar.zst 0%[ ] 3.88G 7.31MB/s in 3m 16s

2024-11-23 03:21:09 (20.3 MB/s) - Connection closed at byte 4162936404. Retrying.

--2024-11-23 03:21:10-- (try: 2) https://mainnet-full-snapshots.base.org/base-mainnet-full-1732178835.tar.zst Connecting to mainnet-full-snapshots.base.org (mainnet-full-snapshots.base.org)|2606:4700:4400::6812:2899|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1871006057191 (1.7T) Saving to: ‘base-mainnet-full-1732178835.tar.zst’

base-mainnet-full-1732178835.tar.zst 0%[ ] 2.69G 31.9MB/s in 1m 46s

2024-11-23 03:23:00 (25.9 MB/s) - Connection closed at byte 4162936404. Retrying.

--2024-11-23 03:23:02-- (try: 3) https://mainnet-full-snapshots.base.org/base-mainnet-full-1732178835.tar.zst Connecting to mainnet-full-snapshots.base.org (mainnet-full-snapshots.base.org)|2606:4700:4400::6812:2899|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1871006057191 (1.7T) Saving to: ‘base-mainnet-full-1732178835.tar.zst’

base-mainnet-full-1732178835.tar.zst 0%[ ] 2.09G 28.9MB/s in 84s

2024-11-23 03:24:26 (25.7 MB/s) - Connection closed at byte 4162936404. Retrying.

--2024-11-23 03:24:29-- (try: 4) https://mainnet-full-snapshots.base.org/base-mainnet-full-1732178835.tar.zst Connecting to mainnet-full-snapshots.base.org (mainnet-full-snapshots.base.org)|2606:4700:4400::6812:2899|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1871006057191 (1.7T) Saving to: ‘base-mainnet-full-1732178835.tar.zst’

bab3nk0v commented 1 hour ago

I am experiencing the same issues. I believe the problem can be addressed by enabling the server to support the Accept-Ranges header. This would allow downloads to resume from where the connection was interrupted instead of restarting from the beginning.