Research alternatives to Hetzner for testnet hosting

[jakubgs]

Since Hetzner hates crypto mining and states so in their policy:

Therefore the following actions are prohibited:

• Operating applications that are used to mine crypto currencies

https://www.hetzner.com/legal/dedicated-server/

We will need to migrate out hosts to an alternative server provider soon. To do that we need to find a suitable alternative.

The general requirement involve:

• CPU: 8+ CPUs
• RAM: 32+ GB
• DISK: ~200 GB OS(mirror), 1+ TB SSD data
• Ability to add new drives as storage requirements change.

We'll need about 21 hosts with similar specs.

[jakubgs]

Here's a lost of some possible hosting services which provide metal/root servers:

• [x] - https://www.netcup.eu/vserver/ - Anti-mining policy
• [ ] - https://upcloud.com/products/private-cloud
• [ ] - https://hostkey.com/dedicated-servers/europe/
• [ ] - https://luxhosting.com/dedicated-servers/ - Expensive
• [x] - https://www.ovhcloud.com/en/bare-metal/ - Anti-mining policy.
• [ ] - https://www.eurovps.com/dedicated-servers - Expensive
• [x] - https://innovahosting.net/dedicated-server-hosting-in-europe - No policy against.
• [x] - https://www.ctrlservers.com/dedicated-servers-germany/ - Very expensive support (50 EUR for drive replacement service)
• [ ] - https://www.fastcomet.com/dedicated-servers-pricing - Expensive
• [x] - https://serverhino.com/dedicated/ - Question sent
• [x] - https://www.scaleway.com/en/elastic-metal/aluminium/ - Anti-mining policy.
• [x] - https://www.scaleway.com/en/elastic-metal/beryllium/ - Anti-mining policy.

[jakubgs]

I'm also discussing other options for hosting with a representative that can help us get a contract with one of these:

• Equinix - https://www.equinix.se/
• ZenLayer - https://www.zenlayer.com/
• Lumen - https://www.lumen.com/
• PhoenixNAP - https://phoenixnap.com/

[jakubgs]

The bandwidth requirements appear to be currently ~150 Mb/s ingress and 250 Mb/s egress:

This min/max graph actually shows it better:

But that will probably change with time.

[jakubgs]

Actually, if we look at 30 days there are some big 600 Mb/s spikes:

[jakubgs]

According to the OVH Cloud policy for Virtual Private Server ("VPS") in Customer Obligations section:

Customer is prohibited from using or allowing the Services to be used for any intrusive activity or any intrusion attempts (including, but not limited to port scans, sniffing, spoofing), and any activity or contentious behavior such as traffic exchanging (Hitleap, Jingling), Black Hat SEO (downloading and uploading videos from and to online gaming platforms), crypto-currency mining, video game bots, etc. Anonymization services or public proxy (including VPN, Tor, P2P, IRC) and cardsharing (CCCam or equivalent) are not permitted on the Services.

https://us.ovhcloud.com/legal/service-specific-terms

But I see no such policy in the Customer Obligations section for dedicated servers, only:

The Customer accepts sole responsibility for the performance of any and all maintenance operations in reference to the Service(s), including but not limited to any updates to the distribution installed on the Service(s) and upgrades.

But sending a separate email to their sales team is probably a good idea to verify.

[jakubgs]

I have called OVH Cloud sales team in Poland and they have told me that the policy has not been updated but the same rules as for VPS also applies to dedicated servers. Just to be sure I also sent an email to sales with the same query:

I'd would like to know what is your policy regarding services related to blockchain networks(testnets specifically) on your dedicated servers. I see you have a restriction in your policy in the VPS section, but no such restriction exists in the dedicated server section. Is that intentional?

But it looks like OVH Cloud will not work out.

EDIT: I've also called their sales team in UK and they checked the French entity policy based on our org being from Switzerland and they said it's against the policy. I've been sent this link as reference:

The Client refrains, as part of or through the Service, from making any activity or behaviour that is contentious such as traffic exchanges (Hitleap, Jingling, etc.), Black Hat SEO (Download, Reupload of videos on online video platforms, etc.), cryptocurrency mining, video game bots, etc.

https://storage.gra.cloud.ovh.net/v1/AUTH_325716a587c64897acbef9a4a4726e38/contracts/df42916-contrat_partDedie-GB-9.1.pdf https://storage.gra.cloud.ovh.net/v1/AUTH_325716a587c64897acbef9a4a4726e38/contracts/2fbf086-contrat_partDedie-FR-20.0.pdf

[jakubgs]

Scaleway seems to have sensible pricing:

https://www.scaleway.com/en/elastic-metal/aluminium/

And the Terms of Use do not seem to have anything about crypto, but I'd need to verify: https://www.scaleway.com/en/terms/

[jakubgs]

I've sent their sales team a question:

I'd would like to know what is your policy regarding services related to blockchain networks(Ethereum testnets specifically) on your dedicated servers. I cannot find any restriction in your Terms of Service relate to blockchain or mining, but I would like to verify that you have no such policy, nor are you planning to institute such a policy in the foreseeable future. We are looking for a service to migrate our Ethereum development servers from Hetzner due to their anti-mining policy, but we wouldn't want to jump out of the frying pan into the fire.

[jakubgs]

Innovahosting seems to have some sensible prices for dedicated servers:

https://innovahosting.net/dedicated-server-hosting-in-europe

And I see no references to crypto or mining in ther Terms of Service: https://innovahosting.net/terms

Sent a query to their sales team:

I would like to know what is your policy regarding services related to blockchain networks(Ethereum testnets specifically) on your dedicated servers. I cannot find any restriction in your Terms of Service relates to blockchain or mining, but I would like to verify that you have no such policy, nor are you planning to institute such a policy in the foreseeable future.

We are looking for a service to migrate our Ethereum development servers from Hetzner due to their anti-mining policy, but we wouldn't want to jump out of the frying pan into the fire.

[jakubgs]

I've received a response from InnovaHosting:

Hello,

WE allowing crypto mining on servers, but we will increase price, due to increased typical electricity usage.

Best regards

Technical Support, IP HOST

They are probably thinking of proof-of-work, so that comment about electricity usage is probably not applicable to us.

[jakubgs]

I've received a response from Scaleway support:

Thank you very much for your request. Regarding your question about the policy related to blockchain networks(Ethereum testnets specifically and mining) i can say to you that we do have a policy about mining and mining is a no go at this moment. only chia mining is a option but there should be some more detailed conversations about that.

For any other questions just let me know, if i can help you with other services just send me a email or we can schedule a short meeting online.

Wish you a nice evening!

No idea why they think Chia is okay, but all other mining is bad, even proof-of-stake testnets.

[jakubgs]

I've also contacted NetCup support: https://www.netcup.eu/support/

Their prices looks sensible:

EDIT: Looks like they have a policy against it: https://www.netcup.eu/bestellen/agb.php

The following activities are expressly prohibited:

• Operation of mining-services such as "Bitcoin", "Ethereum", "OneCoin" or "Monero".

[jakubgs]

I've also contacted the ServeRhino support: https://serverhino.com/support/

The R-Hexa and R-Octo root servers look decent::

EDIT: Actually, there is a policy against it in "Restrictions When Using Services":

4.3. use of the services to receive cryptocurrencies (mining) is prohibited for all the services, except the designated physical servers.

Not sure what they mean by "designated physical servers", so I'll ask about that:

I would like to know what is your policy regarding services related to blockchain networks - specifically Ethereum proof-of-stake testnets - on your dedicated root servers. I found this line in your policy which I'd like to clarify:

4.3. use of the services to receive cryptocurrencies (mining) is prohibited for all the services, except the designated physical servers.

We are looking for a service to migrate our Ethereum development servers from Hetzner due to their anti-cryptocurrency policy to your root servers, but we wouldn't want to jump out of the frying pan into the fire. We are NOT looking to do "mining", we intend to run Ethereum test networks(testnets) for the sake of development of Ethereum software, and not "mining". There is no actual valuable cryptocurrencies being generated on our testnet servers.

Would that be compatible with your policy?

Cheers

[jakubgs]

I've checked the Terms of Service for CtrlServers and found this: https://www.ctrlservers.com/terms-of-service/

Payments made via cryptocurrency / virtual currency will not be refunded via cryptocurrency / virtual currency due to the fluctuating nature of cryptocurrency / virtual currency. Instead a credit will be placed on account. This applies to any and all cryptocurrency / virtual currency.

And nothing else related to cryptocurrencies, so I guess they might be fine with mining. But I sent an email to sales@ctrlservers.com anyway.

[jakubgs]

Response from CtrlServers:

We don't have any such policy to restrict blockchain development or mining but we don't allow projects or activities which involved in steeling, fraud, gambling or similar coins / mining / projects which attract complainants. Also we don't allow to change power profile of servers which damage hardware in long run.

If above terms suits you, may i know what sort of hardware requirement you have and how many servers you are looking?

I provided them with our requirements.

[jakubgs]

It appears they could give us a discount on a bulk order of 22 servers:

Good morning, Please provide details about your avg bandwidth consumption per server and do you plan to order 22 servers at once? So i can get approval for special discount in this case.

We can offer you something like

Dual Intel Xeon (6 or 8 cores per processor) 64GB RAM DDR3 / DDR4 2x 480GB Write Intensive SSD (Hardware RAID1 - Mirror) 1x 960GB Mix Use SSD 1IP IPMI Access

Please let me know if you want any specific change in above configuration.

Kind Regards,

Though the extra storage SSD will need adjustment. We'd probably need:

• Sepolia - 2x 1 TB SSD
• Prater - 2x 1 TB SSD
• Mainnet - 2x 2 TB SSD

Sent them also rough bandwidth requirements:

• Sepolia - ~10 Mb/s igress, ~20 Mb/s egress (3 node)
• Prater - ~60 Mb/s igress, ~100 Mb/s egress (10 nodes)
• Mainnet - ~80 Mb/s igress, ~300 Mb/s egress (8 nodes)

[jakubgs]

Their offer is 169EUR per month for this:

Dual Intel E5-2620 Xeon (2x 6 cores @ 2.00 Ghz)
64GB RAM DDR3
2x 960GB Mix Use SSD
2x 1.92TB Mix Use SSD
100Tb Data Traffic
1Gbps Dedicated Port
1IP
IPMI Access

Which is not amazing, considering we pay 37.3 EUR(AX-41) + 2x 20.9 EUR(2 TB SSD) = 79.1 EUR, but best so far.

Ticket URL: https://www.ctrlservers.com/viewticket.php?tid=781855&c=XuSzHsLy

[jakubgs]

Zahary pointed out that 2 Ghz is kinda low and he had issues syncing on a server like that:

Aim for the fastest single-core performance. The number of cores is less important

I asked them about other available CPUs:

As for CPU, we're more interested in higher single core performance than number of cores. The 2 Ghz of E5-2620 is kinda low for us. For that reason I'd be more interested in CPUs like E3-1270 v6, E-2186G, or E-2274G. Are those also possible in those combinations?

[jakubgs]

Another offer for 139EUR per month:

Intel E3-1270v6 (4 cores, 8 threads @ 4.20 GHz)
32GB DDR4 RAM
2x 240GB SSD (Software RAID1)
2x 960GB SSD
100TB Traffic
1Gbps Dedicated Port
1IP

That's pretty nice. But I think we'd need 64 GB RAM, which would be 162EUR.

[jakubgs]

I called OVHCloud UK sales team to confirm, and according to them it is prohibited:

The Client refrains, as part of or through the Service, from making any activity or behaviour that is contentious such as traffic exchanges (Hitleap, Jingling, etc.), Black Hat SEO (Download, Reupload of videos on online video platforms, etc.), cryptocurrency mining, video game bots, etc.

https://storage.gra.cloud.ovh.net/v1/AUTH_325716a587c64897acbef9a4a4726e38/contracts/df42916-contrat_partDedie-GB-9.1.pdf https://storage.gra.cloud.ovh.net/v1/AUTH_325716a587c64897acbef9a4a4726e38/contracts/2fbf086-contrat_partDedie-FR-20.0.pdf

[jakubgs]

I tried paying for the invoice created by CtrlServers staff, but it seems to have only option to pay via paypal:

And it seems to force you to create an account:

Normally there's a checkbox to pay as a guest, but not here. I'm starting to reconsider using this provider.

[jakubgs]

I explained my issues with payment and this is what I got:

I am sorry but we have no control over paypal flow. It's all depends upon location. May be you can try using "Credit Card" instead of selecting paypal? but it also available location wise. Most of user can see Paypal and Credit Card both options on 2checkout page and some only Paypal.

It might be that we have to pay with a bank transfer because I'm not creating a Paypal account.

[jakubgs]

I have received a Stripe link from CtrlServers and that worked well with a Revolut card.

We should get a server shortly and then we can test it by deploying some Prater nodes without validators.

[jakubgs]

We received the host, but since it has two separate data volumes I had to refactor the bootstrap role:

• https://github.com/status-im/infra-role-bootstrap-linux/commit/43b550d7 - volumes: move DC specific files to dcs folder
• https://github.com/status-im/infra-role-bootstrap-linux/commit/9e1e1067 - volumes: extract volume creation to separate file
• https://github.com/status-im/infra-role-bootstrap-linux/commit/4f0709af - volumes: generic volume finder for most providers
• https://github.com/status-im/infra-role-bootstrap-linux/commit/9cbaa6c4 - volumes: support multiple extra data volumes

And now I can finally add the new host with two volumes:

• https://github.com/status-im/infra-nimbus/commit/c05fad0e - add linux-01.cs-eu-nld1.nimbus.prater host

And the volumes appear to be mounted correctly:

admin@linux-01.cs-eu-nld1.nimbus.prater:~ % df -h /data /docker
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        880G   28K  835G   1% /data
/dev/sdd        880G   28K  835G   1% /docker

Although maybe I should have went with 2TB from the start.

I added it without any validators to nimbus.prater fleet configuration: https://github.com/status-im/infra-nimbus/blob/c05fad0e01ca60a08d3780d19fedc0f54231176f/ansible/group_vars/nimbus.prater.yml#L154-L158

[jakubgs]

The nodes are up and syncing:

admin@linux-01.cs-eu-nld1.nimbus.prater:~ % for port in $(seq 9300 9303); do c 0:$port/eth/v1/node/syncing | jq -c; done
{"data":{"head_slot":"17960","sync_distance":"4865917","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"16916","sync_distance":"4866961","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"6120","sync_distance":"4877757","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"440","sync_distance":"4883437","is_syncing":true,"is_optimistic":false}}

admin@linux-01.cs-eu-nld1.nimbus.prater:~ % for dir in /docker/geth-*; do $dir/rpc.sh eth_syncing | jq -c; done         
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}

Disk utilization is fairly low:

[jakubgs]

The beacon nodes are syncing:

admin@linux-01.cs-eu-nld1.nimbus.prater:~ % for port in $(seq 9300 9303); do c 0:$port/eth/v1/node/syncing | jq -c; done
{"data":{"head_slot":"361644","sync_distance":"4525781","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"498069","sync_distance":"4389356","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"344957","sync_distance":"4542468","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"331474","sync_distance":"4555951","is_syncing":true,"is_optimistic":false}}

With CPU load of about 5.0 on a machine with 8 cores:

Low memory usage but plenty of caching and no SWAPing so far:

And traffic of about 10 Mb/s:

[jakubgs]

Looks like sync failed because one of the drives dedicated to Geth got filled up:

admin@linux-01.cs-eu-nld1.nimbus.prater:/docker % sudo du -hsc /docker/*        
206G    /docker/geth-goerli-01
203G    /docker/geth-goerli-02
235G    /docker/geth-goerli-03
192G    /docker/geth-goerli-04
16K /docker/lost+found
835G    total

admin@linux-01.cs-eu-nld1.nimbus.prater:/docker % df -h /docker
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdd        880G  835G     0 100% /docker

We probably should go with just 2 TB drives for all hosts.

[jakubgs]

I've contacted their support about replacing the drives with 2 TB ones.

[jakubgs]

Lol:

Its possible to add / remove disks but each add / remove require 50EUR remote hands fee

That's silly expensive.

[jakubgs]

I've created an account with InnovaHosting and will purchase an example dedicated server from them. Their DIX11 with a 2.9 Ghz CPU host is probably the best option for us:

But it comes out to 158.99 EUR per month with 64 GB of RAM and 2x 1.6 TB SSDs:

If billed annually then 151 EUR per month, which is 20 EUR lower than CtrlServers offer. Though it would be nice to have a 3rd SSD or drive just for the operating system.

[jakubgs]

I've created a ticket for their sales team:

I'm looking to move several (22 at least) servers from Hetzner to another provider due to their anti-mining policy, which affects us since we do run Ethereum testnet nodes on their hardware currently. We would like to migrate to a provider that doesn't have such policies and as far as I know you do not.

We are interested in a bit of a custom setup. We will need an example host with these parameters:

• CPU with high single core performance, at least 2.9Ghz
• 64 GB of RAM
• 1x small SSD/HDD for OS
• 2x big 1.5+ TB SSD for data WITHOUT RAID

Our use case is running nodes on Ethereum testnets like Prater/Goerli which require lots of storage, but are not production so they do not require high availability or high reliance, as they are only for testing new Ethereum software versions.

As far as I can tell the DIX11 server fits out uses case well, although I would like to have that extra separate SSD/HDD for OS. Is that possible?

My second question is about replacing drives in the future. As our storage needs grow, will it be possible to replace the existing drives with bigger ones in the future? Or would that require purchasing new servers and discontinuing those?

My third question is about location. I don't see an option to pick a location during checkout. Did I miss it or is it not there? Would it be possible to make sure our servers are in europe?

My fourth question is whether you'd have any additional discount to offer if we were going to purchase at least 22 servers of such kind with annual payment?

https://client.innovahosting.net/viewticket.php?tid=874663&c=hAzEsLTR

[jakubgs]

I got a response from support:

We can allocate 22 servers only in Moldova, Chisinau Our top CPU is E5-2680v4

We do have SSD for 800GB and 1.6TB. As for HDD's 1TB, 2TB,4TB,8TB.

We can make a configuration from this elements if yo uare Ok with the location

So I asked some more questisons:

That's good to know, but first I would like to test one server config to see how well it works. Also, E5-2680v4 is far too much, we would be fine with 4 or 6 cores, we just 2.9 Ghz or more frequency. We're more interested in strong single core performance.

Also you didn't answer my other questions so I'll reiterate and expand:

1. Would you have DIX11 servers available in Moldova, Chisinau?
2. Can SSDs be replaced during lifetime of a server with bigger ones?
3. Any chance for smaller SSDs for operating system, like 240 GB?
4. Is there a replacement fee in addition to the increased cost due to bigger SSD?
5. Would we get an extra discount for a bulk order of 22 servers for a year?

Cheers!

Lets see what they say.

[jakubgs]

Response:

1) DIX servers are out of stock for the moment

• We can make combinations from this servers https://innovahosting.net/en/dedicated-server-hosting
• Please check the CPU's
• Maybe some of this would be Ok for you. 2) yes we can replace the disks anytime 3) Now we buy only 800Gb SSD as the smalles one buy we may have a few SSD's for 400GB. 4) We do not charge fee for replaycment, but your monthly cost may increase if the server will be more performant 5) Sure, we can offer an discount but will be able to talk about discounts after we know for sure your configuration

Which I guess means the best option for us is an Advanced-2 or Advanced-3 server:

Just with 1x 800GB SSD and 2x1.6TB SSD.

So I asked:

I see, thanks.

Would it be possible to get this kind of config:

• HP DL360 Gen8
• Single E5-2665v2 or E5-2690v2
• 64 GB RAM
• 1x800 GB SSD
• 2x1.6 TB SSD

Is that possible? Cheers

[jakubgs]

We got an offer from InnovaHost:

For this configuration the price will be 190$ If you will order more servers we can apply 5%-10%-15% discount, depending on quantity

PS: we have 200 and 400GB SSD, if you want to replace 800GB with 200 or 400GB we can change the price from 190$ to 185$

HP DL360 Gen8 Single E5-2665v2 64 GB RAM 1x800 GB SSD 2x1.6 TB SSD

185 USD per month without any discounts applied is a pretty decent offer for this setup. I will get one server to test.

[jakubgs]

I have cancelled the CtrlServers server and our first invoice covered its cost:

Since I cancelled the host yesterday and the first invoice was covering dates 24/01/2023 - 23/02/2023. So no further payment necessary for this service.

• https://github.com/status-im/infra-nimbus/commit/297f3479 - drop linux-01.cs-eu-nld1.nimbus.prater host

[jakubgs]

Bootstrapped and configured the new host from Innova Hosting:

• https://github.com/status-im/infra-nimbus/commit/6e05ef6e - nimbus.prater: add linux-01.cs-eu-nld1.nimbus.prater

Geth and Beacon nodes are up:

admin@linux-01.ih-eu-mda1.nimbus.prater:~ % d
CONTAINER ID   NAMES                     IMAGE                             CREATED              STATUS
3efb87ae8d93   geth-goerli-03-exporter   statusteam/geth_exporter:latest   7 seconds ago        Up 6 seconds
27179793ad86   geth-goerli-03-node       ethereum/client-go:v1.11.1        7 seconds ago        Up 6 seconds
3f31661d4eaa   geth-goerli-02-exporter   statusteam/geth_exporter:latest   28 seconds ago       Up 27 seconds
5cc2ceaabb03   geth-goerli-02-node       ethereum/client-go:v1.11.1        28 seconds ago       Up 27 seconds
13085889bec2   geth-goerli-01-exporter   statusteam/geth_exporter:latest   48 seconds ago       Up 47 seconds
1251018763c7   geth-goerli-01-node       ethereum/client-go:v1.11.1        49 seconds ago       Up 48 seconds
1d620561e313   geth-goerli-04-node       ethereum/client-go:v1.11.1        About a minute ago   Up About a minute

admin@linux-01.ih-eu-mda1.nimbus.prater:~ % for dir in /docker/geth-*; do $dir/rpc.sh eth_syncing | jq -c; done
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}

admin@linux-01.ih-eu-mda1.nimbus.prater:~ % for port in $(seq 9300 9303); do c 0:$port/eth/v1/node/syncing | jq -c; done
{"data":{"head_slot":"61071","sync_distance":"4982539","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"101789","sync_distance":"4941821","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"57836","sync_distance":"4985774","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"19896","sync_distance":"5023714","is_syncing":true,"is_optimistic":true}}

[jakubgs]

So far so good. The load is around 5.2 and memory usage is stable:

Traffic hovers around 10 Mb/s both ways:

Disk utilization is low, no I/O wait:

So so far SSD over SATA seems to work fine.

admin@linux-01.ih-eu-mda1.nimbus.prater:~ % for port in $(seq 9300 9303); do c 0:$port/eth/v1/node/syncing | jq -c; done
{"data":{"head_slot":"212926","sync_distance":"4833525","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"387053","sync_distance":"4659398","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"210637","sync_distance":"4835814","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"169033","sync_distance":"4877418","is_syncing":true,"is_optimistic":true}}

[jakubgs]

At this sync speed it might take two weeks for Beacon Nodes to fully sync from genesis.

[jakubgs]

With current speed the sync should finish in 9 days:

admin@linux-01.ih-eu-mda1.nimbus.prater:~ % for port in $(seq 9300 9303); do c 0:$port/eth/v1/node/syncing | jq -c; done
{"data":{"head_slot":"1080654","sync_distance":"3979731","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"1736828","sync_distance":"3323557","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"1074103","sync_distance":"3986282","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"1037246","sync_distance":"4023139","is_syncing":true,"is_optimistic":true}}

admin@linux-01.ih-eu-mda1.nimbus.prater:~ % sudo du -hsc /data/*          
20G /data/beacon-node-prater-libp2p
21G /data/beacon-node-prater-stable
21G /data/beacon-node-prater-testing
21G /data/beacon-node-prater-unstable

[jakubgs]

I just noticed that we have an old setting that makes most nodes use 1 thread: https://github.com/status-im/infra-nimbus/blob/3e3cff68408cf5a42262fb6fa0d99aeaa85ba90f/ansible/group_vars/nimbus.prater.yml#L53 Which possibly affects speed of syncing from scratch. So I've adjusted the setting

• https://github.com/status-im/infra-nimbus/commit/15d96ca3 - nimbus.mainnet/prater: use more threads than 1

Now we just divide CPUs by number of nodes for all nodes except ones using libp2p branch which will use 1 thread: https://github.com/status-im/infra-nimbus/blob/15d96ca3be374420c491f7d8be365bb3d1671af0/ansible/group_vars/nimbus.prater.yml#L53-L54

[jakubgs]

We can clearly see increased load due to using more threads:

But traffic looks about the same:

Though there is a mild increase in writes:

So it did something I guess.

[jakubgs]

Here's the current progress:

admin@linux-01.ih-eu-mda1.nimbus.prater:~ % for port in $(seq 9300 9303); do c 0:$port/eth/v1/node/syncing | jq -c; done
{"data":{"head_slot":"2891601","sync_distance":"2190709","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"3471284","sync_distance":"1611026","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"2883019","sync_distance":"2199291","is_syncing":true,"is_optimistic":true}}
{"data":{"head_slot":"2253230","sync_distance":"2829080","is_syncing":true,"is_optimistic":true}}

At this rate the Beacon nodes should be synced within the next 4 days. Which would be a total of 10 days.

[jakubgs]

The traffic looks unchanged:

But enabling more threads certainly affected I/O:

[arnetheduck]

Here's the current progress:

which versions are being run here?

[jakubgs]

Same as always: https://github.com/status-im/infra-nimbus/blob/4e20fd2c149d9c873af630a5acb14ec45f2a618f/ansible/group_vars/nimbus.prater.yml#L162-L166

[jakubgs]

Honestly, it's kinda slow, but it's definitely not the hardware, since that's not really being fully utilized here as far as I can tell.

[arnetheduck]

but does the order of responses correspond to those versions?

[jakubgs]

Also, unstable branch seems to be syncing considerably faster than others.

[jakubgs]

Yes, the ports are incremented based on the order in the list:

admin@linux-01.ih-eu-mda1.nimbus.prater:~ % s cat beacon-node-prater-stable | grep rest-port
    --rest-port=9300 \
admin@linux-01.ih-eu-mda1.nimbus.prater:~ % s cat beacon-node-prater-testing | grep rest-port
    --rest-port=9301 \
admin@linux-01.ih-eu-mda1.nimbus.prater:~ % s cat beacon-node-prater-unstable | grep rest-port
    --rest-port=9302 \
admin@linux-01.ih-eu-mda1.nimbus.prater:~ % s cat beacon-node-prater-libp2p | grep rest-port  
    --rest-port=9303 \

[jakubgs]

Average load hangs around 6, and we have 10 cores, so we are under-utilizing the host so far.