search

ArweaveTeam / arweave

The Arweave server and App Developer Toolkit.
https://www.arweave.org
GNU General Public License v2.0
920 stars 204 forks source link

Data Synchronization Imbalance in Coordinated Mining Setup using one public IP #591

Closed kslazarev closed 3 months ago

kslazarev commented 4 months ago

I've encountered an issue with data synchronization in a coordinated mining setup using one public IP and three local machines.

The setup is as follows:

Public IP: 95.54.82.50 Local IPs: 192.168.0.1, 192.168.0.2, 192.168.0.3

Port forwarding (router):

95.54.82.50:1984 -> 192.168.0.1:1984 95.54.82.50:1985 -> 192.168.0.2:1985 95.54.82.50:1986 -> 192.168.0.3:1986

Each node is running Arweave with the corresponding port argument (1984, 1985, 1986).

Issue: The node using port 1984 consistently shows a significantly higher number of synchronization requests compared to the nodes on ports 1985 and 1986.

Screenshot 2024-07-17 at 02 36 00

Test: To rule out partition-related issues, I switched the port forwarding:

95.54.82.50:1985 -> 192.168.0.1:1985 95.54.82.50:1984 -> 192.168.0.2:1984

Result: The node now using port 1984 (previously on 1985) started making more requests than the other two nodes.

Screenshot 2024-07-17 at 02 39 32

Questions:

Any guidance on properly using coordinated mining with one public IP would be greatly appreciated. Thank you for your time and assistance.

Build arweave from master: e39b4cc8af5eb308a700a7fad6886288539b64a0

Configs:

95.54.82.50:1984 ``` ./bin/start \ data_dir data_dir \ mining_addr "GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ peer ams-1.eu-central-1.arweave.net peer fra-1.eu-central-2.arweave.net \ peer sgp-1.ap-central-2.arweave.net peer blr-1.ap-central-1.arweave.net peer sfo-1.na-west-1.arweave.net \ enable randomx_large_pages \ enable randomx_hardware_aes \ enable randomx_jit \ storage_module "1,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "4,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "8,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "13,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "14,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "17,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "19,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "24,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "26,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "27,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "32,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "33,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "38,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "39,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "46,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "47,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "50,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "51,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ enable vdf_server_pull \ vdf_server_trusted_peer "vdf-server-3.arweave.xyz" \ vdf_server_trusted_peer "vdf-server-4.arweave.xyz" \ local_peer "95.54.82.50:1985" \ local_peer "95.54.82.50:1986" \ coordinated_mining \ cm_peer "95.54.82.50:1985" \ cm_peer "95.54.82.50:1986" \ cm_api_secret "********" \ mine \ port 1984 ```
95.54.82.50:1985 ``` ./bin/start \ data_dir data_dir \ mining_addr "GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ peer ams-1.eu-central-1.arweave.net peer fra-1.eu-central-2.arweave.net \ peer sgp-1.ap-central-2.arweave.net peer blr-1.ap-central-1.arweave.net peer sfo-1.na-west-1.arweave.net \ enable randomx_large_pages \ enable randomx_hardware_aes \ enable randomx_jit \ storage_module "2,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "5,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "7,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "15,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "16,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "20,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "21,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "22,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "23,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "28,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "30,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "35,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "36,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "37,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "41,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "43,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ enable vdf_server_pull \ vdf_server_trusted_peer "vdf-server-3.arweave.xyz" \ vdf_server_trusted_peer "vdf-server-4.arweave.xyz" \ local_peer "95.54.82.50:1984" \ local_peer "95.54.82.50:1986" \ coordinated_mining \ cm_peer "95.54.82.50:1984" \ cm_peer "95.54.82.50:1986" \ cm_exit_peer "95.54.82.50:1984" \ cm_api_secret "********" \ mine \ port 1985 ```
95.54.82.50:1986 ``` ./bin/start \ data_dir data_dir \ mining_addr "GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ peer ams-1.eu-central-1.arweave.net peer fra-1.eu-central-2.arweave.net \ peer sgp-1.ap-central-2.arweave.net peer blr-1.ap-central-1.arweave.net peer sfo-1.na-west-1.arweave.net \ enable randomx_large_pages \ enable randomx_hardware_aes \ enable randomx_jit \ storage_module "0,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "3,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "6,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "9,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "10,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "11,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "12,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "18,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "25,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "29,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "31,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "34,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "40,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "42,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "44,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "45,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "48,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ storage_module "49,GVr0CkdQU9TIutkf1eTtm9Ud3Wp-X9rtdIXCHdrlNMM" \ enable vdf_server_pull \ vdf_server_trusted_peer "vdf-server-3.arweave.xyz" \ vdf_server_trusted_peer "vdf-server-4.arweave.xyz" \ local_peer "95.54.82.50:1984" \ local_peer "95.54.82.50:1985" \ coordinated_mining \ cm_peer "95.54.82.50:1984" \ cm_peer "95.54.82.50:1985" \ cm_exit_peer "95.54.82.50:1984" \ cm_api_secret "******" \ mine \ port 1986 ```
kslazarev commented 3 months ago

I've continued to monitor the process, and I'd like to provide an update:

After further observation, it appears that this may not be a critical problem. While there are still some anomalies in the number of synchronization requests from different elements of the coordinated node on graphana, I've noticed that the actual data volume has become more evenly distributed across each machine over the past 2-3 days.

For now, I plan to close this ticket. If I notice any significant changes or if the situation worsens, I will open a new ticket with updated information.