Support offer for testing GPU and NRD approach

[ag3ntt]

Hi man,

I am very grateful and a big fan of your activities in the field of Chia plotting optimization - especially in the field of GPU and at the same time in the field of NRD. I myself run a 1.9 PiB Chia farm and am a hardware and software enthusiast (especially Linux automated with Ansible). In the coming months, Chia will release support for compressed plots, so I'm slowly preparing for a complete re-plotting of my farm. I will also be happy to provide my plotting capacity to my friends or other interested people for a fee.

I support the green-politics of plotting and will gladly lend my time and resources to your support, at least in testing your various approaches.

In which areas will I be able to help:

• GPU plotter testing (I have both Nvidia and AMD GPUs)
• NRD testing (I have a 10 GbE network, powerful HW and plan to purchase 40 GbE QSFP+ to connect some devices)
• In 1 month's time, I plan to physically organize my homelab better - now it is garage punk :) Then I will be able to turn on the required computers or servers at your request, make my network accessible using Wireguard VPN and prepare SSH accounts for you there with your SSH key. I would first draw the network topology, including IP addresses and installed HW.

My homelab - network and servers:

• 10 GbE switch MikroTik CRS312-4C+8XG-RM
• 2x Dell FX2S blades with internal 10 GbE switches and 7x FC630 (2 CPU sockets) or FC830 (4 CPU sockets) servers
• Each server in the Dell FX2S has from 128 GB to 512 GB ECC RAM (2133 MHz or 2400 MHz, 32 or 64GB modules) and uses 20-22 cores Xeon processors 2699v4, 2698v4 or 4650v4.
• Lot of enterprise SAS SSDs (Toshiba PX04SMQ160)
• SuperMicro X10-QBL with 4x 8891v4 and 256GB DDR3 @ 1600 MHz. In the next few days, I will expand the RAM to 512 GB and replace the CPU with 4x 8890v4.

My homelab - desktop computers:

• Desktops: 4x Ryzen 5950x, 1x 3950x, 2x 5600x, each with 32 to 128GB RAM (3200/3600 MHz) with 1 or 2.5 GbE LAN (or 10 GbE Intel X540-AT2)
• About 20 NVME/SSD 1TB MLC/TLC drives (Samsung 970 Pro, 980 Pro, 860 Pro, WD SN850, Gigabyte Gen 4)
• High-endurance PCIe drives (Intel Optane 900p 280 GB and 480 GB, Samgung 983 ZET 960 GB)
• GPUs Nvidia: 1x RTX3090, 2x RTX3080, 6x RTX3070, 2x RTX3070 Ti
• GPUs AMD: 2x 6900 XT, 2x 6800 XT (but I know that for now arc plotter supports only CUDA, so it is Nvidia only)
• Other accessories: LSI 9211-8i HBAs, SAS Expanders, PCIe 3.0 x16 extension cables, riser cards, high quality PSUs, etc.

What I plan to buy/put together in the next 1-2 months:

• Mellanox CX314A QSFP+ 40 GbE cards + QSFP+ cables
• Cabel Mikrotik XQ+BC0003-XS+ (QSFP+ to 4x SFP+)
• Powerful desktop with 13900KF + 128GB DDR5 @ 5600 MHz (I am also considering the AMD variant with Ryzen 7950x)
• Supermicro H11DSi-NT with 2x AMD Epyc 7742 (2x 64 cores) and 512 GB DDR 4 @ 3200 MHz.

In the coming hours and days, I'll run my first arc plotter benchmarks and submit them as a GitHub ticket, or set up some meaningful Google spreadsheet for record keeping.

Thank you for your work and I believe that my efforts will help you or community.

Btw, Happy New Year 2023! 😊

[smartbitcoin]

@ag3ntt , thank you so much for you kindly offer. my big issue for arc_plot is I don't have enough resource ( hardwares ) for testing and verify the design, I only have two PC one is new gen 13 intel which just build for dev and another old AM4 amd . Your helps really change all the things like compatibility , stability and best setup for GPU plotting. Few things I really want the answers is: 1.) Performance gain from DDR5 ? my best ram still ddr4 3200. so I have no idea what 's the ratio ddr5 's gain in plotting.

2.) GPU now is much faster than server with 512G ddr3. so 40G nic with such server re-purpose to a NRD will beat best SSD or just not a option ?

3.) what's the best stable SSD setup for high speed GPU plotting? now GPU plotting will have file write error if ssd was too slow. but how to define the speed of SSD as minimum requirement is still vague answer.

4.) rtx 4090 's performance ? I just curious the next gen GPU raw power and how to optimize it.

Except git ticket, we still have a place called discussion here. so please feel free to put anything here https://github.com/smartbitcoin/arc_plot/discussions

[ag3ntt]

Hi man,

thank you for fast answer. I will try to answer your "questions" from my point of view.

1) I think DDR5 frequencies will be very useful. I will try to speed up my 13900KF or 7950x order with 128GB @ 5600 MHz memory.

2) After the first benchmarks with RTX 3090, 3080 and 3070/3070 Ti I will see the implications on temporary storage. For storage I'll test an Intel Optane superfast PCIe card, then a high end consumer grade MLC NVME (Samsung 970 Pro), some mid-range SSDs like the Crucial MX500 and of course NRDs. I am very curious to see what speeds the NRD will achieve and if it can utilize more than 10 GbE. It's going to be a lot of experimenting and measuring + charting, but I'm enjoying it.

3) I'll try to help define the exact minimum storage requirements (latency, sequential/random read/write, etc). I'll also try to use some 7.2k RPM HDD or some oldest SSD to better understand the storage aspect requirements.

4) I'll look at the budget and if possible buy an RTX 4090 for our testing purposes :)

All these scenarios and possibilities/combinations are very interesting. I'm very interested in what will be the most efficient ways ($$$ for electricity for 1TB of plots) when I can combine GPUs, NRD, modern desktop CPUs with 128GB limit or repurposed servers with 1-2TB ECC RAM (DDR3 1600, DDR4 2400 or 3200 MHz).

Have a nice day.

Jan

pá 30. 12. 2022 v 16:12 odesílatel SmartBitCoin @.***> napsal:

@ag3ntt https://github.com/ag3ntt , thank you so much for you kindly offer. my big issue for arc_plot is I don't have enough resource ( hardwares ) for testing and verify the design, I only have two PC one is new gen 13 intel which just build for dev and another old AM4 amd . Your helps really change all the things like compatibility , stability and best setup for GPU plotting. Few things I really want the answers is: 1.) Performance gain from DDR5 ? my best ram still ddr4 3200. so I have no idea what 's the ratio ddr5 's gain in plotting.

2.) GPU now is much faster than server with 512G ddr3. so 40G nic with such server re-purpose to a NRD will beat best SSD or just not a option ?

3.) what's the best stable SSD setup for high speed GPU plotting? now GPU plotting will have file write error if ssd was too slow. but how to define the speed of SSD as minimum requirement is still vague answer.

4.) rtx 4090 's performance ? I just curious the next gen GPU raw power and how to optimize it.

Except git ticket, we still have a place called discussion here. so please feel free to put anything here https://github.com/smartbitcoin/arc_plot/discussions

— Reply to this email directly, view it on GitHub https://github.com/smartbitcoin/arc_plot/issues/13#issuecomment-1367965697, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVHLK6WG3QBZJSTCRCRUI43WP33U7ANCNFSM6AAAAAATM5A64A . You are receiving this because you were mentioned.Message ID: @.***>

[smartbitcoin]

Let me know if you have any issue setup , configure and run arc_plot. I am always here for help. You are linux veteran, NRD , kernel turning , fiber network won't be challenge for you lol. In case you have rtx 3090 x2 with NVLINK, I wish I can remote debug it as I don't have such setup so rtx 3090 not support multi GPU yet.

[reb0rn21]

pls post results here, I guess you should also try fast nvme in raid 0 to be sure drive is not overloaded

I bet with fast GPU and nvme you will need defo fast ddr5 to make nice gains, I think whole system bottleneck will be there and it need to be monitored as chipset/mem/storage