Chia-Network / bladebit

A high-performance k32-only, Chia (XCH) plotter supporting in-RAM and disk-based plotting
Apache License 2.0
339 stars 109 forks source link

bladebit cudaplot Completed writing plot in 50+ seconds its too long #405

Open zlotr opened 10 months ago

zlotr commented 10 months ago

hi, when bladebit cudaplot the plot writing time is too long. is there a solution for this? I'm plotting with 2x nvidia 3080. 512 ram 48 cpu and 2x 2TB samsung 980 pro nwme raid 0 xfs file system ubuntu 22.04. I'm running 2 bladebits at the same time. The times are almost the same. bladebit version alpha 4.3 (When I try other versions, Completed Plot time is too long)

Generating plot 48 / 999999: 552902f432b15e12882744c0a931367c804b2fa172e7958434a7a3392f4f6f67 Plot temporary file: /home/mnt/pw/plot-k32-c01-2023-09-10-11-26-552902f432b15e12882744c0a931367c804b2fa172e7958434a7a3392f4f6f67.plot.tmp

Generating F1 Finished F1 in 2.75 seconds. Table 2 completed in 6.19 seconds with 4294967296 entries. Table 3 completed in 13.09 seconds with 4294859086 entries. Table 4 completed in 12.91 seconds with 4294669810 entries. Table 5 completed in 12.21 seconds with 4294338834 entries. Table 6 completed in 9.92 seconds with 4293752916 entries. Table 7 completed in 6.95 seconds with 4292384371 entries. Finalizing Table 7 Finalized Table 7 in 3.98 seconds. Completed Phase 1 in 68.00 seconds Marked Table 6 in 2.16 seconds. Marked Table 5 in 2.16 seconds. Marked Table 4 in 2.16 seconds. Marked Table 3 in 2.17 seconds. Completed Phase 2 in 8.66 seconds Compressing Table 2 and 3... Step 1 completed step in 4.72 seconds. Step 2 completed step in 4.36 seconds. Completed table 2 in 9.08 seconds with 3439658323 / 4294859086 entries ( 80.09% `), Compressing tables 3 and 4... Step 1 completed step in 4.42 seconds. Step 2 completed step in 6.05 seconds. Step 3 completed step in 4.34 seconds. Completed table 3 in 14.82 seconds with 3465687623 / 4294669810 entries ( 80.70% ). Compressing tables 4 and 5... Step 1 completed step in 4.42 seconds. Step 2 completed step in 6.14 seconds. Step 3 completed step in 4.37 seconds. Completed table 4 in 14.94 seconds with 3532205597 / 4294338834 entries ( 82.25% ). Compressing tables 5 and 6... Step 1 completed step in 4.42 seconds. Step 2 completed step in 6.35 seconds. Step 3 completed step in 4.45 seconds. Completed table 5 in 15.22 seconds with 3712345399 / 4293752916 entries ( 86.46% ). Compressing tables 6 and 7... Step 1 completed step in 4.38 seconds. Step 2 completed step in 7.03 seconds. Step 3 completed step in 4.73 seconds. Completed table 6 in 16.14 seconds with 4292384371 / 4292384371 entries ( 100.00% ). Serializing P7 entries [PlotWriter] Command buffer full. Waiting for commands. [PlotWriter] Waited 0.000000 seconds for a Command to be available. [PlotWriter] Command buffer full. Waiting for commands. [PlotWriter] Waited 1.880000 seconds for a Command to be available. Completed serializing P7 entries in 4.77 seconds. Completed Phase 3 in 74.97 seconds Completed Plot 1 in 151.63 seconds ( 2.53 minutes )

/home/mnt/pw/plot-k32-c01-2023-09-10-11-26-552902f432b15e12882744c0a931367c804b2fa172e7958434a7a3392f4f6f67.plot.tmp -> /home/mnt/pw/plot-k3 2-c01-2023-09-10-11 -26-552902f432b15e12882744c0a931367c804b2fa172e7958434a7a3392f4f6f67.plot Completed writing plot in 54.45 seconds

bdelgado1995 commented 10 months ago

Those 'Command buffer full' messages apparently reflect that the I/O sub-system is not keeping up and plotting is stalling. But, I am also getting these messages on several NVME drives (some PCI-E 3.0 and some PCI-E 4.0) which doesn't sound right since the drives are good performance.

zlotr commented 10 months ago

I get different results when using different bladebit versions on the same computer. The most stable one right now is alpha3.4. In other versions, the times are very high. I am using the latest version of nvidia cuda. Last time I used beta v3.1, the plot writing time was low but the plot creation time was very high.

hajes commented 10 months ago

there is some bug...Harold is working on it. In my case, RAMdisk helped if you have 512GB RAM. Otherwise, you have to wait for official stable release. Harold said "it is fixed in testing version"