tmpfs much slower than RAM?

[mikekq]

i compile on ubuntu server 20.04,and create a RAM disk use this command: mount -t tmpfs -o size=110g tmpfs /ram the start the plot with: ./chia-plot -t /mnt/nvme0n1/ -2 /ram/ -d /mnt/sda/plots/ -p poolkey -f farmkey -r 28 and it takes about 4000secs to plot,about 1500secs slower than the madmax's demo,the nvme0n1 disk write speed is about 2g/s. then i try this: dd if=/dev/zero of=/ram/test bs=4k count=1000000 and it shows the write speed on /ram is 2g/s，it's much slower than the RAM's speed i already have vm.swappiness=0 in my /etc/sysctl.conf

can someone show me your tmpfs benchmark with dd command?

[DowJhob]

try mount -t tmpfs -o size=110G tmpfs /ram

https://superuser.com/questions/1321488/linux-tmpfs-write-speed-slower-than-ram-speed

[SNGDude]

i think you should use size=100G with capital.

[szilardx]

My issue with tmpfs is that it seems to be slower than an nvme ssd. Testing on two identical servers DL380 with 2x CPU E5-2680 v3 On one server, using an nvme ssd ( Viper M.2 VP4100 ) the total plot time is 1700 sec. On the other, with RAMdisk as temp2 (120GB) + regular SSD as temp1 it is 2300 sec.

Also testing ramdisk speed with dd, 1GB file, the nvme produces higher speeds. (1 G/s vs 2 G/s).

[pyroxide]

I always get terrible results when setting swappiness too low. swappiness=10 with something like a 4GB or 8GB swapfile makes things much smoother.

[savs90]

Sorry for the off topic question ,but i really dont want to create a new issue for this thing. I read that the new minimum system RAM requirement so you can use the RAM as tmp dir is 128 GiB. So my question is this 128 GiB as a around 138 GB or it is a typo and a system with 128 GB will be fine for tmp2 ?

[mikekq]

here is my plot log,use tmpfs on /ram,the plot server is duel Xeon(R) CPU E5-2680 v4 with 192g RAM seems table #3 #4 #5#6 is much slower than demo

Number of Threads: 52 Number of Buckets: 2^7 (128) Pool Public Key: xxxxx Farmer Public Key:xxxxxx Working Directory: /mnt/nvme1n1/ Working Directory 2: /ram/ Plot Name: plot-k32-2021-06-13-13-31-e357b4018da4e4d4fce5952e51131f4735e861529af70aa8fa931da03302be19 [P1] Table 1 took 8.60566 sec [P1] Table 2 took 88.6923 sec, found 4294890201 matches [P1] Table 3 took 334.527 sec, found 4294817270 matches [P1] Table 4 took 600.479 sec, found 4294673151 matches [P1] Table 5 took 502.349 sec, found 4294494700 matches [P1] Table 6 took 487.178 sec, found 4294019299 matches [P1] Table 7 took 235.951 sec, found 4293046889 matches Phase 1 took 2257.82 sec [P2] max_table_size = 4294967296 [P2] Table 7 scan took 15.8456 sec [P2] Table 7 rewrite took 56.0565 sec, dropped 0 entries (0 %) [P2] Table 6 scan took 111.94 sec [P2] Table 6 rewrite took 180.692 sec, dropped 581405100 entries (13.5399 %) [P2] Table 5 scan took 113.549 sec [P2] Table 5 rewrite took 174.041 sec, dropped 762095879 entries (17.7459 %) [P2] Table 4 scan took 108.25 sec [P2] Table 4 rewrite took 143.78 sec, dropped 828906597 entries (19.3008 %) [P2] Table 3 scan took 101.873 sec [P2] Table 3 rewrite took 117.437 sec, dropped 855114051 entries (19.9104 %) [P2] Table 2 scan took 93.0336 sec [P2] Table 2 rewrite took 140.713 sec, dropped 865626414 entries (20.1548 %) Phase 2 took 1375.1 sec Wrote plot header with 268 bytes [P3-1] Table 2 took 70.6506 sec, wrote 3429263787 right entries [P3-2] Table 2 took 61.6744 sec, wrote 3429263787 left entries, 3429263787 final [P3-1] Table 3 took 72.8122 sec, wrote 3439703219 right entries [P3-2] Table 3 took 67.4861 sec, wrote 3439703219 left entries, 3439703219 final [P3-1] Table 4 took 78.8765 sec, wrote 3465766554 right entries [P3-2] Table 4 took 58.5594 sec, wrote 3465766554 left entries, 3465766554 final [P3-1] Table 5 took 83.9041 sec, wrote 3532398821 right entries [P3-2] Table 5 took 66.4886 sec, wrote 3532398821 left entries, 3532398821 final [P3-1] Table 6 took 87.0631 sec, wrote 3712614199 right entries [P3-2] Table 6 took 64.0534 sec, wrote 3712614199 left entries, 3712614199 final [P3-1] Table 7 took 70.7516 sec, wrote 4293046889 right entries [P3-2] Table 7 took 70.3517 sec, wrote 4293046889 left entries, 4293046889 final Phase 3 took 858.313 sec, wrote 21872793469 entries to final plot [P4] Starting to write C1 and C3 tables [P4] Finished writing C1 and C3 tables [P4] Writing C2 table [P4] Finished writing C2 table Phase 4 took 73.478 sec, final plot size is 108809283840 bytes Total plot creation time was 4564.78 sec

[ghost]

here is my plot log,use tmpfs on /ram,the plot server is duel Xeon(R) CPU E5-2680 v4 with 192g RAM seems table #3 #4 #5#6 is much slower than demo

Number of Threads: 52 Number of Buckets: 2^7 (128)

Each of your CPU's only have 14 physical cores, so advise specifying 28 instead of 52 threads.

Using 256 buckets should also improve your speed with less RAM usage.

[DowJhob]

here is my plot log,use tmpfs on /ram,the plot server is duel Xeon(R) CPU E5-2680 v4 with 192g RAM seems table #3 #4 #5#6 is much slower than demo Number of Threads: 52 Number of Buckets: 2^7 (128)

Each of your CPU's only have 14 physical cores, so advise specifying 28 instead of 52 threads.

Using 256 buckets should also improve your speed with less RAM usage.

no, he use dual cpu then can use 54thread of 56 possibles

[DowJhob]

here is my plot log,use tmpfs on /ram,the plot server is duel Xeon(R) CPU E5-2680 v4 with 192g RAM seems table #3 #4 #5#6 is much slower than demo

Number of Threads: 52 Number of Buckets: 2^7 (128) Pool Public Key: xxxxx Farmer Public Key:xxxxxx Working Directory: /mnt/nvme1n1/ ... ... [P4] Writing C2 table [P4] Finished writing C2 table Phase 4 took 73.478 sec, final plot size is 108809283840 bytes Total plot creation time was 4564.78 sec

nvme1n1 - what your partition?

[mikekq]

solved,the problem is ram,on dual cpu system,the rams should be 8 dimms or 4 dimms,it's 32gX6 before,when add 2 dimms to 32gX8,the plot looks fine now,here is th log now:

Process ID: 51511 Number of Threads: 56 Number of Buckets: 2^8 (256) Pool Public Key: xxxxxxxxxxxxxxxxx Farmer Public Key: xxxxxxxxxxxxxxxxx Working Directory: /mnt/nvme0n1/ Working Directory 2: /ram/ Plot Name: plot-k32-2021-06-15-23-08-4280f3b513d82df150895fa4598327d46f05e35094a94edaa1e17c2cdfdb51a4 [P1] Table 1 took 9.42594 sec [P1] Table 2 took 80.2867 sec, found 4295069049 matches [P1] Table 3 took 107.802 sec, found 4295014365 matches [P1] Table 4 took 146.839 sec, found 4294986546 matches Copy to /mnt/sdb/new/plot-k32-2021-06-15-22-39-c573c7ba53775a0d38bbfb04516f1a66d0f82bede5c358cdf08f596da0a92ec9.plot finished, took 414.398 sec, 250.462 MB/s avg. [P1] Table 5 took 153.079 sec, found 4295002431 matches [P1] Table 6 took 143.937 sec, found 4295080843 matches [P1] Table 7 took 111.823 sec, found 4295089830 matches Phase 1 took 753.222 sec [P2] max_table_size = 4295089830 [P2] Table 7 scan took 5.53357 sec [P2] Table 7 rewrite took 41.2224 sec, dropped 0 entries (0 %) [P2] Table 6 scan took 19.4048 sec [P2] Table 6 rewrite took 39.7051 sec, dropped 581351402 entries (13.5353 %) [P2] Table 5 scan took 29.7392 sec [P2] Table 5 rewrite took 40.9329 sec, dropped 761959330 entries (17.7406 %) [P2] Table 4 scan took 17.6642 sec [P2] Table 4 rewrite took 31.5403 sec, dropped 828840432 entries (19.2979 %) [P2] Table 3 scan took 20.9677 sec [P2] Table 3 rewrite took 37.5794 sec, dropped 855103585 entries (19.9092 %) [P2] Table 2 scan took 23.8539 sec [P2] Table 2 rewrite took 37.0341 sec, dropped 865692179 entries (20.1555 %) Phase 2 took 362.266 sec Wrote plot header with 268 bytes [P3-1] Table 2 took 58.4657 sec, wrote 3429376870 right entries [P3-2] Table 2 took 21.7388 sec, wrote 3429376870 left entries, 3429376870 final [P3-1] Table 3 took 57.9807 sec, wrote 3439910780 right entries [P3-2] Table 3 took 21.7685 sec, wrote 3439910780 left entries, 3439910780 final [P3-1] Table 4 took 57.2127 sec, wrote 3466146114 right entries [P3-2] Table 4 took 23.0118 sec, wrote 3466146114 left entries, 3466146114 final [P3-1] Table 5 took 60.3758 sec, wrote 3533043101 right entries [P3-2] Table 5 took 22.515 sec, wrote 3533043101 left entries, 3533043101 final [P3-1] Table 6 took 57.5845 sec, wrote 3713729441 right entries [P3-2] Table 6 took 21.8817 sec, wrote 3713729441 left entries, 3713729441 final [P3-1] Table 7 took 75.0388 sec, wrote 4295089830 right entries [P3-2] Table 7 took 27.0445 sec, wrote 4294967296 left entries, 4294967296 final Phase 3 took 509.569 sec, wrote 21877173602 entries to final plot [P4] Starting to write C1 and C3 tables [P4] Finished writing C1 and C3 tables [P4] Writing C2 table [P4] Finished writing C2 table Phase 4 took 72.5749 sec, final plot size is 108835604769 bytes Total plot creation time was 1697.71 sec

[DowJhob]

my 2x2690v4 with 9xDDR4-16GB run at 1300 - 1600sec (relate at ssd)

non balanced mem setup only disable interleave little less speed - not dramatical