search

JohnnyFFM / Poc1to2Converter

Converts POC1 plots to POC2 plots in Windows
The Unlicense
11 stars 1 forks source link

Parallel conversion coming? #5

Closed peteraclausen closed 6 years ago

peteraclausen commented 6 years ago

I have 108 plot files totaling 418 TB. I'm running the latest version of your converter with great success, but a 2TB plot takes about 11 hours to convert inline. So at this rate, it will take me 95 days to convert all my plots.

Will a parallel version be released that will allow converting a few dozen plots at once?

My server has dual E5-2680 V4 CPUs and 64GB of RAM, and all 108 drives are directly connected via SAS2 expanders via 4 x 4 x 6Gbps channel HBAs, so I have the I/O to convert 20-30 plots at once.

This converter and forked Blago miner are otherwise working great. Thanks for all the work putting this together!

JohnnyFFM commented 6 years ago

Hi! You can already process in parallel. Just open another instance of the application (basically one per drive). You can easily convert several drives at the same time.

On SATA/SAS I've converted 10 disks at the same time, no issues. Same speed as a single drive, almost no CPU load, just I/O. On USB, I would not recommend to convert more than two drives per controller in parallel.

Just one things you have to consider: Each instance of the application needs RAM. If you have for example 8GB and want to convert 10 drives at the same time, limit the memory usage of each app to something like 500-700MB (leave a little for overhead / OS).

Best,

Johnny

peteraclausen commented 6 years ago

Ah perfect. I fired up another 15 instances, allocating 1,024 MB to each, and they are all chugging along now at the same speed as when doing just one. I still have 40GB of free RAM, but I'm going to let these finish up before adding more. Will try 32 instances on the next round.

Thanks!

Crazydane