Extended MadMax support

[admn-monkey]

As of v1.0.47 PSChiaPlotter - Show-PsChiaPlotter

• RAMDrive not seen in Temp Drive selection.

• Current version of MadMax (Windows - Fury Road) allows for plotting Phase 1 on one drive (tmpdir) and then Phase 2 on another drive (tmpdir2 - RAM drive) then Phase 3 and 4 go back to the (tmp) defined drive. I believe a future release is to be able to define a drive for each individual phase and also a destination.

It doesn't appear as though PSChiaPlotter allows for that configuration. If I'm wrong please let me know what I'm not seeing.

[MrPig91]

Yeah unfortunately PSChiaPlotter doesn't pick up all the possible drives available. That being said you are able to switch to "Basic Plotting" where you can manually enter the folder paths for the tmp directory and tmp 2 directory. I don't believe madmax fury road has the ability to enter in more than 2 tmp directories. The newest feature I see is -G which alternates tmp and tmp 2 to even out the wear on SSDs when the -n value is greater than 1

[Jacek-ghub]

@MrPig91,

My take is that PSCP goal is to:

1. run "on autopilot" in a highly optimized way
2. provide stats
3. enable fine-tuning

Prioritized In that order.

Majority of people ask basically the same questions (how to do this, that, and back to square one). For whatever reason Chia legacy is to specify those rather primitive settings that the program should figure out by itself (e.g., as an improvement MM has "-1" as "run forever" option for number of plots - what should potentially be a default). Why do I need to think how many threads to assign to an instance, when plotter providers are not sharing threads usage / utilization. The same for memory.

With that in mind, at the moment, PSCP only recognizes basic volumes; however, a lot of people are asking for UNC, RAID, RAM drive names. Using that "Basic Plotting" is just a cop-out that really hurts PSCP (IMO). It should be fairly straightforward to even add a 'USER SPECIFIED' drive option to "Advanced Plotting," where for time being end user could provide those other paths (and PSCP would only do a basic check whether that path exists).

Again, don't take me wrong. In my opinion, PSCP is THE BEST plotting manager. With the addition of MM, hopefully it will gain more traction now. It would be nice to have some PR right now.

@admn-monkey, You can still get it done in Advanced section, as long as you map that RAM drive to another letter. Check some other threads that talk either about RAID or UNC names. RAM disk is basically the same problem.

Thank you, Jacek

[admn-monkey]

@MrPig91 ,

Ah, yes, thank you. Switched to Basic and put the drive letters in; running a test cycle now which appears to be working beautifully. I know using the PSChiaPlotter doesn't really 'do' much over command line with MadMax FR but I wouldn't be running Windows if I didn't like having a GUI. :) To clarify, I'm serial plotting; I'm no longer parallel plotting. Through several weeks of test cycles I can get the fastest plot cycle using MadMax for a single plot with a raid array and ramdrive, then repeat, repeat, etc. Faster than parallel plotting using dual NVMe and dual SSD including testing with those in and out of RAID configurations. The only way I'd be able to increase plotting speed at this point is move to server hardware to obtain more threads and more ram than a workstation class system can provide. That's just not worth the cost.

@Jacek-ghub,

My apologies, but I don't understand what you mean by mapping the RAM drive to another letter. I have it mapped to a drive letter using IMDisk that's seen by Windows, command prompt, and by powershell, but under PSCP Advanced mode only my disk based volumes are shown (RAID, NVMe, SSD, USB). I assumed you might have meant use the subst command but that had no effect.

[Jacek-ghub]

@admn-monkey,

Sorry for that.

You can issue the following command (from the command prompt):

net use x: \\localhost\f$\folder

where f$ is your RAM drive (assuming you have it at f:), and folder is optional (I think). The x: is whatever letter you want to assign it to. This way, you do the mickey mouse manouvers to map your RAM drive UNC name to a new letter, and bypass PSCP limitations (instead of PSCP doing it for you). Once done, you should be good on Advanced tab.

Also, my take is that your statement about moving to a server setup may not be that beneficial. Yes, you will have many more cores, but that implies, you will have your RAM, RAID, etc. as obvious bottlenecks. Also, your standard CPU runs at $300-400, where you will need to pay $1,000 to $2,000 to maybe triple the speed. The only issue with consumer grade CPU is that Z490/590 chipsets may max out at 128GB of RAM (so, you cannot plot it completely in RAM). But of course, if you have to manage a bunch of those plotters, then going the server route minimizes maintenance issues.

By the way, what are you using for your RAID?

[admn-monkey]

@Jacek-ghub ,

No worries, thanks for the additional info; much appreciated!

I'm using HP P822 controllers with HP D2600 LFF 12 bay chassies now. The internal Z490 motherboard Intel RAID0 came out about even with not using RAID with the separate drives running parallel plots (2 each M.2 NVMe and 2 each WD Red SSD drives). With the D2600 12 drive RAID0 dual channel (tmpdrive) into the P822, 110Gb RAM drive (tmpdrive2), along with another D2600 for a RAID6 destination drive it's humming along with MadMax in 43 minutes per plot, copy time included, compared to around 1.7 hours in any other onboard configuration I tried. Granted I do realize that's an apples to oranges comparison. Just trying to squeeze every drop of speed from what I have.

[Jacek-ghub]

@MrPig91,

Additional issues with MM support (v.1.0.48).

1. MM allows "-1" to be specified as the number of plots. It implies "run-forever." Would be really nice to either use the same convention, or let the user select "Run Forever" that would apply to both MM and chia.exe
2. No provision to provide "-G" (alternate temp1 and temp2 for consecutive runs). Maybe in addition to the drive/path for that alternative plotter, it would be good to add another edit field where people could just write in plotter specific settings (that are [not yet] covered by PSCP)
3. Copy phase. MM has a much better option to xfr the plot to the final drive. It runs the moving process in parallel to the next run. Therefore, when starting just one plotter instance at a time, the move time is added to the processing time, where when starting an instance with -n >1, the moving thread potentially slightly increases the next plot timing, but that increase may be negligible (as most of that time is waiting for a slow IO). Actually, this is a bit problematic,, as depending how the final/final move is done it could be preferred to run the plotter either as a single instance, or as a continuous one.

Also, I think those are not MM specific:

1. Is there a way to add a path to 2nd Temp Drive? Without that all those temp files end up in the root folder
2. Aborting PSCP while a job is running is not cleaning up temp files, where aborting a job does.

[MrPig91]

1. I could add a check box for "Plot Infinite", but I feel this feature is already indirectly available as you can input a large number in the total plots field. I might add this as it would be a somewhat easy thing to add.

2. I would not be able to use -G directly but could add this feature in PSChiaPlotter to handle alternating temp drives (for why i can't use -G on MM see next point.

3. I am unable to run MM or chia.exe with -n greater than 1 due to how the log files are being created. The log files are created by redirecting the output of these processes to a log file instead of the console. If n is greater than one then all plotting logs would go into one log file instead of having 1 log file for each plotting process. These log files are parsed by other functions including Get-ChiaPlotProgress and Get-ChiaPlottingStatistics and these depend on those log files being a single plotting process. I am not saying it would be impossible but it would be incredibly difficult for me to implement.

4. You can change the path while the drop down of the 2nd temp drive combobox is down (I know this is poorly implemented at the moment).

5. Yeah, this is because the application closes before it reaches the lines to delete those tmp files. I can add something to the closing function to delete those tmp files though which I will try to add in the next update.

[Jacek-ghub]

1 Plot Infinite I think that you are arguing an engineering point, not really end-user simplicity. Yes, one can enter 11111111 as the number of plots, but there is a reason why MM is using -1 for that.

3. -n greater than 1 Let me go over once more about the importance of -n greater than 1.

Let's assume that we run the standard chia.exe. After it does all the work, it forks out the final move process. As one job takes four to ten hours (a single job, or parallel jobs), we tend to ignore that copy time, as it is only a small fraction of the work process. Also, depending on a setup, for most people that copy process takes between 10 and 20 minutes. However, when we consider that the box is producing 20-30 plots per day, that copy process amounts to one/two lost plots.

The problem is more visible with MM, as just one instance is taking over all computing resources, thus finishing a job in 30 minutes to a couple of hours. When we start pushing our boxes to get MM finish below that 1 hour mark, that 10-20 minutes copy process starts to be really visible. Due to that fact, MM starts the final copy phase, after it starts the next process (if -n greater than 1). More or less, that copy process is not interfering too much with the computing one, as it is mostly waiting on disks. With a potential addition of a third tmp folder used for the final assembly of the plot, they will further eliminate potential read/write contention in that folder.

I do understand what you wrote, and was actually thinking about it as well, seeing that PSCP was kicking off one MM at a time. Maybe one way to approach this issue would be to add a "log-dup" process, that would monitor the combined log, and producing individual logs, just for PSCP. This way, all the other tasks that are there right now will not need to be changed.

4. Maybe you could make that little edit field a bit wider? At this moment, the connotation for that is more like a special selection, where with a wider box, it may better imply that it is an edit field.

[edwanh]

How could i set different buckets for phase3+4?

[MrPig91]

Plot infinite has now been added. I don't plan to add drive swapping for now. Phase 3/4 buckets has also been added.