search

tydeno / Ploto

A Windows PowerShell based Chia Plotting Manager. Lets you automatically spawn and move plots. Sends you notifications on the go.
MIT License
32 stars 5 forks source link

Two Jobs stuck and Get-PlotoOutDrives Returns available -1 #72

Closed chefsas closed 3 years ago

chefsas commented 3 years ago

I am having a couple issues that are resulting in my system only spawning a single job. I've set my system to deploy 3 jobs in parallel. The system is constantly logging that there are 2 aborted jobs and it will not remove them either automatically through the cleanup process or by using the Remote-AbortedPlotoJobs command. Here's the output of that command:

PS C:\Users\Scott> Remove-AbortedPlotoJobs PlotoRemoveAbortedJobs @ 5/26/2021 1:23:49 PM : Found aborted Jobs to be deleted: PlotoRemoveAbortedJobs @ 5/26/2021 1:23:49 PM : Cleaning up... Stop-PlotoJob : Cannot bind argument to parameter 'JobId' because it is null. At C:\Users\Scott.chia\mainnet\plotter\Ploto.psm1:1422 char:34

Stop-PlotoJob : Cannot bind argument to parameter 'JobId' because it is null. At C:\Users\Scott.chia\mainnet\plotter\Ploto.psm1:1422 char:34

PlotoRemoveAbortedJobs @ 5/26/2021 1:23:49 PM : Removed Amount of aborted Jobs: 2

It says it removed 2 jobs, but the same 2 jobs are still in there (this is the same behavior as I see when I run Start-PlotoSpawns and it runs the periodic removal cleanup job. It says it removed 2 aborted jobs constantly but these same 2 jobs are always stuck in there. There are zero logfiles in the potter folder (other than the 1 running job) and there are zero .tmp files on the plotting drive (other than the ones in the active job) and no .tmp files on the destination drive.

Here's what the system says when I run get-plotojobs:

PS C:\Users\Scott> get-plotojobs

JobId : 8e7d5af8-47b0-4dee-bf83-13aabca9e268 Status : 2.2 StartTime : 05/26/2021 10:43:54 TempDrive : D: T2Drive : OutDrive : E: PID : 12600 PlotSizeOnTempDisk : 157.59 GB PlotSizeOnOutDisk : 0.00 GB ArgumentList : -k 32 -b 3390 -r 10 -t D:\ -d E:\ PlotId : 1d5f985c3d443939107bd69e38b71a2f04fd7b9bc917bed789ab455789e5c463 LogPath : C:\Users\Scott.chia\mainnet\plotter\PlotoSpawnerLog_26_5_10_43_8e7d5af8-47b0-4dee-bf83-13aabca9e2 68_Tmp-D_Out-E.txt StatLogPath : C:\Users\Scott.chia\mainnet\plotter\PlotoSpawnerLog_26_5_10_43_8e7d5af8-47b0-4dee-bf83-13aabca9e2 68_Tmp-D_Out-E@Stat.txt CompletionTime : Still in progress CompletionTimeP1 : 9285.991 seconds. CPU (311.680%) Wed May 26 13:18:41 2021 CompletionTimeP2 : CompletionTimeP3 : CompletionTimeP4 : EndDate :

JobId : Status : Aborted StartTime : TempDrive : T2Drive : OutDrive : PID : None PlotSizeOnTempDisk : PlotSizeOnOutDisk : ArgumentList : PlotId : LogPath : C:\Users\Scott.chia\mainnet\plotter\Ploto.psm1 StatLogPath : CompletionTime : None CompletionTimeP1 : @("OutDrive", "TempDrive", "Starting plotting progress into temporary dirs:", "ID", "F1 complete, time","Starting phase 1/4", "Computing table 1","Computing table 2", "Computing table 3","Computing table 4","Computing table 5","Computing table 6","Computing table 7", "Starting phase 2/4", "Time for phase 1","Backpropagating on table 7", "Backpropagating on table 6", "Backpropagating on table 5", "Backpropagating on table 4", "Backpropagating on table 3", "Backpropagating on table 2", "Starting phase 3/4", "Compressing tables 1 and 2", "Compressing tables 2 and 3", "Compressing tables 3 and 4", "Compressing tables 4 and 5", "Compressing tables 5 and 6", "Compressing tables 6 and 7", "Starting phase 4/4", "Writing C2 table", "Time for phase 4", "Renamed final file", "Total time", "Could not copy", "Time for phase 3", "Time for phase 2") CompletionTimeP2 : @("OutDrive", "TempDrive", "Starting plotting progress into temporary dirs:", "ID", "F1 complete, time","Starting phase 1/4", "Computing table 1","Computing table 2", "Computing table 3","Computing table 4","Computing table 5","Computing table 6","Computing table 7", "Starting phase 2/4", "Time for phase 1","Backpropagating on table 7", "Backpropagating on table 6", "Backpropagating on table 5", "Backpropagating on table 4", "Backpropagating on table 3", "Backpropagating on table 2", "Starting phase 3/4", "Compressing tables 1 and 2", "Compressing tables 2 and 3", "Compressing tables 3 and 4", "Compressing tables 4 and 5", "Compressing tables 5 and 6", "Compressing tables 6 and 7", "Starting phase 4/4", "Writing C2 table", "Time for phase 4", "Renamed final file", "Total time", "Could not copy", "Time for phase 3", "Time for phase 2") CompletionTimeP3 : @("OutDrive", "TempDrive", "Starting plotting progress into temporary dirs:", "ID", "F1 complete, time","Starting phase 1/4", "Computing table 1","Computing table 2", "Computing table 3","Computing table 4","Computing table 5","Computing table 6","Computing table 7", "Starting phase 2/4", "Time for phase 1","Backpropagating on table 7", "Backpropagating on table 6", "Backpropagating on table 5", "Backpropagating on table 4", "Backpropagating on table 3", "Backpropagating on table 2", "Starting phase 3/4", "Compressing tables 1 and 2", "Compressing tables 2 and 3", "Compressing tables 3 and 4", "Compressing tables 4 and 5", "Compressing tables 5 and 6", "Compressing tables 6 and 7", "Starting phase 4/4", "Writing C2 table", "Time for phase 4", "Renamed final file", "Total time", "Could not copy", "Time for phase 3", "Time for phase 2") CompletionTimeP4 : @("OutDrive", "TempDrive", "Starting plotting progress into temporary dirs:", "ID", "F1 complete, time","Starting phase 1/4", "Computing table 1","Computing table 2", "Computing table 3","Computing table 4","Computing table 5","Computing table 6","Computing table 7", "Starting phase 2/4", "Time for phase 1","Backpropagating on table 7", "Backpropagating on table 6", "Backpropagating on table 5", "Backpropagating on table 4", "Backpropagating on table 3", "Backpropagating on table 2", "Starting phase 3/4", "Compressing tables 1 and 2", "Compressing tables 2 and 3", "Compressing tables 3 and 4", "Compressing tables 4 and 5", "Compressing tables 5 and 6", "Compressing tables 6 and 7", "Starting phase 4/4", "Writing C2 table", "Time for phase 4", "Renamed final file", "Total time", "Could not copy", "Time for phase 3", "Time for phase 2") EndDate :

JobId : Status : Aborted StartTime : TempDrive : T2Drive : OutDrive : PID : None PlotSizeOnTempDisk : PlotSizeOnOutDisk : ArgumentList : PlotId : LogPath : C:\Users\Scott.chia\mainnet\plotter\PlotoSpawnerConfig.json StatLogPath : CompletionTime : None CompletionTimeP1 : CompletionTimeP2 : CompletionTimeP3 : CompletionTimeP4 : EndDate :

In addition, when I run the get-plototempdrives and get-plotooutdrives commands, the get-plotooutdrives says -1 in the AvailableAmountToPlot variable. Is this what is causing my system to only spawn a single job? Or does this not matter since it's the final drive?

PS C:\Users\Scott> get-plotooutdrives

cmdlet Get-PlotoOutDrives at command pipeline position 1 Supply values for the following parameters: OutDriveDenom: out

DriveLetter : E: ChiaDriveType : Out VolumeName : OUT FreeSpace : 3623.81 TotalSpace : 0 IsPlottable : True HasPlotInProgress : True AmountOfPlotsInProgress : 1 AmountOfPlotsToTempMax : 0 AvailableAmountToPlot : -1 PlotInProgressID : 1d5f985c3d443939107bd69e38b71a2f04fd7b9bc917bed789ab455789e5c463 PlotInProgressPhase : 2.2

PS C:\Users\Scott> get-plototempdrives temp

DriveLetter : D: ChiaDriveType : Temp Disk : SanDisk Ultra 3D NVMe DiskType : SSD DiskBus : NVMe VolumeName : TEMP FreeSpace : 773.7 TotalSpace : 931.5 hasFolder : False IsPlottable : True HasPlotInProgress : True AmountOfPlotsInProgress : 1 AmountOfPlotsToTempMax : 3 AvailableAmountToPlot : 2 PlotInProgressID : 1d5f985c3d443939107bd69e38b71a2f04fd7b9bc917bed789ab455789e5c463 PlotInProgressPhase : 2.2

Here's my config file for your reference:

{ "PlotterName": "SASCHIA", "EnableAlerts": "true", "EnablePlotoFyOnStart": "true", "ChiaWindowStyle": "normal",

"DiskConfig": [ { "TempDriveDenom": "temp", "Temp2Denom": "", "OutDriveDenom": "out", "EnableT2": "false" } ],

"JobConfig": [ { "IntervallToCheckInMinutes": "5", "InputAmountToSpawn": "100", "WaitTimeBetweenPlotOnSeparateDisks": "15", "WaitTimeBetweenPlotOnSameDisk": "60", "MaxParallelJobsOnAllDisks": "3", "MaxParallelJobsOnSameDisk": "3", "MaxParallelJobsInPhase1OnAllDisks" : "3",
"StartEarly": "true", "StartEarlyPhase": "4",
"BufferSize": "3390", "Thread": "10", "Bitfield": "true", "FarmerKey": "", "PoolKey": "" } ],

"SpawnerAlerts": [ { "DiscordWebhookURL": "https://discord.com/api/webhooks/<>", "WhenJobSpawned": "true", "WhenNoOutDrivesAvailable": "true", "WhenJobCouldNotBeSpawned": "true" } ],

"PlotoFyAlerts": [ { "DiscordWebhookURL": "https://discord.com/api/webhooks/<>", "PeriodOfReportInHours": "1", "PathToPloto": "C:/Users/Scott/.chia/mainnet/plotter/Ploto.psm1" } ] }

tydeno commented 3 years ago

What version of Ploto are you currently running? You can see that right at the top when you open Ploto.psm1

chefsas commented 3 years ago

I was running a previous version; however, I noticed there was a new version and I updated it and restarted everything. Now I'm running 1.0.9.4.8. It still will not spawn more than 1 job.

chefsas commented 3 years ago

One thing of note is now when I run the commands for temp and out drives, they seem to be displaying correctly.

PS C:\Users\Scott> Get-PlotoTempDrives temp

DriveLetter : D: ChiaDriveType : Temp Disk : SanDisk Ultra 3D NVMe DiskType : SSD DiskBus : NVMe VolumeName : TEMP FreeSpace : 720.1 TotalSpace : 931.5 hasFolder : False IsPlottable : True HasPlotInProgress : True AmountOfPlotsInProgress : 1 AmountOfPlotsToTempMax : 3 AvailableAmountToPlot : 2 PlotInProgressID : 1dxxxxxxxxxxxxxxxxxxxxxxxxx PlotInProgressPhase : 3.1

PS C:\Users\Scott> Get-PlotoOutDrives out

DriveLetter : E: ChiaDriveType : Out VolumeName : OUT FreeSpace : 3725.19 TotalSpace : 3725.48 IsPlottable : True HasPlotInProgress : True AmountOfPlotsInProgress : 1 AmountOfPlotsToTempMax : 34 AvailableAmountToPlot : 33 PlotInProgressID : 1dxxxxxxxxxxxxxxxxxxxxxxx PlotInProgressPhase : 3.1

tydeno commented 3 years ago

Okay that looks better. If you start PlotoSpawns with -verbose, what does it say? It should display the cap of which param it reaches as defined by config.

Start-PlotoSpawns -verbose
chefsas commented 3 years ago

PS C:\Users\Scott> start-plotospawns -verbose PlotoManager @ 5/26/2021 3:24:01 PM : Ploto Manager started. PlotoManager @ 5/26/2021 3:24:01 PM : Loading config from C:\Users\me.chia\mainnet\config... VERBOSE: Loaded config successfully PlotoManager @ 5/26/2021 3:24:01 PM : InputAmountToSpawn: 100 PlotoManager @ 5/26/2021 3:24:01 PM : Intervall to wait: 5 PlotoManager @ 5/26/2021 3:24:01 PM : PlotoFy is set to startup. Checking for active PlotoFy jobs...

Id Name PSJobTypeName State HasMoreData Location Command

261 PlotoFy BackgroundJob Running True localhost ... PlotoManager @ 5/26/2021 3:24:02 PM : Detected aborted jobs. Removing them... PlotoRemoveAbortedJobs @ 5/26/2021 3:24:02 PM : Found aborted Jobs to be deleted: PlotoRemoveAbortedJobs @ 5/26/2021 3:24:02 PM : Cleaning up... Stop-PlotoJob : Cannot bind argument to parameter 'JobId' because it is null. At C:\Users\Scott.chia\mainnet\plotter\Ploto.psm1:1422 char:34

Stop-PlotoJob : Cannot bind argument to parameter 'JobId' because it is null. At C:\Users\Scott.chia\mainnet\plotter\Ploto.psm1:1422 char:34

PlotoRemoveAbortedJobs @ 5/26/2021 3:24:02 PM : Removed Amount of aborted Jobs: 2 VERBOSE: Loaded Alarmconfig successfully VERBOSE: PlotoSpawner @ 05/26/2021 15:24:02: Invoking PlotoJobs started. VERBOSE: PlotoSpawner @ 05/26/2021 15:24:03: No Jobs spawned as either no TempDrives available or max parallel jobs reached. Max Parallel Jobs: 3Current amount of Jobs: 3

chefsas commented 3 years ago

There's actually only one job running right now.

chefsas commented 3 years ago

OMG I'm an idiot. One second.

chefsas commented 3 years ago

Seriously I'm an idiot. So I was storing a copy of the powershell script and the JSON in the /plotter folder and the script was interpreting those two files as jobs. I moved those out of that folder, restarted, and everything is good.

Sorry!!!