A PowerShell module for running concurrent jobs based on threads rather than processes
PowerShell's built-in BackgroundJob jobs (Start-Job) are run in separate processes on the local machine. They provide excellent isolation but are resource heavy. Running hundreds of BackgroundJob jobs can quickly absorb system resources by creating hundreds of processes. There is no throttling mechanism and so all jobs are started immediately and are all run currently.
This module extends the existing PowerShell BackgroundJob to include a new thread based ThreadJob job. It is a lighter weight solution for running concurrent PowerShell scripts that works within the existing PowerShell job infrastructure. So these jobs work with existing PowerShell job cmdlets.
ThreadJob jobs will tend to run much faster because there is lower overhead and they do not use the remoting serialization system as BackgroundJob jobs do. And they will use up fewer system resources. In addition output objects returned from the job will be 'live' since they are not re-hydrated from the serialization system. However, there is less isolation. If one ThreadJob job crashes the process then all ThreadJob jobs running in that process will be terminated.
This module exports a single cmdlet, Start-ThreadJob, which works similarly to the existing Start-Job cmdlet. The main difference is that the jobs which are created run in separate threads within the local process.
Also ThreadJob jobs support a ThrottleLimit parameter to limit the number of running jobs, and thus running threads, at a time. If more jobs are started then they go into a queue and wait until the current number of jobs drops below the throttle limit.
PS C:\> Start-ThreadJob -ScriptBlock { 1..100 | % { sleep 1; "Output $_" } } -ThrottleLimit 2
PS C:\> Start-ThreadJob -ScriptBlock { 1..100 | % { sleep 1; "Output $_" } }
PS C:\> Start-ThreadJob -ScriptBlock { 1..100 | % { sleep 1; "Output $_" } }
PS C:\> get-job
Id Name PSJobTypeName State HasMoreData Location Command
-- ---- ------------- ----- ----------- -------- -------
1 Job1 ThreadJob Running True PowerShell 1..100 | % { sleep 1;...
2 Job2 ThreadJob Running True PowerShell 1..100 | % { sleep 1;...
3 Job3 ThreadJob NotStarted False PowerShell 1..100 | % { sleep 1;...PS C:\> $job = Start-ThreadJob { Get-Process -id $pid }
PS C:\> $myProc = Receive-Job $job
# !!Don't do this. $myProc is a live object!!
PS C:\> $myProc.Kill()# start five background jobs each running 1 second
PS C:\> Measure-Command {1..5 | % {Start-Job {Sleep 1}} | Wait-Job} | Select TotalSeconds
PS C:\> Measure-Command {1..5 | % {Start-ThreadJob {Sleep 1}} | Wait-Job} | Select TotalSeconds
TotalSeconds
------------
5.7665849 # jobs creation time > 4.7 sec; results may vary
1.5735008 # jobs creation time < 0.6 sec (8 times less!)You can install this module from PowerShell Gallery using this command:
Install-Module -Name ThreadJob -Scope CurrentUser