parallel polling perhaps with Start-ThreadJob

[timriker]

We should be able to poll basic loadbalancer details and then thread the data collection on all the nodes to decrease build times when we have multiple pairs to poll

[epacke]

I looked into this before but never got it to work. It'd be nice indeed!

[timriker]

I've been doing some experimenting. It looks like the thread support does not include any data sharing. Stdin, to the thread, and stdout from the thread. The loop would have to serialize all the input data, and then the thread would have to serialize the entire set of cached data for that node as well as any logged errors, etc. Should work, but that's a lot of overhead. It also makes the debugger less than useful as the debugger, at least in vscode, won't walk though Start-Job or Start-ThreadJob code. There would have to be an easy way to turn the pooling back into a Foreach so that we can still debug.

[epacke]

I have had an ambition to rewrite the backend script in node. That would make it possible. But it takes quite a lot of time. :)

[timriker]

That would be sweet. All the rest json parsing might be easier in node. :)

[timriker]

I did some experimentation with Powershell. The ScriptBlock acts as a separate script. No functions, variables, etc from the main script get passed on. The entire polling code would need to be inside the scriptblock and would not be debuggable. Ugh. https://docs.microsoft.com/en-us/powershell/module/threadjob/start-threadjob Here's a sample script that passes input into threads, gets logs entries and then final data back one line at a time.

#! /usr/bin/pwsh
#Requires -Version 5

# ThreadJob sample with json feedback from threads
# by Tim Riker <Tim@Rikers.org>

# requires ThreadJob which is included in 6.x, and available for 5.x
# to install, as adminstrator, run: Install-Module -Name ThreadJob
# to verify run: Get-Command Start-ThreadJob | select name

try {
    $oldPreference = $ErrorActionPreference
    $ErrorActionPreference = "stop"
    if(Get-Command Start-ThreadJob){
        #
    }
} catch {
    "Start-ThreadJob is required"
    "As Administrator, try: Install-Module -Name ThreadJob"
    exit
} Finally {
    $ErrorActionPreference=$oldPreference
}

$logs=@()
$results=@{}
$jobs=@()
Measure-Command {
    ForEach($num in 0..2) {
        $params=@{num=$num;name="Job-$num"}
        $jobs += Start-ThreadJob -ThrottleLimit 2 -Name $params.name -InputObject $params -ScriptBlock {
            function internallog {
                Param ([string]$LogType, [string]$Message)
                $CurrentTime = $(Get-Date -UFormat "%Y-%m-%d %H:%M:%S")
                $log=@{datetime=$CurrentTime; type=$LogType; message=$Message}
                $log | ConvertTo-Json -Compress
            }
            $arr = @($input)
            $output=@{output=1;input=$arr[0]}
            Foreach($round in 0..3){
                internallog "normal" ("line1 job="+$arr.num+"`r`nline2 round=$round")
                Start-Sleep 1
            }
            $output.output=($arr.num * 10)
            $output | ConvertTo-Json -Compress
        }
    }
    # loop getting information from jobs
    do {
        $remaining=0
        foreach($job in $jobs){
            if ($job.State -ne "Completed") {
                $remaining++
            }
            if ($job.HasMoreData) {
                $lines=Receive-Job -Job $job
                Foreach($line in $lines) {
                    $obj=ConvertFrom-Json $line
                    # process contents of $obj, if log, add to global log and echo to screen, else store results.
                    if($obj.datetime) {
                        $logs += $obj
                    } elseif ($obj.input.name) {
                        $results.add($obj.input.name,$obj)
                    }
                    Write-Host "Temp:$line"
                }
            }
        }
        Write-Host -NoNewLine "Remaining: $remaining `r"
        Start-Sleep 1
    } until ($remaining -eq 0)

    # wait for jobs to shut down
    $jobs | Wait-Job
    $jobs | Remove-Job
} | Select-Object TotalSeconds
$logs | ConvertTo-Json
$results | ConvertTo-Json

notice internallog() is part of the script block.