search

proxb / PoshRSJob

Provides an alternative to PSjobs with greater performance and less overhead to run commands in the background, freeing up the console and allowing throttling on the jobs.
MIT License
541 stars 87 forks source link

How to log job output to a file after the jobs are completed #173

Open ghost opened 6 years ago

ghost commented 6 years ago

Tried doing this Get-RSJob | Receive-RSJob | Format-Table -AutoSize | Out-File $outfile

Got below error The WriteObject and WriteError methods cannot be called from outside the overrides of the BeginProcessing, ProcessRecord, and EndProcessing methods, and they can only be called from within the same thread. Validate that the cmdlet makes these calls correctly, or contact Microsoft Customer Support Services.

I am calling the script containing Start-RSJob from another script using Invoke-Expression { script file + params }

MVKozlov commented 6 years ago

any info about your environment ? I can't reproduce this

ghost commented 6 years ago

Please run the script below, This needs a SQL Server instance. It is intermittent though (may have to run multiple times). I am using Windows10

SCRIPT

$serverName = "localhost" $databaseName = “master"

1..31| Start-RSJob -Throttle 31 -Name {$_} -ScriptBlock {

                            Invoke-Sqlcmd -ServerInstance

$Using:serverName -Database $Using:databaseName -Query "select getdate()" -OutputSqlErrors $true -QueryTimeout 65536 | out-null

                            [pscustomobject]@{
                                FileToProcess = $_
                                Server=$Using:serverName
                                Database=$Using:databaseName
                                Time = $(Get-Date)
                            }

                    } | Wait-RSJob -ShowProgress | Receive-RSJob

-WarningVariable wv -ErrorVariable er; $wv; $er

OUTPUT

The WriteObject and WriteError methods cannot be called from outside the overrides of the BeginProcessing, ProcessRecord, and EndProcessing methods, and they can only be called from within the same thread. Validate that the cmdlet makes these calls correctly, or contact Microsoft Customer Support Services. The WriteObject and WriteError methods cannot be called from outside the overrides of the BeginProcessing, ProcessRecord, and EndProcessing methods, and they can only be called from within the same thread. Validate that the cmdlet makes these calls correctly, or contact Microsoft Customer Support Services. FileToProcess Server Database Time

        4 localhost master   1/10/2018 10:46:43 PM
       10 localhost master   1/10/2018 10:46:44 PM
       11 localhost master   1/10/2018 10:46:43 PM
       13 localhost master   1/10/2018 10:46:43 PM
       15 localhost master   1/10/2018 10:46:43 PM
       16 localhost master   1/10/2018 10:46:44 PM
       18 localhost master   1/10/2018 10:46:44 PM
       20 localhost master   1/10/2018 10:46:44 PM
       22 localhost master   1/10/2018 10:46:44 PM
       23 localhost master   1/10/2018 10:46:44 PM
       25 localhost master   1/10/2018 10:46:44 PM
       26 localhost master   1/10/2018 10:46:45 PM
       28 localhost master   1/10/2018 10:46:45 PM
       29 localhost master   1/10/2018 10:46:45 PM
       30 localhost master   1/10/2018 10:46:45 PM
       31 localhost master   1/10/2018 10:46:45 PM
       24 localhost master   1/10/2018 10:46:45 PM
       21 localhost master   1/10/2018 10:46:46 PM
       27 localhost master   1/10/2018 10:46:46 PM
       19 localhost master   1/10/2018 10:46:46 PM
       17 localhost master   1/10/2018 10:46:46 PM
       14 localhost master   1/10/2018 10:46:46 PM

The pipeline has been stopped. The WriteObject and WriteError methods cannot be called from outside the overrides of the BeginProcessing, ProcessRecord, and EndProcessing methods, and they can only be called from within the same thread. Validate that the cmdlet makes these calls correctly, or contact Microsoft Customer Support Services. 9 localhost master 1/10/2018 10:46:47 PM The pipeline has been stopped. 12 localhost master 1/10/2018 10:46:47 PM 8 localhost master 1/10/2018 10:46:47 PM 7 localhost master 1/10/2018 10:46:47 PM 6 localhost master 1/10/2018 10:46:47 PM 5 localhost master 1/10/2018 10:46:47 PM 2 localhost master 1/10/2018 10:46:47 PM 1 localhost master 1/10/2018 10:46:48 PM 3 localhost master 1/10/2018 10:46:48 PM

Also how do I track exceptions for jobs for which HasErrors == true? How to re-direct job errors to a file?

Id Name State HasMoreData HasErrors Command

2654 bcp_Orders_Partit... Completed False False ...

2656 bcp_Orders_Partit... Completed False True ... 2662 bcp_Orders_Partit... Completed False True ... 2665 bcp_Orders_Partit... Completed False True ... 2666 bcp_Orders_Partit... Completed False True ... 2668 bcp_Orders_Partit... Completed False True ... 2669 bcp_Orders_Partit... Completed False True ... 2671 bcp_Orders_Partit... Completed False True ... 2673 bcp_Orders_Partit... Completed False True ... 2675 bcp_Orders_Partit... Completed False True ... 2677 bcp_Orders_Partit... Completed False True ... 2678 bcp_Orders_Partit... Completed False False …

On January 10, 2018 at 2:30:08 PM, Max Kozlov (notifications@github.com) wrote:

any info about your environment ? I can't reproduce this

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/proxb/PoshRSJob/issues/173#issuecomment-356710712, or mute the thread https://github.com/notifications/unsubscribe-auth/AP6v6dpjXYEuJkCECfVjfOkBDlWQSlBxks5tJQ--gaJpZM4RZZvp .

MVKozlov commented 6 years ago

The error itself not in RSJob module Seems there is something in Invoke-SqlCmd use direct output to $host, so it is not thread safe this code variant show that error catched inside Invoke-SQLCmd and you can use this method to catch errors :)

$serverName = "localhost"
$databaseName = "master"

1..31 | Start-RSJob -Throttle 31 -Name { $_ } -ScriptBlock {

    $params = @{
        ServerInstance = $Using:serverName
        Database = $Using:databaseName
        Query = "select getdate()"
        OutputSqlErrors = $true
        QueryTimeout = 65536
    }
    $err = $null
    try {
        Invoke-SqlCmd @params | Out-Null
    }
    catch {
        $err = $_
    }

    [pscustomobject]@{
        FileToProcess = $_
        Server=$Using:serverName
        Database=$Using:databaseName
        Time = $(Get-Date)
        Error = $err
    }

} | Wait-RSJob -ShowProgress | Receive-RSJob -WarningVariable wv -ErrorVariable er
$wv
$er
ghost commented 6 years ago

Thank you. Something funky with Invoke-Sqlcmd

On January 11, 2018 at 1:57:12 AM, Max Kozlov (notifications@github.com) wrote:

The error itself not in RSJob module Seems there is something in Invoke-SqlCmd use direct output to $host, so it is not thred safe this code variant show that error catched inside Invoke-SQLCmd

$serverName = "localhost"$databaseName = "master" 1..31 | Start-RSJob -Throttle 31 -Name { $_ } -ScriptBlock {

    $params = @{
            ServerInstance = $Using:serverName
            Database = $Using:databaseName
            Query = "select getdate()"
            OutputSqlErrors = $true
            QueryTimeout = 65536
    }
    $err = $null
    try {
            Invoke-SqlCmd @params | Out-Null
    }
    catch {
            $err = $_
    }

[pscustomobject]@{
            FileToProcess = $_
            Server=$Using:serverName
            Database=$Using:databaseName
            Time = $(Get-Date)
            Error = $err
}

} | Wait-RSJob -ShowProgress | Receive-RSJob -WarningVariable wv -ErrorVariable er$wv$er

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/proxb/PoshRSJob/issues/173#issuecomment-356843597, or mute the thread https://github.com/notifications/unsubscribe-auth/AP6v6UcojtqmQ5Y9QCAZRoKhbg6BCwmNks5tJbDHgaJpZM4RZZvp .