Out-File not reporting error on failure to write to UNC network path when stream is being repeatedly opened and closed.

SimonESHT commented 3 weeks ago

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest released version
[X] Search the existing issues.
[X] Refer to the FAQ.
[X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

I am trying to extract a representative sample of a 200MB csv file by writing the header and every 500th row to a new file for testers to use. I relied on out-file -append to add each row matching the modulus condition to the destination file on network share, but what I found is that I had slightly fewer rows in the sample file than expected. Repeated runs produced slightly different numbers of rows in the destination file (expected 2014, actual ranged between 1992-2011). I did not get an terminating error.

The behaviour does NOT occur if I use a location on my local hard drive for the destination file.

$destfile = "\\UNCSHARE\Folder\Export_Sample_$(get-date -Format "yyyyMMdd_HHmmss").txt"

$Original = get-content "\\UNCSHARE\Folder\200MB_Export_20231208_1545.txt"

[int64]$ln = 0
[int64]$SampleCount = 0

foreach ($line in $Original) {
    $ln++
    if ($ln -eq 1 -or $ln % 500 -eq 0) {
        $line | Out-File -FilePath $destfile -Append -ErrorAction Stop
        $SampleCount++
    }
}
write-host $SampleCount

(get-content $destfile).count

I tried a suggested alternative of switching to the native API call:

#$line | Out-File -FilePath $destfile -Append -ErrorAction Stop -Encoding utf8
[System.IO.File]::AppendAllText($destfile, "$line`n")

That still doesn't produce a terminating error in Powershell 7.4.2, but does in Windows Powershell 5.1

Exception calling "AppendAllText" with "2" argument(s): "The process cannot access the file 
'\\UNCSHARE\Export_sample20240424_164810.txt' because it is being used 
by another process."

Out-File to a UNC path doesn't produce a terminating error in Powershell 7.4.2 or 5.1.

Expected behavior

For my example file. 
write-host $SampleCount
--2014
(get-content $destfile).count 
--2014

Actual behavior

For my example file. 
write-host $SampleCount
--2014
(get-content $destfile).count 
--a number ranging between 1995 and 2011 (depending on the number of times it failed to write a single line to the destination)

Error details

No response

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.2
PSEdition                      Core
GitCommitId                    7.4.2
OS                             Microsoft Windows 10.0.19045
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

rhubarb-geek-nz commented 3 weeks ago

Could you do a single pipeline rather than continually opening and closing?

Also, does setting $ErrorActionPreference at the root make any difference?

$destfile = "\\UNCSHARE\Folder\Export_Sample_$(get-date -Format "yyyyMMdd_HHmmss").txt"

$srcFile = '\\UNCSHARE\Folder\200MB_Export_20231208_1545.txt'

[int64]$ln = 0
[int64]$SampleCount = 0

$ErrorActionPreference = 'Stop'

Get-Content -LiteralPath $srcFile | ForEach-Object {
    $SampleCount++
    $ln++
    if ($ln -eq 1 -or $ln % 500 -eq 0)
    {
        $_
    }
} | Set-Content -LiteralPath $destFile

write-host $SampleCount

(get-content $destfile).count

SimonESHT commented 3 weeks ago

Thanks, Yes, I found that to be an effective workaround. See the Stack Overflow thread [here]https://stackoverflow.com/questions/78378535/powershell-out-file-append-silently-skips-a-few-rows-when-used-inside-foreach-l The point of raising this ticket though was to alert that the Out-File cmdlet is not raising a terminating error when it cannot open the file for write access. The expected behaviour (as seen in PS5.1 with the native API call) is that it should as I illustrated above

Exception calling "AppendAllText" with "2" argument(s): "The process cannot access the file 
'\\UNCSHARE\Export_sample20240424_164810.txt' because it is being used 
by another process."

To be clear, I have found an alternative approach, but the behaviour of Out-File in PS Core, when writing to UNC network stream, does not make it apparent that any error has occurred.

I did try adding $ErrorActionPreference ="STOP" at the root of the script but it still didn't report an error.

Stack Overflow
Powershell out-file -append silently skips a few rows when used inside foreach loop
I am trying to extract a representative sample of a 200MB csv file by writing the header and every 500th row to a new file for testers to use. My first attempt was knowingly sub-optimal but seemed ...

PowerShell / PowerShell