Process.WaitForExit() hangs forever on Linux when .NET Core process is run as a child of a daemon

[riku76]

On Linux I have a native daemon that is running my .NET Core 3.1.1 application as a child and this child is spawning another native child (eg. df or grep) the WaitForExit never returns even if the StandardOutput/Error are NOT redirected. From htop I can see that the child exits, shows quickly as a Zombie and then goes away, so it looks like .NET Core is closing the handle, but somehow ignoring that it exited. Also, the same code works fine if executed on command-line OR under Mono 6.4.0 (even under daemon). Platform is Raspberry Pi 4 (armv7).

var psi = new System.Diagnostics.ProcessStartInfo("/bin/df", "-h"); psi.UseShellExecute = false; psi.CreateNoWindow = true;

var p = new System.Diagnostics.Process(); p.StartInfo = psi;

p.WaitForExit(); //Hangs if running under a daemon process

As a workaround it is possible to redirect the StandardOutput, use ReadToEnd on it to wait and then dispose the process object. However, the application exit code is then lost.

[tmds]

Does the issue happen also when the .NET application itself is the daemon? Are you using systemd? or another init system? What should I do to reproduce this?

[riku76]

Haven't tried with the .NET app itself as the daemon. Running under systemd. In my scenario a native daemon app is starting the .NET Core 3.1.1 app, which is then using the System.Diagnostics.Process to spawn another native app, with the code above.

[riku76]

I tested that it does not repro if .NET Core app is directly ran as daemon, you need to be a child process of daemon for it to repeat. Here is full .NET test code that I used (and that reproduces the problem):

`namespace HangTest { class Program { static void Main(string[] args) { using(var sw=new System.IO.StreamWriter("/tmp/HangTest")) { sw.AutoFlush=true; for(int i=0;i<100;i++) { sw.WriteLine("HangTest iteration #" + i); var psi = new System.Diagnostics.ProcessStartInfo("/bin/df", "-h"); psi.UseShellExecute = false; psi.CreateNoWindow = true;

                using(var p = new System.Diagnostics.Process())
                {
                    p.StartInfo = psi;
                    p.Start();

                    p.WaitForExit(); //Hangs if running as a child of a daemon process
                }
                System.Threading.Thread.Sleep(1000);
            }
        }
    }
}

} `

From /tmp/HangTest it is easy to monitor whether it works or hangs.

[tmds]

you need to be a child process of daemon for it to repeat.

How can I reproduce this?

[tester346]

@riku76

I'm not sure if it may help you, but check this approach:

https://stackoverflow.com/a/60355879/10522960

public static void ExecuteScriptRx(string path, int processTimeOutMilliseconds, out string logs, out bool success, params string[] args)
{
    StringBuilder output = new StringBuilder();
    StringBuilder error = new StringBuilder();

    using (var process = new Process())
    using (var disposables = new CompositeDisposable())
    {
        process.StartInfo = new ProcessStartInfo
        {
            WindowStyle = ProcessWindowStyle.Hidden,
            FileName = "powershell.exe",
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            UseShellExecute = false,
            Arguments = $"-ExecutionPolicy Bypass -File \"{path}\"",
            WorkingDirectory = Path.GetDirectoryName(path)
        };

        if (args.Length > 0)
        {
            var arguments = string.Join(" ", args.Select(x => $"\"{x}\""));
            process.StartInfo.Arguments += $" {arguments}";
        }

        output.AppendLine($"args:'{process.StartInfo.Arguments}'");

        // Raise the Process.Exited event when the process terminates.
        process.EnableRaisingEvents = true;

        // Subscribe to OutputData
        Observable.FromEventPattern<DataReceivedEventArgs>(process, nameof(Process.OutputDataReceived))
            .Subscribe(
                eventPattern => output.AppendLine(eventPattern.EventArgs.Data),
                exception => error.AppendLine(exception.Message)
            ).DisposeWith(disposables);

        // Subscribe to ErrorData
        Observable.FromEventPattern<DataReceivedEventArgs>(process, nameof(Process.ErrorDataReceived))
            .Subscribe(
                eventPattern => error.AppendLine(eventPattern.EventArgs.Data),
                exception => error.AppendLine(exception.Message)
            ).DisposeWith(disposables);

        var processExited =
            // Observable will tick when the process has gracefully exited.
            Observable.FromEventPattern<EventArgs>(process, nameof(Process.Exited))
                // First two lines to tick true when the process has gracefully exited and false when it has timed out.
                .Select(_ => true)
                .Timeout(TimeSpan.FromMilliseconds(processTimeOutMilliseconds), Observable.Return(false))
                // Force termination when the process timed out
                .Do(exitedSuccessfully => { if (!exitedSuccessfully) { try { process.Kill(); } catch {} } } );

        // Subscribe to the Process.Exited event.
        processExited
            .Subscribe()
            .DisposeWith(disposables);

        // Start process(ing)
        process.Start();

        process.BeginOutputReadLine();
        process.BeginErrorReadLine();

        // Wait for the process to terminate (gracefully or forced)
        processExited.Take(1).Wait();

        logs = output + Environment.NewLine + error;
        success = process.ExitCode == 0;
    }
}

/// <summary>
/// Extension methods associated with the IDisposable interface.
/// </summary>
public static class DisposableExtensions
{
    /// <summary>
    /// Ensures the provided disposable is disposed with the specified <see cref="CompositeDisposable"/>.
    /// </summary>
    public static T DisposeWith<T>(this T item, CompositeDisposable compositeDisposable)
        where T : IDisposable
    {
        if (compositeDisposable == null)
        {
            throw new ArgumentNullException(nameof(compositeDisposable));
        }

        compositeDisposable.Add(item);
        return item;
    }
}

[EHerzog76]

Process.WaitForExit() hangs forever Process.WaitForExit(60000) hangs until the timeout is reached. => on Linux and on Windows

In the case, when you use RedirectStandardOutput -) and you read the result synchron with Proess.ReadToEnd() -) and you start a child-process e.g.: Process.StartInfo.FileName = "powershell.exe"; Process.StartInfo.Arguments = "-NonInteractive -NoProfile -Command "& {Write-Output \"Create a session with New-PSSession ...\"; Invoke-Command -Session $session -ScriptBlock { Get-Process | ConvertTo-Json }; Write-Output \"Delete your session ...\"}";

Until now the only work-a-round is to read the StandardOutput asynchron with Process.BeginOutputReadLine(); Working code pattern can be found here: https://stackoverflow.com/questions/139593/processstartinfo-hanging-on-waitforexit-why/53504707#53504707

[EHerzog76]

The code pattern from tester346 works also, because it do the job asynchronous...