search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.14k stars 4.71k forks source link

System.Console.ReadLine() does not block in pipes in Cygwin/MSYS2: doc or bug? #50780

Open kaby76 opened 3 years ago

kaby76 commented 3 years ago

Description

I've been noticing a problem with C# programs for at least a couple of years having to do with pipes in Cygwin-like shells. Since I am now trying to rewrite my Antlr language toolchain as dotnet tools, I ran into the problem again and decided to try to characterize it and find a workaround.

The problem is that System.Console.ReadLine() returns null with a Cygwin/MSYS2 Bash shell piped input, in which I know there is data on the pipe. If I call System.Console.ReadLine() a second time, and it then returns a non-null result!

The documentation more or less says that ReadLine() is blocking:

But, it does not say so explicitly in the doc "ReadLine() is blocking in pipes/redirection". In fact, I always assumed so, and virtually everywhere (e.g., StackOverflow), everyone also assumes that ReadLine() blocks. That is why there is a ReadAsync() version for Read().

I decided to make two programs "w" and "r" (source here) that duplicates the problem.

The "w" program simply writes to stdout, and the "r" program reads stdin using ReadLine(), and it is called in Bash as w.exe | r.exe. Note, I have tried also using System.Console.In, etc., and the behavior is the same. When "r" is run in a pipe, ReadLine() fails intermittently (see this screenshot). On subsequent calls, the data in the pipe is available (uncomment the commented code to see it in action).

Here is the writer code:

using System;
using System.Collections.Generic;
using CommandLine;

namespace w
{
    public class Config
    {
        [Option("version", Required = false)]
        public bool? Version { get; set; }

        [Value(1)]
        public IEnumerable<int> Times { get; set; }

        [Option("type", Required = false)]
        public string? Type { get; set; }
    }

        class Program
        {
        static void Main(string[] args)
        {
            int count = 1;
            var result = Parser.Default.ParseArguments<Config>(args)
            .WithParsed(options =>
            {
                if (options.Times != null)
                {
                    var i = options.Times.GetEnumerator();
                    if (i.MoveNext()) count = i.Current;
                }
            }
            )
            .WithNotParsed(errors =>
            {
            });

            System.Console.Error.WriteLine("Parse completed of C:\\Users\\kenne\\Documents\\AntlrVSIX\\test\\A.g4");
            for (int j = 0; j < count; ++j)
            {
                var s = @"{
  ""FileName"": ""C:\\Users\\kenne\\Documents\\AntlrVSIX\\test\\A.g4""
}";
                System.Console.WriteLine(s);
            }
        }
    }
}

The reader code is:

using System;
using System.Collections.Generic;

namespace r
{
    class Program
    {
    static void Main(string[] args)
    {
        //bool first = true;
        List<string> lines = new List<string>();
        for (; ; )
        {
        var s = System.Console.ReadLine();
        if (s == null)
        {
            //if (lines.Count == 0 && first)
            //    continue;
            break;
        }
        lines.Add(s);
        }
        System.Console.WriteLine("Got " + String.Join(" ", lines));
    }
    }
}

On Linux, it works exactly as you would expect. In Cmd.exe and Powershell.exe, it works exactly as you would expect.

The question is whether this is a bug in ReadLine() or a doc issue?

The workaround is to ignore the ReadLine() result if null, and try again. But, it's not clear when to quit trying.

Configuration

MSYS2: $ pacman -Q asciidoc 9.0.4-1 autoconf 2.69-5 autoconf2.13 2.13-3 autogen 5.18.16-1 automake-wrapper 11-1 automake1.10 1.10.3-4 automake1.11 1.11.6-4 automake1.12 1.12.6-4 automake1.13 1.13.4-5 automake1.14 1.14.1-4 automake1.15 1.15.1-2 automake1.16 1.16.2-2 automake1.6 1.6.3-3 automake1.7 1.7.9-3 automake1.8 1.8.5-4 automake1.9 1.9.6-3 base 2020.12-1 bash 5.1.004-1 bash-completion 2.11-1 bc 1.07.1-2 binutils 2.35.1-1 bison 3.7.4-1 brotli 1.0.9-1 bsdtar 3.5.0-1 btyacc 20200910-1 bzip2 1.0.8-2 ca-certificates 20190110-1 cmake 3.18.4-1 coreutils 8.32-1 curl 7.74.0-4 dash 0.5.11.2-1 db 5.3.28-3 diffstat 1.63-1 diffutils 3.7-1 docbook-xml 4.5-2 docbook-xsl 1.79.2-1 dos2unix 7.4.2-1 expat 2.2.10-1 expect 5.45.4-2 file 5.39-2 filesystem 2021.01-1 findutils 4.7.0-1 flex 2.6.4-1 gawk 5.1.0-1 gcc 10.2.0-1 gcc-libs 10.2.0-1 gdb 9.2-3 gdbm 1.18.1-3 getent 2.18.90-2 gettext 0.19.8.1-1 gettext-devel 0.19.8.1-1 git 2.30.0-1 glib2 2.66.4-1 gmp 6.2.1-1 gnupg 2.2.25-1 gperf 3.1-2 grep 3.0-2 groff 1.22.4-1 gzip 1.10-1 heimdal 7.7.0-2 heimdal-libs 7.7.0-2 help2man 1.47.16-1 icu 67.1-1 inetutils 1.9.4-2 info 6.7-3 intltool 0.51.0-2 isl 0.22.1-1 jsoncpp 1.9.4-1 less 563-2 libarchive 3.5.0-1 libargp 20110921-2 libasprintf 0.19.8.1-1 libassuan 2.5.4-1 libatomic_ops 7.6.10-1 libbz2 1.0.8-2 libcrypt 2.1-2 libcurl 7.74.0-4 libdb 5.3.28-3 libedit 20191231_3.1-2 libexpat 2.2.10-1 libffi 3.3-1 libgc 8.0.4-1 libgcrypt 1.8.7-1 libgdbm 1.18.1-3 libgettextpo 0.19.8.1-1 libgnutls 3.7.0-1 libgpg-error 1.39-1 libgpgme 1.15.0-1 libguile 2.2.7-1 libhogweed 3.6-1 libiconv 1.16-2 libiconv-devel 1.16-2 libidn2 2.3.0-1 libintl 0.19.8.1-1 libksba 1.4.0-1 libltdl 2.4.6-9 liblz4 1.9.3-1 liblzma 5.2.5-1 libmetalink 0.1.3-3 libnettle 3.6-1 libnghttp2 1.41.0-1 libnpth 1.6-1 libopenssl 1.1.1.i-1 libp11-kit 0.23.21-1 libpcre 8.44-1 libpcre16 8.44-1 libpcre2_16 10.36-1 libpcre2_32 10.36-1 libpcre2_8 10.36-1 libpcre2posix 10.36-1 libpcre32 8.44-1 libpcrecpp 8.44-1 libpcreposix 8.44-1 libpipeline 1.5.3-2 libpsl 0.21.1-2 libreadline 8.1.0-1 librhash 1.4.0-2 libsqlite 3.34.0-1 libssh2 1.9.0-1 libtasn1 4.16.0-1 libtool 2.4.6-9 libunistring 0.9.10-1 libunrar 6.0.3-1 libunrar-devel 6.0.3-1 libutil-linux 2.35.2-1 libuv 1.40.0-1 libxml2 2.9.10-6 libxslt 1.1.34-3 libzstd 1.4.8-1 m4 1.4.18-2 make 4.3-1 man-db 2.9.3-1 mingw-w64-x86_64-brotli 1.0.9-2 mingw-w64-x86_64-bzip2 1.0.8-2 mingw-w64-x86_64-c-ares 1.16.1-1 mingw-w64-x86_64-ca-certificates 20200601-2 mingw-w64-x86_64-cmake 3.19.2-1 mingw-w64-x86_64-curl 7.74.0-2 mingw-w64-x86_64-expat 2.2.10-1 mingw-w64-x86_64-gcc-libs 10.2.0-6 mingw-w64-x86_64-gettext 0.19.8.1-10 mingw-w64-x86_64-gmp 6.2.1-1 mingw-w64-x86_64-jansson 2.13.1-1 mingw-w64-x86_64-jemalloc 5.2.1-1 mingw-w64-x86_64-jsoncpp 1.9.4-1 mingw-w64-x86_64-libarchive 3.5.1-1 mingw-w64-x86_64-libffi 3.3-2 mingw-w64-x86_64-libiconv 1.16-2 mingw-w64-x86_64-libidn2 2.3.0-1 mingw-w64-x86_64-libmetalink 0.1.3-3 mingw-w64-x86_64-libpsl 0.21.1-2 mingw-w64-x86_64-libssh2 1.9.0-2 mingw-w64-x86_64-libsystre 1.0.1-4 mingw-w64-x86_64-libtasn1 4.16.0-1 mingw-w64-x86_64-libtre-git r128.6fb7206-2 mingw-w64-x86_64-libunistring 0.9.10-2 mingw-w64-x86_64-libuv 1.40.0-1 mingw-w64-x86_64-libwinpthread-git 9.0.0.6090.ad98746a-1 mingw-w64-x86_64-lz4 1.9.2-1 mingw-w64-x86_64-make 4.3-1 mingw-w64-x86_64-mpc 1.2.1-1 mingw-w64-x86_64-mpdecimal 2.5.0-1 mingw-w64-x86_64-mpfr 4.1.0-3 mingw-w64-x86_64-ncurses 6.2-2 mingw-w64-x86_64-nettle 3.6-2 mingw-w64-x86_64-nghttp2 1.41.0-1 mingw-w64-x86_64-openssl 1.1.1.i-1 mingw-w64-x86_64-p11-kit 0.23.21-4 mingw-w64-x86_64-pkg-config 0.29.2-3 mingw-w64-x86_64-python 3.8.7-1 mingw-w64-x86_64-python-appdirs 1.4.3-1 mingw-w64-x86_64-python-attrs 19.3.0-1 mingw-w64-x86_64-python-cachecontrol 0.12.6-1 mingw-w64-x86_64-python-certifi 2019.11.28-1 mingw-w64-x86_64-python-chardet 3.0.4-1 mingw-w64-x86_64-python-colorama 0.4.3-1 mingw-w64-x86_64-python-contextlib2 0.6.0-1 mingw-w64-x86_64-python-distlib 0.3.1-1 mingw-w64-x86_64-python-html5lib 1.1-1 mingw-w64-x86_64-python-idna 2.9-1 mingw-w64-x86_64-python-lockfile 0.12.2-1 mingw-w64-x86_64-python-msgpack 1.0.0-1 mingw-w64-x86_64-python-ordered-set 3.1.1-1 mingw-w64-x86_64-python-packaging 20.3-1 mingw-w64-x86_64-python-pep517 0.9.1-1 mingw-w64-x86_64-python-pip 20.0.2-1 mingw-w64-x86_64-python-progress 1.5-1 mingw-w64-x86_64-python-pyparsing 2.4.7-1 mingw-w64-x86_64-python-pytoml 0.1.21-1 mingw-w64-x86_64-python-requests 2.23.0-1 mingw-w64-x86_64-python-retrying 1.3.3-1 mingw-w64-x86_64-python-setuptools 47.1.1-1 mingw-w64-x86_64-python-six 1.15.0-1 mingw-w64-x86_64-python-urllib3 1.25.9-1 mingw-w64-x86_64-python-webencodings 0.5.1-1 mingw-w64-x86_64-readline 8.0.004-1 mingw-w64-x86_64-rhash 1.4.0-1 mingw-w64-x86_64-sqlite3 3.34.0-1 mingw-w64-x86_64-tcl 8.6.10-1 mingw-w64-x86_64-termcap 1.3.1-6 mingw-w64-x86_64-tk 8.6.10-2 mingw-w64-x86_64-xz 5.2.5-2 mingw-w64-x86_64-zlib 1.2.11-8 mingw-w64-x86_64-zstd 1.4.5-1 mintty 1~3.4.4-1 mpc 1.1.0-1 mpdecimal 2.5.0-1 mpfr 4.1.0-1 msys2-keyring 1~20201002-1 msys2-launcher 1.0-1 msys2-runtime 3.1.7-4 msys2-runtime-devel 3.1.7-4 msys2-w32api-headers 8.0.0.5683.629fd2b1-1 msys2-w32api-runtime 8.0.0.5683.629fd2b1-1 nano 5.4-1 ncurses 6.2-1 nettle 3.6-1 openssh 8.4p1-1 openssl 1.1.1.i-1 p11-kit 0.23.21-1 pacman 5.2.2-9 pacman-contrib 1.4.0-1 pacman-mirrors 20201208-1 pactoys-git r2.07ca37f-1 patch 2.7.6-1 patchutils 0.4.2-1 pcre 8.44-1 pcre2 10.36-1 perl 5.32.0-2 perl-Authen-SASL 2.16-2 perl-Clone 0.45-2 perl-Convert-BinHex 1.125-1 perl-Encode-Locale 1.05-1 perl-Error 0.17029-1 perl-File-Listing 6.14-1 perl-HTML-Parser 3.75-1 perl-HTML-Tagset 3.20-2 perl-HTTP-Cookies 6.09-1 perl-HTTP-Daemon 6.12-1 perl-HTTP-Date 6.05-1 perl-HTTP-Message 6.26-1 perl-HTTP-Negotiate 6.01-2 perl-IO-HTML 1.004-1 perl-IO-Socket-SSL 2.068-1 perl-IO-Stringy 2.113-1 perl-LWP-MediaTypes 6.04-1 perl-Locale-Gettext 1.07-7 perl-MIME-tools 5.509-1 perl-MailTools 2.21-1 perl-Module-Build 0.4231-1 perl-Net-HTTP 6.19-1 perl-Net-SMTP-SSL 1.04-1 perl-Net-SSLeay 1.89_01-3 perl-TermReadKey 2.38-2 perl-Test-Pod 1.52-1 perl-TimeDate 2.33-1 perl-Try-Tiny 0.30-1 perl-URI 5.05-1 perl-WWW-RobotRules 6.02-2 perl-XML-Parser 2.46-3 perl-YAML-Syck 1.32-4 perl-inc-latest 0.500-1 perl-libwww 6.50-1 pinentry 1.1.0-2 pkgconf 1.7.3-2 pkgfile 21-1 python 3.8.6-1 quilt 0.66-2 rebase 4.4.4-2 reflex 20200715-1 scons 3.1.2-4 sed 4.8-1 swig 4.0.2-1 tar 1.32-1 tcl 8.6.10-1 texinfo 6.7-3 texinfo-tex 6.7-3 tftp-hpa 5.2-3 time 1.9-1 ttyrec 1.0.8-2 tzcode 2020d-1 unrar 6.0.3-1 util-linux 2.35.2-1 vim 8.2.1895-1 wget 1.20.3-1 which 2.21-2 windows-default-manifest 6.4-1 xmlto 0.0.28-2 xz 5.2.5-1 zlib 1.2.11-1 zstd 1.4.8-1

dotnet-issue-labeler[bot] commented 3 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

kaby76 commented 3 years ago

This is a fundamental problem with C# dotnet runtime, so it must go here. I don't care how it is labeled. Is ReadLine() blocking or not in pipes? Is it supposed to be? Why does it not work consistently across shells and OSes? If it's not blocking, how do I detect the end of the pipe input?

kaby76 commented 3 years ago

I tested this with the Cygwin configuration flag pipe_byte set and unset. It makes no difference: ReadLine() does not block on the pipe. I still have to call ReadLine() a second time. It's likely Read() also does not work, but I haven't tested it with the pipe_byte configuration flag (I have done so previously--it also does not block).