Polling for input / non-blocking reproc_read

[brechtsanders]

Hi Daan, Would it be possible to have a variation of reproc_read that doesn't block but just returns a status code indicating if data is waiting to be read or not? Or alternatively a reproc_read that blocks for a specified period (e.g. in milliseconds) which returns 0 if no data is waiting to be read, > 0 if data is available and < 0 in case of error? Groeten, Brecht

[DaanDeMeyer]

There's the timeout option in reproc_options. However, setting it to zero currently defaults to waiting indefinitely. However, I can add a new constant REPROC_NONBLOCKING that actually sets the timeout to zero.

Can you tell me a little bit more about your scenario/use case? Adding the constant will work but reproc_read's implementation already completely against the Windows way of doing async I/O, so I don't think it's going to be very performant if you start calling it very often. Maybe we can find an even better solution (I'll add the constant regardless though).

[brechtsanders]

Can you maybe add a small example application that waits reads all output from a process? The ideal test process to read from should be something that doesn't immediately sends output (maybe after a second), and that sends output, but not continously (e.g. some data, 1 second pause, then some more data). I would like to know how to read this in a simple (single thread) application using reproc.

My typical use case would be calling commands to build software (run a shell and in that shell run things line ./configure and make).

Also note that the comments don't say which unit the timeout option is in (milliseconds?).

[DaanDeMeyer]

Which comments do you mean? The docstring of timeout in reproc_options mentions that it takes a value in milliseconds.

The reproc drain and git-status examples show how to read all output from a process. If you don't want reproc_read to block indefinitely, Take the git-status example and pass a timeout to reproc_options. You can then check if reproc_read timed out by comparing the return value against REPROC_ETIMEDOUT. The same can be done with the drain example that takes away some of the boilerplate.

Do I understand correctly that you want to run a shell with reproc and then run commands in that shell? If that's the case, I'm curious what prevents you from running the commands themselves through reproc. Wouldn't that be a simpler approach?

If necessary, I can add a shell option to reproc_options to allow running commands in a shell if you need specific shell features (although I would recommend seeing if you can work without a shell first).

[brechtsanders]

Looks like I was looking at an older version where milliseconds is only mentioned in the comments above reproc_wait, not above timeout in reproc_options.

My build scripts are basically shell scripts. One line may set an environment variable that is accessed in another line, so executing seperately doesn't make sense here. Just invoking bash and reading/writing from/to it is fine for me. It's not that interactive. I already have this implemented in my current build tool, but I had some occasional issues. Your library seems so much better than reinventing the wheel...

[DaanDeMeyer]

Are you processing the output from the shell in any way? reproc already supports inheriting stdin/stdout/stderr from the parent or redirecting them to /dev/null or pipes but if you want to simply store the shell commands to run in a script, I can add support to redirect to file descriptors/handles as well.

You could then start the shell, redirect stdin to a file containing the shell commands to execute and redirect stdout/stderr to separate log files. After starting the shell, you could simply wait for the shell to exit using reproc_wait or similar. Of course, this only works if you don't need to process the output in any way while it is being produced by the shell. If you're really lucky, everything could turn into a oneliner using reproc_run where you pass the commands to execute as the input option in reproc_options and the output automatically gets written to the correct files without requiring any extra work from the parent process.

[brechtsanders]

My current implementation also uses pipes to write to/read from the shell. The last command sent to the shell is exit, so yes its possible to just wait for the shell to exit.

[DaanDeMeyer]

Are you doing anything special with the output you read from the shell?