Kotlin / kotlinx-io

Kotlin multiplatform I/O library
Apache License 2.0
1.28k stars 57 forks source link

Provide Sink/Source implementations writing to stdout/reading from stdin #375

Open fzhinkin opened 2 months ago

fzhinkin commented 2 months ago

Some applications may require reading from stdin or writing to stdout, but there's no built-in factories/functions returning corresponding Sink/Source implementations.

It should be trivial to provide them for all targets except JS running in the browser. For the latter, we can return an always-exhausted Source and a Sink writing to console.log.

lppedd commented 2 months ago

Very welcomed enhancement. I was considering implementing this kind of project, and having stdin and stdout for Native implemented here would definitely make my life easier.

stderr would also be useful.

JakeWharton commented 2 months ago

On ~native~ POSIX, being able to wrap a raw sink / raw source around any file descriptor would be convenient. The biggest downside is exposing the type-unsafe FD parameter as just an Int or whatever. But presumably it would be the implementation you'd use for stdin and stdout on native anyway, so having it public saves people from getting it wrong when they want to read/write some other FD.

fzhinkin commented 2 months ago

we can return an always-exhausted Source and a Sink writing to console.log.

console.log/error won't work here as is: RawSink is dealing with raw bytes, not a strings. We can, probably, try to reinterpret these bytes as string ourselves, and then emit the result into console.log, but I'm not sure if it's worth the effort.

fzhinkin commented 2 months ago

Another option to consider: if these stdin/out/err Source/Sinks should be buffered or Raw. Buffered sinks and sources could reduce chances of closing application-wide streams/files accidentally by writing:

stdoutSink().buffered().use {
  it.writeString("Goodbye, /proc/self/fd/1")
}

Instead, it would be:

val out: Sink = stdoutSink()
out.writeString("Hey!")
out.flush()
lppedd commented 2 months ago

console.log/error won't work

For the browser I'd just avoid implementing it for the first iteration, let's see how usable the outcome is first.
For Node.js you can use process.stdin/stdout/stderr with fs.readSync I suppose, although for this kind of streams it would cool to have a suspending variant at some point.

fzhinkin commented 2 months ago

Yeap, for NodeJs everything works with existing FileSink and FileSource implementations (assuming their constructors will accept FD instead of a file path).

lppedd commented 2 months ago

It looks like you'll just have to create secondary constructors.

I see that the Node.js version uses readFileSync. You could switch to readSync and keep your own Buffer instance around, which would also avoid having to assert (!!) it.

qwwdfsad commented 2 months ago

Another option to consider: if these stdin/out/err Source/Sinks should be buffered or Raw.

When re-doing readln, we explicitly ruled out buffering to avoid interference with other platform sources (e.g. System.in or System.console()) -- the idea was to be able to have independent readLine calls between various sources. The argument mostly holds if we are going to provide top-level access like Source.stdin

qwwdfsad commented 2 months ago

On a side note, we might be able to finally address the annoying windows console encoding issue

fzhinkin commented 2 months ago

The argument mostly holds if we are going to provide top-level access like Source.stdin

Then we have to use some alternative buffering implementation as the current one prefetch data eagerly.

fzhinkin commented 2 months ago

Source.stdin could be Source, not a RawSource. And we can use either new implementation, or parameterize RealSource to buffer only requested number of bytes from the underlying input stream / fd. That'll prevent unwanted interference with other platform sources.

However, that'll work until the first .buffered() call on Source.stdin as a newly created buffered source will buffer eagerly. And since we encourage users to use RawSource where possible, Source.stdin will likely be wrapped into yet another buffered source somewhere, where its origin will be unknown:

fun parseJson(source: RawSource) {   
   val buffered = source.buffered()
   ...
}

// jq.kt
fun main(args: Array<String>) {
   val inputFileName = getFileName(args)
   if (inputFileName == "-") {
     parseJson(Source.stdin)
   } else {
      SystemFileSystem.source(Path(inputFileName)).use {
         parseJson(it)
      }
   }
}

Yet another issue is that Source.exhausted needs to read at least a single byte from an underlying source, so it will always cause interference.

fzhinkin commented 3 weeks ago

After some discussions, it seems like the fact that stdin-backed Source will mess up with readln is not a problem (or, to be more precise, it's not a problem we can solve for Source):

It's worth mentioning that such a source won't replace something similar to Java's Console class and if interactive I/O through a virtual terminal is required, some other interface should be introduced.

The same considerations are also applicable to stdout/err-backed Sinks.

fzhinkin commented 3 weeks ago

Other questions, however, remain open:

JakeWharton commented 3 weeks ago

On Java, should these sinks/sources follow changes of corresponding System.in/out/err streams?

I'm a pretty strong "no" on this.

The analogous behavior on the JVM would be calling System.out to obtain an OutputStream, and wanting that OutputStream instance to change where it writes to if someone calls System.setOut.

This problem isn't limited to the JVM. If you close the stdin FD on native, the next opened file will get the "standard" STDIN FD number.

fzhinkin commented 6 days ago

This problem isn't limited to the JVM. If you close the stdin FD on native, the next opened file will get the "standard" STDIN FD number.

It seems to be the same as

follow changes of corresponding System.in/out/err streams

So if we want not to follow, on native a duplicate FD is needed. As a bonus, duplicating an FD will solve an issue with accidentally closing a stdin/out/err stream.