Provide a function to read standard input other than line-by-line

[mindula]

We are developing a language server written in ballerina. We are using stdio to connect with VSCode. According to language server protocol [1], it sends the following message

Content-Length: ...\r\n
\r\n
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "textDocument/didOpen",
    "params": {
        ...
    }
}

We are not able to read the rest of the JSON file but only the first line that says “Content-Length”. This is because ballerina only provides a way to read line by line in readln. Please provide a function to read content by length or by character.

[1] https://microsoft.github.io/language-server-protocol/specifications/specification-current/#contentPart

[BuddhiWathsala]

@mindula, I'm looking into other languages to come up with a design for this use case. Will you be able to share how other language server implementations handle this scenario?

[BuddhiWathsala]

@mindula @manuranga , WDYT about this workaround?.

import ballerina/io;
import ballerina/lang.value;

public function main() returns error? {
    string fullContent = "";
    string delta = io:readln("");
    fullContent = delta;

    while delta != "" {
        delta = io:readln("");
        fullContent += delta;
    }
    json j = check value:fromJsonString(fullContent);
}

[manuranga]

The issue here is not reading multiple lines. Issues is, there is no way to read anything less than a line, eg read char by char or to read given number of bytes.

If we use your above code it will get stuck at second io:readln("") and wait forever, since server is not ending that line (ie no new line char at the end).

[BuddhiWathsala]

IINM, your requirement is to read the standard input partially without pressing the "enter" key.

I tried to implement a similar use case in Java, but it seems, those kinds of implementations are highly platform-dependent and not portable[1, 2].

If you could share some samples, I can look more into this.

[1] https://www.darkcoding.net/software/non-blocking-console-io-is-not-possible/ [2] https://stackoverflow.com/questions/1066318/how-to-read-a-single-char-from-the-console-in-java-as-the-user-types-it

[manuranga]

I assume these restriction only apply if we write using a console, in our use case the input is piped. Maybe simple System.in.read works in this case, please check by piping in some input eg: cat myfile.txt | java -jar myBal.jar and reading char by char.

[BuddhiWathsala]

I assume these restriction only apply if we write using a console, in our use case the input is piped. Maybe simple System.in.read works in this case, please check by piping in some input eg: cat myfile.txt | java -jar myBal.jar and reading char by char.

@manuranga, I tried a sample and it is working as expected when we use a pipe as you mentioned. We can provide an API to support that. I'll come up with a design and proceed with this task.

[BuddhiWathsala]

Here, we need to provide APIs to cover the following cases:

1. Read a character from standard input
2. Read n number of characters from standard input

There are two possible approaches to provide these use cases as per my initial research.

The existing read API is:

string content = io:readln(any a);

Approach 01

Here, the default case is reading a character. If n > 1 then it will return a string.

string|'string:Char content = io:readChars(any a, int n = 1);

Approach 02

'string:Char content = io:readChar(any a);
string content = io:readNChars(any a, int n);

A slight inconsistency is there with readln API. But I guess this will be a better option when we consider other file APIs.

@daneshk @jclark, please provide your feedback on this.

[manuranga]

Just to clarify, in our use-case, we only know the amount of bytes not chars. The proposed solution still works for us since we can read char-by-char and concat, but we'll not be able to use the multi char version.

[BuddhiWathsala]

@manuranga, just to clarify. If you have a file myfile.txt and you know the number of bytes to read, isn't it possible to use the following APIs that are already in I/O?.

io:fileReadBytes(string path) returns readonly & byte[]|io:Error;
io:fileReadBlocksAsStream(string path, int blockSize = 4096) returns stream<Block, io:Error?>|io:Error

[manuranga]

cat myfile.txt was just an example, in reality it's will be a program, eg vscode-client | java -jar my-ls-server.jar.

[BuddhiWathsala]

If you need byte related APIs, then we can change the aforementioned approaches as follows:

Approach 01

byte[]|byte content = io:readBytes(any a, int n = 1);

Approach 02

byte content = io:readByte(any a);
byte[] content = io:readBytes(any a, int n);

Note, returned byte content can be set as readonly values.

[manuranga]

This will work for us, but even the single char one is enough for us. So please take a decision thinking about what is best from the stdlib point of view.

[jclark]

Specifying number of characters doesn't make much sense to me. The underlying system call will take a number of bytes, and you don't know how many bytes a given number of characters will need.

[BuddhiWathsala]

If you need byte related APIs, then we can change the aforementioned approaches as follows:

Approach 01

byte[]|byte content = io:readBytes(any a, int n = 1);

Approach 02

byte content = io:readByte(any a);
byte[] content = io:readBytes(any a, int n);

Note, returned byte content can be set as readonly values.

Here, I meant n as the number of bytes as they already know it.

[jclark]

I don't understand the any a parameter.

I strongly dislike io:readBytes(any a, int n = 1) with a default of 1. It will not be obvious to a reader that io:readBytes(a) means io:readBytes(a, 1).

[BuddhiWathsala]

Here, the any a parameter is influenced by the existing read API

string content = io:readln(any a);

any a value is the default message that is being printed to the console before the reading.

[jclark]

Arghhhh. That is ghastly! Where did that come from?

[jclark]

There's two different meanings for console io:

1. in Unix terms, /dev/tty - not redirected; always connected to the terminal that a command-line user is interacting through
2. • stdin/stdout/stderr - for a command-line user when not redirected, same as 1, but can be redirected to a file

These MUST NOT be mixed up.

io:println is doing console IO type (2)

A function that takes a prompt (which should be a string) and reads input makes sense for console IO type (1). This would be for getting input interactively from a user. This needs an option to turn off echo (for reading passwords).

@shafreenAnfar io:readln needs fixing ASAP

[daneshk]

@jclark thanks for your input. We will get it fix as soon as possible and will ship with the next release.

[BuddhiWathsala]

A possible approach to handle such use cases is as follows:

Console Read/Write

// Read from the console. These APIs will block till the user presses enter.
string line = io:consoleReadLine();
string passwd = io:consoleReadPassword();

// Write to the console, write line.
io:consoleWriteLine(io:Printable... values);

Stdin/Stdout/Stderr operations

Currently we have these APIs.

// Write to stdout and stderr
io:fprint(io:stdout|io:stderr , string msg);
io:fprintln(io:stdout|io:stderr, string msg);

Need to implement the following APIs.

// Read from stdin
string line = io:fscanLine()
byte b = io:fscanByte();
byte[] bArr =io:fscanBytes(int size);

@jclark, please provide your feedback on this.

[jclark]

-1 to both those.

console stuff shouldn't be in the io module.

Using scan is not appropriate: C fscanf is reading formatted data, but your scan is not doing this.

Using the f prefix is not appropriate: the f in the print case is there because the first argument is a file descriptor.

[jclark]

For reading a line from stdin, I would do just:

string line = io:readln();

I think the reading bytes feature should be part of a general solution for reading and writing to file descriptors (at least stdin/stdout/stderr).