Randomly skipping characters/bytes from System.in

[Barteks2x]

When reading data from standard input some characters are skipped. Most of the time it's near 100 skipped characters / 20000 character input (if that matters: I copy and paste the text).

Operating system: Debian 8 (I didn't test it on Windows yet) Java version:

java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

Netbeans version: 8.0.2 Gradle plugin version: 1.3.6

It works correctly when using gradle from command line and in normal netbeans projects.

Here is example code I used to test it: http://pastebin.com/euwxQTZu Randomly generated input (generated by code above): http://pastebin.com/J1wLmrDJ Example output (it's different everytime I run it): http://pastebin.com/gnpKn98k Diff (generated using "diff input output > diff.txt"): http://pastebin.com/4ukg2kCU

[kelemen]

How exactly do you pass the input to stdin?

[Barteks2x]

I paste the whole text into netbeans "output". Last time I tried it didn't work only on windows because of gradle bug. Build.gradle for the example code:

apply plugin: 'java'

sourceCompatibility = '1.8'
[compileJava, compileTestJava]*.options*.encoding = 'UTF-8'

task run(type: JavaExec) {
    main = 'TestInput'
    classpath=  sourceSets.main.runtimeClasspath
    standardInput = System.in
}
repositories {
    mavenCentral()
}

dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.10'
}

[kelemen]

Although it is probably not the issue you are having but there is also a bug in Gradle (it was a long time since I verified it though): You have to press Ctrl + D otherwise your input will not be forwarded. Regardless, I'll check again what is causing the issue you are seeing.

[Barteks2x]

The bug you are talking about exists only on windows (and maybe mac, I don't have mac to test it) This issue is something completely different. And because of the gradle bug it can't be reproduced on windows.

[kelemen]

I have found a bug on my end but unfortunately it is not this (possible failure for unicode characters with two bytes UTF-16 encoding). On my end, the only OS specific is probably the character set. Is the character skipping deterministic for you? That is, will the same input produce the same (incorrect) output always?

[Barteks2x]

It seems to be completely random. I can't test it on windows simply because System.in doesn't work on windows at all, so I don't know if it's OS specific. I can also test it on (X)ubuntu 14.10.

[kelemen]

It "works" on Windows with very limited usefulness. If you press Ctrl + D in the output window, the whole input will be forwarded (and you can no longer send any more input). Anyway, I'll try to check into Gradle's sources because my end should be deterministic (altough there might be a little non-determinism in how NB forwards the input to me).

[Barteks2x]

I also tested it when using System.in directly from buildscript - the same effect. It works correctly when I use "gradle run" command (outside of IDE). Buildscript I tested it with: http://pastebin.com/80PHM2Nn

[kelemen]

It is definitely a Gradle bug. I have written out all the bytes I provide just before Gradle receives them and I have all bytes properly forwarded to Gradle. The bytes get lost after Gradle reads them from me.

Anyway, I can verify your issue. I will try find the bug in Gradle's source and/or report the issue to them.

[kelemen]

Though it seems there is bug in my code as well, so let me fix that one first. Though it results in duplicating some characters.

[kelemen]

What I was seeing was my test code's fault. Now that I fixed it, I cannot reproduce your problem. That is, the only way I lose bytes when I don't press an enter before closing the stream with Ctrl + D. I have tried this on Windows only though. My code is not platform specific though (only depends on the current charset but I tested with UTF-8 as well).

Does your issue happen even if your last character is an enter?

[kelemen]

Altough, on second thought, you terminate with '!', so your loop wouldn't terminate if the missing line terminator would be the cause.

[Barteks2x]

I had to terminate it with something or it would simply never finish if it skipped some characters. And this is the way I discovered it (reading 11000 character input terminated with '!'). I tested it on windows and it works correctly. It seems to happen only on linux (tested on Xubuntu 14.10 and Debian 8).

[Barteks2x]

And if it's gradle bug - it's specific to the way this plugin forwards input to it. It works correctly when I use gradle from command line.

[kelemen]

When you are using the command line, this input forwarding code of Gradle is not used.

[kelemen]

That is, Gradle has to send the input from me to the daemon via TCP/IP.

[kelemen]

Altough still not this issue but I have found why Gradle does not forward stdin on Windows: The reason is because its LineBufferingOutputStream will only flush its buffer once it receives the platform specific line separator (CRLF on Win). However, that's wrong assumption since NB will only add an "\n" when you press Enter.

[kelemen]

I have added a global config (enabled by default) to enable replacing LF characters on the standard input with the platform specific line separator. I expected that this change will allow to reproduce your problem on all platforms. However, I cannot reproduce the issue on Windows. The only strange behaviour is that I lose the end of the input stream if I don't press an Enter key after copy-pasting the input.

[Barteks2x]

I'm not sure if it's the same issue or something completely different. Sometimes when I run my program it initially gets all input but several seconds later it stops receiving input, and after writing a few lines - it works again. If it's the same issue (it seems to be the case) it doesn't stop receiving input for some time, but skips constat (or not constant?) amount of bytes/characters.

[kelemen]

Looking at Gradle's code, it seems pretty likely that the problem is in Gradle's DisconnectableInputStream. I did not yet take the time to check for bugs in it. However, it seems that there was a recent fix in it. Can you try if your problem is still there using the latest nightly of Gradle? You can just configure "2.5-20150523220017+0000" for the version of Gradle to be used in the project properties.

[kelemen]

Actually, the fix was done in 2012. I don't know what I was thinking :) Anyway, I still think that this class might be the culprit.