TMC plugin hangs Netbeans on comet connect attempt

[langfod]

Currently I am seeing the TMC plugin hanging/crashing Netbeans when a connection to comet fails in an unexpected manner.

I am guessing this is be caused by my upstream ISP doing proxy tricks as TMC had no issues using a connection at another location.

This has been tested on a newly installed Windows machine with Windows firewall disabled and no other software firewall or anti-virus installed.

If my network is connected at started then Netbeans just hangs ("MoocTailoring" is the only TMC message that makes it to the log file)

If the network connection is disconnected before starting Netbeans, a handshake failed message is logged as expected and Netbeans starts up normally.

INFO [fi.helsinki.cs.tmc.model.PushEventListener]: Comet handshake with failed. Will retry.

If the network is then reconnected and an attempt to "Download/Update Exercises" is made, Netbeans again hangs completely.

According to Wireshark the attempted URL is: http://tmc.mooc.fi:8080/comet

Below is my version information and the Wireshark summary.

Product Version         = Netbeans with TMC 201411181905
Operating System        = Windows 8.1 version 6.3 running on amd64
Java; VM; Vendor        = 1.8.0_25; Java HotSpot(TM) 64-Bit Server VM 25.25-b02;
fi.helsinki.cs.tmc.maven.wrapper [1.5 1.5 201410240045]
fi.helsinki.cs.tmc [0.8.5 141024]
fi.helsinki.cs.tmc.application [1.0 141024]

Wireshark summary:

Source           Destination     Prot Len Info
192.168.1.199    148.251.139.135 TCP  66  58173→8080 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
148.251.139.135  192.168.1.199   TCP  66  8080→58173 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=128
192.168.1.199    148.251.139.135 TCP  54  58173→8080 [ACK] Seq=1 Ack=1 Win=65536 Len=0
192.168.1.199    148.251.139.135 HTTP 257 GET /comet HTTP/1.1 
148.251.139.135  192.168.1.199   TCP  54  8080→58173 [ACK] Seq=1 Ack=204 Win=6912 Len=0
148.251.139.135  192.168.1.199   TCP  54  8080→58173 [FIN, ACK] Seq=1 Ack=204 Win=6912 Len=0
192.168.1.199    148.251.139.135 TCP  54  58173→8080 [ACK] Seq=204 Ack=2 Win=65536 Len=0
192.168.1.199    148.251.139.135 TCP  54  58173→8080 [FIN, ACK] Seq=204 Ack=2 Win=65536 Len=0
148.251.139.135  192.168.1.199   TCP  54  8080→58173 [ACK] Seq=2 Ack=205 Win=6912 Len=0

[mpartel]

Unfortunately our comet stuff is known to be broken server-side at the moment (hasn't been a big priority as its function is very non-essential). But this definitely shouldn't cause NB to hang.

Are you sure NB isn't making a connection attempt to something else just before it hangs? Do you see anything going to port 80 on 148.251.139.135?

I suspect the hang is caused by the exercise downloading attempt rather than comet because the comet handshake should be fully asynchronous. I think we might have a UI-blocking network operation in there somewhere but I wasn't able to find it yet.

[langfod]

I have the checkForUpdates and spyware disabled so on startup with a fully configured preferences file the only connection is that connection to comet.

If I remove the password from the preferences file then Netbeans starts up just fine. I can go into the TMC settings and it will get the course list. As soon as I put in a password and press OK, Netbeans locks up. The preferences window even stays up.

I will try to get a packet trace of it working at another location later today and try to compare them.

[jamo]

It seems that this happens only in rare cases on windows 8.. (I haven't been able to reproduce this yet) What anti- apps and firewall do you use?

One thing you could try is to use https://tmc.mooc.fi/mooc as the server address.

For further debugging Process Monitor from Sysinternals might provide helpful insights.

Tracing only the NetBeans app and then dumping it to .csv or to .xls and in Excel trying to filter out any errors.

I'd be happy to help analysing the file. As it might contain more information than one might want to share in public you can send it to my personal email address which should be visible in my github profile

[langfod]

I will try to put together those traces soon. At the moment it looks like it may be a cometd/BayeuxClient issue. The client.handshake() call in PushEventListener is blocking.

8 WebSocketClient threads start but handshake() never returns...

[jamo]

Without checking the code I'd would say those should be running on separate threads; thus it shouldn't matter too much... Though it might make antivirus/firewall more eager to kill it.

[jamo]

Came to some reports of Logitech Process Monitor application (LVPrcSrv.exe) causing issues if running during installation of NetBeans. Maybe a bit far-fetched but worth checking whether you (or others with similar issues) have it installed.

[langfod]

Okay so what I am seeing is being caused by my upstream ISP attempting to proxy the connection to cometUrl (http://tmc.mooc.fi:8080/comet) . Although the actual site is not responding the proxy still accepts a connection but then causes the call to call to BayeuxClient.handshake() to never return. So this is really some bad code in the cometd library.

However, the call to BayeuxClient.handshake() in fi.helsinki.cs.tmc.model.PushEventListener is blocking the return of its constructor which in turn blocks the initialization of the module in fi.helsinki.cs.tmc.actions.TmcModuleInstall . This causes the UI to appear to freeze on startup if the preferences are already completed. On a new setup this will cause the the UI to appear to freeze when the preferences are saved.

Also, even when the attempt fails properly (Connection refused) the WebClientSocket stays open and the other inner listener (PushEventListener .eventBus.subscribeDependent() ) attempts to reconnect causes another two sets of WebClientSocket scheduler and threads. (24 WebSocketClient threads now). This causes three "Comet handshake with failed." log messages every few seconds.

Setting the "cometUrl" to null in fi.helsinki.cs.tmc.data.Course keeps these attempts from happening.

For now, would it be prudent to suggest that those courses that are not using this feature set the CourseDb results to not include the cometURL?

[jamo]

Thanks for your work on debugging this.

It sounds like the proxy server doesn't work as expected; keeping connection open for non existing resource. Wonder how easy setting it up for my test environment might be :)

Removing comet url completely doesn't sound like a nice plan, even temporarily and may have unforeseen consequences; I'll have to try it first in staging environment. (or just fix this issue).

(Unsetting the value in configs on the server will only default it to its default value.)

We'll figure out soon what is the nicest fix for this. Eventually we should fix both the comet server and the client implementation.

[langfod]

For a lab test perhaps just a service that accepts the connection and input but sends back nothing. I think that would be close to what I the ISP proxy server server is doing.

PushEventListener.initClientIfPossible() checks if the Course has the cometUrl set or if it is null and then does not start the connection attempts.

I was able to test this by just changing Course.getCometUrl() to return null instead of the value received by the server. This also kept it from constantly attempting to connect to a server that is not running.

On Tue, Jan 6, 2015 at 4:04 PM, Jarmo Isotalo notifications@github.com wrote:

Thanks for your work on debugging this.

It sounds like the proxy server doesn't work as expected; keeping connection open for non existing resource. Wonder how easy setting it up for my test environment might be :)

Removing comet url completely doesn't sound like a nice plan, even temporarily and may have unforeseen consequences; I'll have to try it first in staging environment. (or just fix this issue).

(Unsetting the value in configs on the server will only default it to its default value.)

We'll figure out soon what is the nicest fix for this. Eventually we should fix both the comet server and the client implementation.

— Reply to this email directly or view it on GitHub https://github.com/testmycode/tmc-netbeans/issues/94#issuecomment-68933385 .

[jamo]

@langfod I'm just relasing updates regarding this issue. Please let us know whether it fixes the problems you had.

[langfod]

It does indeed work around the issue.

The actual WebSocket thread is still locked up of course but at least TMC can still continue :)

[jamo]

Ok, thanks for confirming it.

Yep, the issue with web sockets persists still but now its not blocking.