High CPU

[mpcjanssen]

After killing the jupyter notebook the kernels seem to keep running and taking a constant 1-3% of CPU power.

[mpcjanssen]

This probably has to do with the fact that we can't detect when a client goes away.

[mpcjanssen]

Actual problem is constant polling probably because of the readable callback on the zmq sockets.

[mpcjanssen]

it's a tclzmq issue. The following script has high cpu:

package require zmq

zmq context ctx

vwait forever

[mrcalvin]

It is an issue by design, as internal to tclzmq, the ZMQ events are married with the Tcl event loop using (as you write) constant polling. tclzmq would need to adopt another integration technique (e.g., using ZMQ_USE_FD plus Tcl_CreateFileHandler), but this would lead to platform-specific pieces of implementation (e.g., on Win, Tcl has no Tcl_CreateFileHandler!). However, in absence of readable/ writeable handlers, polling should be paused ... maybe, as a workaround, use a default handler with some sleep time to cool down the polling?

[mpcjanssen]

Using Tcl event handlers with after and manually polling works fine. I am thinking about adding a switch to disable the event source integration

[mpcjanssen]

This is fixed by a combination of:

• https://github.com/mpcjanssen/tclzmq/commit/3e78eea16353ae511498298988d26456db3e1891
• https://github.com/mpcjanssen/tcljupyter/commit/70dbc8356ae2a7725a352f79019e948c3490ad77

[mpcjanssen]

I am tempted to build a minimal zmq client in Tcl only for Tcljupyter use which would use socket readable handlers

[mrcalvin]

This is fixed by a combination of:

• mpcjanssen/tclzmq@3e78eea
• 70dbc83

I used to solve this by using after in the writable (or, readable) handler, no need to deactivate event-source registration ... there is no difference to this end. But interval polling will dwarf throughput (among other issues), which is acceptable for an interactive application (like Jupyter notebooks), but not so acceptable in other settings.

[mrcalvin]

I am tempted to build a minimal zmq client in Tcl only for Tcljupyter use which would use socket readable handlers

I had the same thought, once, but it would require programming against Windows APIs directly (as I said, there is no Tcl file-handler abstraction under Win), I did not want to jump into this rabbit hole.

[mpcjanssen]

Why would a readable event handler on the zmq sockets not work? No platform API in sight.

[mrcalvin]

Why would a readable event handler on the zmq sockets not work? No platform API in sight.

Oh, you mean, create a Tcl channel type for ZMQ sockets? Well, right now, ZMQ sockets are no Tcl channels ... the reason being, probably, that ZMQ sockets are not sockets in the purest sense, you will have to extract the sibling sockets for each ZMQ socket ... and this will bring platform APIs in sight. If you don't, from what I found out back then, you will end up using polling again just under the hood of a Tcl channel ...

but maybe I am just wrong?

[mpcjanssen]

No just a plain Tcl server socket. This obviously will only support tcp:// type 0MQ sockets.

[mpcjanssen]

Cleanest way would be to do a blocking poll on all sockets in a separate thread and remove the event sourcing. And wrap this in a higher level API.