Feature Request: Windows Support

[dhirschfeld]

I've opened this issue to collect any discussion around what would be involved in supporting Windows. This issue can then hopefully provide a technical resource for any sufficiently motivated individual to take a crack at it!

[dhirschfeld]

I'm assuming this should at least be possible since VSCode provides terminal support on both Windows and Linux.

xref: Microsoft/vscode#143

[dhirschfeld]

It also seems spyder provides terminal support on Windows based on xterm.js and pywinpty

https://github.com/spyder-ide/spyder-terminal

[takluyver]

@ccordoba12 @andfoy : What's your experience providing terminals in Spyder as part of Spyder? I've seen winpty before and it looked far more complex than I want to support, but it looks like you're doing a lot of the hard work to make it installable from Python.

Maybe the answer here is for someone to write a Windows equivalent of terminado, and Jupyter to use that on Windows.

[andfoy]

@takluyver, Until now, we haven't got problems with our Windows console implementation, we even wrote tests for pywinpty, and its stable to use. Our server is also based on tornado, so if you would like to merge some changes from our implenentation at Spyder-terminal, I could send a PR.

[ccordoba12]

@andfoy, please help @takluyver to implement Windows support for Terminado. It shouldn't be so complicated because you did it quite easily in spyder-terminal.

[andfoy]

@takluyver Is it be feasible to make Terminado dependent on pexpect to simplify the current term_manager logic and write something more close to spyder-ide/spyder-terminal/server/term_manager.py ?

[Schmetzler]

I wrote a wrapper to wrap the ptyproc functions for winpty.

#winpty_wrapper.py
"""Wrapper for winpty to Terminado
"""

from winpty import PTY

class WPTYW(object):
    next_fd = 0
    delayafterterminate = 1

    @staticmethod
    def spawn(argv,env,cwd):
        command = r'C:\windows\system32\cmd.exe'
        #command = r'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe'
        pty = WPTYW(command,80,25)
        return pty

    def __init__(self,command,cols,rows):
        self.fd = WPTYW.next_fd
        WPTYW.next_fd += 1

        self.cols = cols
        self.rows = rows

        self.pty = PTY(self.cols,self.rows)
        self.pty.spawn(command)

    def isalive(self):
        return self.pty.isalive()

    def kill(self,sig):
        self.pty.close()
        del self.pty
        del self

    def setwinsize(self,cols,rows):
        self.cols = cols
        self.rows = rows
        self.pty.set_size(cols,rows)

    def getwinsize(self):
        return (self.cols, self.rows)

    def read(self,length=1):
        return self.pty.read(length)

    def write(self,data):
        return self.pty.write(data)

and replaced line 18 in management.py with from winpty_wrapper.py import WPTYW as PtyProcessUnicode

but jupyter crashes with the following when I try to launch a terminal:

Traceback (most recent call last):
  File "E:\Programme\Anaconda3\Scripts\jupyter-notebook-script.py", line 5, in <module>
    sys.exit(notebook.notebookapp.main())
  File "E:\Programme\Anaconda3\lib\site-packages\jupyter_core\application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "E:\Programme\Anaconda3\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "E:\Programme\Anaconda3\lib\site-packages\notebook\notebookapp.py", line 1417, in start
    self.io_loop.start()
  File "E:\Programme\Anaconda3\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "E:\Programme\Anaconda3\lib\site-packages\tornado\ioloop.py", line 862, in start
    event_pairs = self._impl.poll(poll_timeout)
  File "E:\Programme\Anaconda3\lib\site-packages\zmq\eventloop\ioloop.py", line 122, in poll
    z_events = self._poller.poll(1000*timeout)
  File "E:\Programme\Anaconda3\lib\site-packages\zmq\sugar\poll.py", line 99, in poll
    return zmq_poll(self.sockets, timeout=timeout)
  File "zmq/backend/cython/_poll.pyx", line 123, in zmq.backend.cython._poll.zmq_poll (zmq\backend\cython\_poll.c:2186)
  File "zmq/backend/cython/_poll.pyx", line 116, in zmq.backend.cython._poll.zmq_poll (zmq\backend\cython\_poll.c:2036)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq\backend\cython\_poll.c:2811)
    raise ZMQError(errno)
zmq.error.ZMQError: Unknown error

[takluyver]

@andfoy it already depends on ptyprocess, which is the library underlying pexpect. I think this is the right level of abstraction: we don't need the pexpect APIs for what terminado does. Is that workable for you?

@Schmetzler I think the problem there is that your 'fd' is just a sequential number, not a real file descriptor. The fd is used in the event loop to know when there is data ready to read from the pty. I'm not sure if anything equivalent will be possible with the Windows version, which is one of the things putting me off it.

[andfoy]

@takluyver, do we need the fd number of the input pipe? I do actually have access to the names of those file descriptors, as they are given by winpty

[takluyver]

You could try that, but I'm not sure whether it will work with the event loop on Windows. IIRC the Windows select() function, and presumably any similar calls pyzmq might use, only accepts fds of network sockets. I don't think it would work for a pipe, if that's what winpty is using. Give it a try, though, I don't know how winpty works, so it's possible that you can use it like that.

[andfoy]

It seems that tornado.ioloop.add_handler does not supports pipes/file handles on Windows, only sockets, as @takluyver pointed out. So unless the output of the winpty file handles can be redirected to a socket, we cannot adapt Terminado to use winpty

[andfoy]

Another solution would be to use a periodic callback on Windows, as we do on spyder-terminal

[dhirschfeld]

I'm not up to speed with the issues so this may be completely OT, but the docstring for the tornado Waker class seems suggestive. Maybe there's something useful there?

class Waker(interface.Waker):
    """Create an OS independent asynchronous pipe.
    For use on platforms that don't have os.pipe() (or where pipes cannot
    be passed to select()), but do have sockets.  This includes Windows
    and Jython.
    """

[andfoy]

Thanks for the suggestion @dhirschfeld, I will take a look at it tomorrow

[takluyver]

To be honest with you: I don't want to support Windows for terminado. It's fiddly enough supporting Unix correctly, but at least that's a well scoped problem, and I can work with well-understood PTYs and signals. Windows doesn't have these things natively, and I suspect that trying to bridge the gap is likely to be a constant source of bugs and misunderstandings.

I can also try things easily on my Linux machine, whereas starting a Windows VM and putting together the pieces to try things out is more effort.

If someone wants to do a Windows clone of terminado and maintain it, I think we'd be happy to use that in Jupyter. Maybe spyder-terminal is already compatible enough that we could use it? But I don't want to take responsibility for Windows code in terminado at the moment.

[andfoy]

@takluyver, @dhirschfeld, do you prefer if we extract our terminal server to another Python package that can be used independently without any dependence on any Spyder package? What do you think @ccordoba12?

[takluyver]

That sounds good to me.

[andfoy]

After discussing with @ccordoba12, we concluded that separating the server from spyder-terminal is more difficult than I expected initially. To execute some routines on Qt, we need some Javascript functions that are embedded on a static file served by our server, so we don't think we would separate it from our sources. The only solution that I think of, is to move the server to my personal account, so that you can use it at Jupyter @dhirschfeld, however, you must be aware that I won't have much time to maintain it.

[takluyver]

We may be willing to maintain it under the Jupyter org, if you like. I can ask the others tomorrow. I just don't want to end up personally responsible for the Windows stuff which I don't really use or know much about.

[dhirschfeld]

First off, I'd just like to say I'm perfectly fine with and support @takluyver not wanting to personally maintain Windows code/workarounds in terminado and I hope that opening this issue didn't come across as demanding that you do so!

That said, I'd still like to have Windows teminal support in JupyterLab and whilst all the experts are in the room (so to speak) it would be good to figure out what the path of least resistance is towards that goal. It seems there are two potential options:

1. Fork terminado to provide Windows support
2. Integrate the spyder terminal into JupyterLab

1 has the benefit that terminado is already integrated into JupyterLab so if it (or a fork with the same api) supported Windows the job would already be done (IIUC). From the discussion above it seems that the design of terminado (or limitations of Windows) would make this a non-trivial task.

2 has the benefit of already supporting Windows but it would have to be integrated into JupyterLab which might require changes in both projects?

Either approach would require someone with the time/skills/motivation to step up and do the work and someone would have to be willing to maintain it to avoid bit-rot. I'm certainly willing to help with the maintenance part as I would expect to be using it regularly but I'm not sure I have the requisite (javascript in particular) skills to actually implement either solution. I'm certainly willing to help in any way I can though...

[dhirschfeld]

Finally, if neither option is is feasible it might still be possible to run terminado as-is using Windows Subsystem for Linux.

I've shown that it's possible in #32 but I expect it'll require some work in JupyterLab to enable it there...

[takluyver]

No problem - it didn't come across as demanding. It's something that makes sense to do, and it would be good to have it supported in Jupyter, I just realised that if I don't push back a bit, I could easily find myself maintaining something I don't know much about or have much personal motivation for.

For me, the ideal next steps would be for someone familiar with Windows to fork terminado and make it work with pywinpty. For the first pass, don't worry about trying to keep compatibility with Unix, just make it work on Windows. spyder-terminal is probably a useful reference, but it sounds like we can't use it directly.

Once that's done, and we can see how big the differences are, we can decide whether we want to integrate Windows support back into terminado, or publish it as a separate module which Jupyter uses.

[andfoy]

What are the integration functionalities you need to support between JupyterLab and the terminal?

[andfoy]

I can adapt spyder-terminal, however, someone has to maintain it later on

[takluyver]

I'm not quite sure what you're asking. Jupyter uses terminado - you can see the code here and in a couple of adjacent files. So the more similar to terminado we can make the API of the Windows equivalent, the easier it is for Jupyter to integrate.

We've just discussed it in our weekly meeting, and we're happy to have the module in the Jupyter Github organisation, whether it's a separate Windows thing or we combine it with terminado. So it's not going to be abandonware once it gets written. :-)

[andfoy]

Ok, I'll help you with this. Once you create the new repository, ping me to migrate the server :-)

[takluyver]

I've given it a boring descriptive name for now - we can rename it if we come up with something smarter: https://github.com/jupyter/win-tornado-terminals

[Schmetzler]

Wow, very cool... I just recently stumbled across this project (, added my 2 cents) and it seems to get windows support soon... I am very impressed how quick this is going on.

Thanks to you guys ... looking forward to the solution (and maybe I can add something to it too... though I am not a windows user [just need to use it on work])

[dhirschfeld]

Since work towards this end is happening in the win-tornado-terminals project I'll close this as a wontfix for terminado.

Thanks all for your help in finding a way forward for Windows terminals in Jupyter!