Asyncio hangs on freebsd

[meskarune]

I wrote a tcp listener that works perfectly on linux but when I try and ctrl-c in freebsd it hangs and I have to kill the process id to get the program to shut down. The code is below:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import asyncio

class announce():
    """ Return message uppercase """
    def __init__(self):
        """ Start the tcp server """
        host = "localhost"
        port = 2000
        loop = asyncio.get_event_loop()
        coro = asyncio.start_server(self.TCPRequestHandler, host, port, loop=loop)
        server = loop.run_until_complete(coro)
        try:
            loop.run_forever()
        except:
            server.close()
        loop.run_until_complete(server.wait_closed())
        loop.close()
        print("started tcp listener")

    @asyncio.coroutine
    def TCPRequestHandler(self, reader, writer):
        data = yield from reader.read(100)
        message = data.decode("utf-8")
        addr = writer.get_extra_info('peername')
        self.msg("Received {0} from {1}".format(message,addr))
        writer.write(data.upper())
        yield from writer.drain()
        writer.close()

    def msg(self, message):
        print (message)

def main():
    announce()

if __name__ == "__main__":
    main()

I am on the lastest version of freebsd, 10.3 and I ran that in a python3 virtual enviroment.

[Martiusweb]

Can you tell where it hangs exactly? Is KeyboardInterrupt raised and bubbles up to the try/except, or does it hang in wait_closed()?

[meskarune]

I ran the program with python -vvv and here is what I got:

# extension module loaded from '/usr/home/meskarune/virtualenv/optimus-outdated/lib/python3.4/lib-dynload/unicodedata.so'
import 'stringprep' # <_frozen_importlib.SourceFileLoader object at 0x804c50cf8>
import 'encodings.idna' # <_frozen_importlib.SourceFileLoader object at 0x804c509b0>
^C

It doesn't appear that it saw the keyboard interrupt.

[Martiusweb]

I don't think the verbose mode would help much, nor PYTHONASYNCIODEBUG=1. Can you add print() statements?

try: loop.run_forever() except: print("Exception bubbled, asking sever to terminate") server.close()

...

    print("waiting on wait_closed()")
    loop.run_until_complete(server.wait_closed())
    print("waited on wait_closed()")

I'm not familiar with freebsd, we need to know if it's KeyboardError that is hijacked, or wait_closed() that hangs because it's expecting a notification from the selector which never come.

[meskarune]

Linux:

started tcp listener
^CException, asking sever to terminate
waiting on wait_closed()
waited on wait_closed()

Freebsd:

started tcp listener
^C

Then hangs forever until I kill it.

edit, here is the full code that I used on both:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import asyncio

class announce():
    """ Return message uppercase """
    def __init__(self):
        """ Start the tcp server """
        host = "localhost"
        port = 2000
        loop = asyncio.get_event_loop()
        coro = asyncio.start_server(self.TCPRequestHandler, host, port, loop=loop)
        server = loop.run_until_complete(coro)
        try:
            print("started tcp listener")
            loop.run_forever()
        except:
            print("Exception, asking sever to terminate")
            server.close()
        print("waiting on wait_closed()")
        loop.run_until_complete(server.wait_closed())
        print("waited on wait_closed()")
        loop.close()

    @asyncio.coroutine
    def TCPRequestHandler(self, reader, writer):
        data = yield from reader.read(100)
        message = data.decode("utf-8")
        addr = writer.get_extra_info('peername')
        self.msg("Received {0} from {1}".format(message,addr))
        writer.write(data.upper())
        yield from writer.drain()
        writer.close()

    def msg(self, message):
        print (message)

def main():
    announce()

if __name__ == "__main__":
    main()

[gvanrossum]

This tracker isn't really meant to help you debug issues in your own code. I won't close the issue since you apparently are having a reasonable conversation but I recommend finding a forum for help, e.g. the python-tulip list.

[meskarune]

Uh...I got this code from your own documentation (there was an example of a tcp server) and none of the examples on there work on freebsd for me. If there is something in the code that is wrong I would love to know what it is I am doing, and then I will be filing a bug report for you to fix your own documentation.

[gvanrossum]

Okay... (The heritage of your code wasn't clear to me from reading your description, I didn't recognize the example from the docs.)

When a program hangs on ^C it's usually stuck in a non-daemon thread that holds a lock.

Asyncio in general doesn't handle ^C well; it's a direct subclass of BaseException and that's not always handled properly. I think we have an issue open for handling BaseException consistently, maybe there's even a PR, but we're kind of worried that it would make things worse for other error conditions.

[meskarune]

Oh ok. I mean I am just trying to be helpful. It's weird/unexpected that the same code works on Linux but not on Freebsd using the same python version. If there is some further information I can give to help I will be happy to provide it here or in another bug report on BaseException.

[gvanrossum]

Welcome to the wonderful world of asynchronous programming and distinct network stacks. :-)

If you can get to the bottom of this please do report back here. Unfortunately I cannot really help.

[1st1]

Uh...I got this code from your own documentation (there was an example of a tcp server)

Could you please point out where in the docs you found this code?

[meskarune]

https://docs.python.org/3/library/asyncio-protocol.html

I did throw things into a class, but the exact same behavior happens with every example in that page that I tried. I'd like to make sure this isn't something wonky with my setup though so I'm testing on other freebsd servers to check and talking with the freebsd guys on irc. So far I've tried on 2 servers and its not working as you'd expect.

[1st1]

Just to confirm, could you please run [1] unmodified and see if you can't ^C out of it?

[1] https://docs.python.org/3/library/asyncio-protocol.html#tcp-echo-server-protocol

[meskarune]

I think I've figured it out. It works on freebsd outside a python virtual environment but not inside one. I tried on linux in a virtual environment and its also working there. So I guess you can close and I'll try to continue figuring out what the heck is happening.

[1st1]

Also, could you please try to run it with uvloop? It doesn't use Python IO, and wondering if it exhibits the same behaviour or not...

[meskarune]

inside my virtual enviroment:

pip install uvloop 
Collecting uvloop
  Downloading uvloop-0.4.33.tar.gz (1.9MB)
    100% |████████████████████████████████| 1.9MB 260kB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module> 
      File "/tmp/pip-build-exocvb8g/uvloop/setup.py", line 11, in <module> 
        raise RuntimeError('uvloop requires Python 3.5 or greater') 
    RuntimeError: uvloop requires Python 3.5 or greater

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-exocvb8g/uvloop/

I can get a python3.5 virtual env on freebsd and test again.

[kaniini]

hi, can you provide:

• uname -a
• either pkg info python34 or pkg info python35 output

from your freebsd machine?

this is strange, i am running asyncio on hundreds of freebsd machines and your example is working for me.

[kaniini]

oh, I just read you are using virtualenv in a later post? can you verify the virtualenv is using python 3.4 or 3.5? the virtualenv and pip commands are normally bound to the python27 package, you may need to explicitly request the 3.4/3.5 virtualenv.

[meskarune]

uname -a

FreeBSD juno 10.3-RELEASE-p4 FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64

python --version outside of virtualenv: Python 2.7.11

Inside virtualenv: Python 3.4.4

I have both python2 and python3 installed, but in freebsd python2 is the default.

I created a new virtualenv on another server with "virtualenv -p python3 asyncio" and again ran into the same problem. Outside the virtualenv it works. :/

[kaniini]

Can you run the process under truss please?

[1st1]

Should this issue be closed?

[1st1]

Closing this.