select() always immediately returns EndpointOUTfiles as readable

[ali1234]

Relevant code: https://github.com/ali1234/pymtpd/blob/master/mtp/function.py#L109

I am trying to use select() because threads are a nightmare, and I need to handle inotify as well as USB communications.

I have a collection of files to poll: ep0, which works fine. The inotify fds, which also work fine. And outep, which is ep2, and is where the MTP operations arrive.

Problem is that when I select() on these fds, outep is always returned in the list of ready fds, and then outep.read() hangs until an actual message arrives.

I realise this probably isn't caused by your python code, but do you have any idea what is going on? Is this expected behavior of functionfs?

[ali1234]

Oh, I didn't push the inotify code yet. But the same thing happens without it.

[vpelletier]

I remember something about ep0 being select-able, but all other endpoints requiring asyncIO. There are a few examples in the kernel source tool directory. I am not familiar with libaio and I think I fell back on threads to not have to tackle that.

Googling a bit, I find a ton of stuff called "aio" for python, but that looks unrelated to libaio. I guess some ctypes magic may be needed to interface with libaio, and then exposing it somehow on the endpoint file objects.

[ali1234]

Thanks, I will try to figure out this asyncIO stuff.

The specific problem I have with threads is that if a thread is blocked in ep.read() and then the main thread crashes (eg due to a typo) then the main thread will catch the exception, but can't tear down the gadget because the child thread still has the functionfs locked. At best this means you have to delete a load of stuff manually, at worst it can completely deadlock the process because each thread is waiting on the other, and they both are locked inside the kernel. That ends in a hung task and a forced reboot.

[vpelletier]

I would expect disabling the gadget to make blocking reads return (python raising IOError, with exc.errno == errno.ESHUTDOWN).

Though I do not know if it is practical to do it in teardown path... So far, python-functionfs steers clear from stuff requiring root, as I want to use functionfs' ability to be used by unprivileged users to increase security. As a consequence, I did not try to imagine how to automate the parts I described in shell in tests/README.rst .

[ali1234]

I am trying the libaio approach, based on aio_simple.c. I think this should make a select()able transparent file-like object from an out endpoint, if I can just get ctypes to cooperate:

class KAIOReader(object):
    def __init__(self, file):
        self.file = file
        self.filefd = file if type(file) == int else file.fileno()
        self.evfd = _libc_call(_libc.eventfd, 0, 0)
        logger.debug('eventfd = %d' % self.evfd)

        # TODO: should be a struct
        self.ctx = ctypes.c_buffer(0, 1000)
        self.buf = ctypes.c_buffer(0, 512)
        _libaio_call(_libaio.io_setup, 1, ctypes.pointer(self.ctx))

        self.submit()

    def fileno(self):
        return self.evfd

    def submit(self):
        iocb = IOCB64()
        # (inline) io_prep_pread(iocb, self.filefd, pointer(self.buf), len(self.buf), 0);
        iocb.aio_filedes = self.filefd
        iocb.aio_lio_opcode = 0 #IO_CMD_READ
        iocb.aio_reqprio = 0
        iocb.u.c.buf = ctypes.cast(ctypes.pointer(self.buf), ctypes.c_void_p)
        iocb.u.c.nbytes = len(self.buf)
        iocb.u.c.offset = 0

        iocb.u.c.flags |= 1 #IOCB_FLAG_RESFD
        iocb.u.c.resfd = self.evfd

        _libaio_call(_libaio.io_submit, ctypes.pointer(self.ctx), 1, ctypes.pointer(ctypes.pointer(iocb)));

        logger.debug('submit read')

    def read(self, n):
        logger.debug('got event')
        os.read(self.evfd, 8)
        logger.debug(self.buf)
        e = IOEvent64()
        ret = _libaio_call(_libaio.io_getevents, ctypes.pointer(self.ctx), 1, 1, ctypes.pointer(ctypes.pointer(e)), None)
        if ret == 1 and e.obj.aio_fildes == self.filefd:

            tmp = bytes(self.buf[:e.res])
            logger.debug('Read %d bytes' % e.res)
        else:
            raise Exception("Failed")
        self.submit()
        return tmp

[ali1234]

I managed to get it working in the end:

https://github.com/ali1234/pymtpd/commit/376702c3fc8ff15032375da9364f6016841e356e

[vpelletier]

I implemented a wrapper for libaio.

I got a simple test case working, although it's not using eventfd.

Let's see if I can with functionfs...

[vpelletier]

Ah, and I just got your latest post. I was too slow :) .

[ali1234]

Nice, your wrapper is a lot better than mine.

For functionfs you need to watch out for a few things:

1. You can't io_submit until after binding the gadget or it will block forever. But you can't bind the gadget until after the endpoints are open. So you can't do the first io_submit at the same time as you open the endpoint - it has to be done separately later.

2. Unlike close(fd), io_destroy can only be called once or it returns an error. This can be a problem because functionfs calls close() twice - once in exit and once in del.

[ali1234]

I ported my KAIOReader to your ctypes wrapper (using the low level calls) and it works:

https://github.com/ali1234/pymtpd/blob/master/mtp/kaio.py

[vpelletier]

Thanks a lot for testing and reporting. Happy you could integate it and get shorter code.

About io_destroy call time, what about calling when ep0 gets closed ? Given the issue you brought up about threads and blocking io, I start to think eventfd could be provided by default by python-functionfs. I still have to figure out a natural api though.

Sadly I will not have much time this weekend to work on it.

[ali1234]

Not sure about how to expose it best, but I implemented a writing wrapper around my interrupt endpoint:

https://github.com/ali1234/pymtpd/blob/master/mtp/kaio.py#L95

This implements an in-kernel write queue. I'm not sure it is strictly necessary but it is possible I write two events in a row and the inquirer blocks on an inquiry after receiving the first one, so this allows me to send a hundred or so before I have to deal with an inquiry.

I do wonder what is he correct way to marshall the buffers that go into the io_submit. I just let them go out of scope at the moment and it feels wrong.

[vpelletier]

I do wonder what is he correct way to marshall the buffers that go into the io_submit. I just let them go out of scope at the moment and it feels wrong.

I believe it is fine as long as you do not try to use io_event.obj: I think kernel copies the content of iocb and buf on submission, but when you get the completion event the userland memory will likely have been garbage-collected by python.

In the pythonic wrapper layer in python-libaio, I keep more references alive on for this reason.

Also, I just pushed a crude netcat-on-usb example, though not using a kernel queue for writes.

[vpelletier]

I implemented a writer in latest example/usbcat version.

I'm still not very satisfied with the high-level libaio wrapper, so I can't recommend moving to it for now.

Also, I'm hitting a few kernel issues, which does not help.

[vpelletier]

I believe this issue can be closed: python-libaio is released, python-functionfs can work with it, and there is example code on how to work with event-driven IN and OUT endpoint files.