event-driven - python interface

[arrenglover]

It would be good to have some interface for sending events from/to python from c++ modules. YARP solves this using SWIG and bindings, however we had some problems creating the bindings for event-driven due to the use of shared pointers.

The other option is to write readers for events in python. However, that means manual update of the python is required to match any changes made in C++. This could lead to bugs :warning: :beetle:

I heard @Iaxama was working on something. What is the plan?

[Iaxama]

At the moment I have this script that uses the yarp bindings to grab the events into python. I also thought that having the bindings automatically generated with SWIG would be the best option but I am not an expert and I didn't want to mess up in the interface file without knowing exactly what I was supposed to do. However there is no direct dependency of the pyhton script on our libraries except maybe the event decoding which would need to be updated in case we change the meaning of the bits, but that is also something we try to update as seldom as possible, so it should not be that hard to maintain.

[marco-monforte]

The problem with SWIG was that it was not able to translate some particular C++ functions (like 'union').

I agree that a set of functions to read different types of events (a thing that we already have) and a set of functions to encode and then output events through a simple BufferedPort would be easy enough to maintain.

[arrenglover]

@Iaxama @marco-monforte is there any changes to this given the new python code integrated into the library recently? The new stuff is all offline, so I guess doesn't really impact this issue.

Alternatively, after some more experience should we do this in a different way? is it still needed?

[marco-monforte]

I don't know much about all the new stuff that Massi, Sim and maybe someone else did but, from my understanding, we now have everything we need to read/write/process events in python. As you said, this is all offline, but if we need something online it should be just a matter of taking pieces of code, so probably it is not needed anymore.

Let's wait for @Iaxama to reply, but I guess we can close this issue.

[arrenglover]

loading a file and processing it offline is quite different to reading a yarp port and decoding online, in terms of the interface and the fact that reading online you want to guarantee some level of speed. I see these two operations as quite different.

How are you currently interfacing the cameras to your python models for inference?

[marco-monforte]

When the inference module was running in python, I was reading data like for a normal port with data.get(0).asDouble(). As for the spatial coordinates, I was reading directly the pixel values from your tracker; the timestamp, instead, was converted to ms using the usual formula with 2^30 and 80e-9.

Before you had this "pixel value writing", however, I was reading the encoded value as above with get() and then using a "decode" function to extract the single groups of bits, hence coordinate values, from it. This decode function was the same used for offline processing.

Currently, the model is in C++.

[arrenglover]

Okay so in the end you can use C++ ports to read the data. 👍

Do you see any big problems with reading the data live directly into python? Can we only read AE? or read other types of events? Do you know if we have any problems with real-time reading of events? Maybe @Iaxama will have to let us know too.

[marco-monforte]

I didn't test everything thoroughly, but I'd say that the only problem could be the real-time reading when the amount of events is large. I was reading from the tracker, so a single event per iteration. There were some events dropped but because the inference loop was taking some milliseconds. Hopefully, Massi will be more precise on this, but I also recall seeing some tests and the python visualization was pretty nice.

About other types of events, I'd say there is no problem. What would change is only the decode function to extract more info, like the Sigma for Gaussian AEs.

[Iaxama]

So at the moment I have the script that i have mentioned a few comments above which is at an experimental stage but that works reasonably fast. It only deals with AE at the moment. However it requires a small patch (one added line) in the yarp bindings interface, and I didn't want to mess up with pull requests to yarp without making sure that there are no better ways of doing things. I attach the script here if you want to take a look at it. If you think is worthwhile merging it somehow I will spend some time integrating things nicely with our python libraries and yarp bindings.

decolle_experimental_event_grabber_yarp_grabber.zip

[arrenglover]

if you think that in the future we are going to have projects that require live importing of the event-stream in python I think it is worthwhile to build on this work and integrate it.

Alternatively, if there is better way of handling this data - e.g. it's better practice to move everything to c++ to ensure faster operation for real-time - then we should focus on that direction instead.

what do you both think?

[Iaxama]

I think that anything that requires learning will be developed 99% of the times in python and, given the current trend, i think that many will need to work on learning stuff in the future. Even though real time operations would still be better handled in c++, before moving to the c++ efficient implementation, testing and development will still be done in python and even though it might be better practice to move everything to c++ for real time implementations, i think is useful having a way to do testing on the python side beforehand.

[marco-monforte]

I think that a helper script to read/write events in python would be nice to have anyway, up to you if you prefer a grabber class as Massi did or just helper functions that the user can call to decode/encode what he's reading live.

C++ is better, that's sure, but it's not easy to integrate neural networks with it; and, depending on the task, one might not require such low latencies at the beginning and python would be an easy test bench.

[arrenglover]

@Iaxama while I agree, your answer is that having both things done will be better, which is of course true. However, we factor in that working on this more means not working on something else.

If C++ is required to do real-time inference - then I would rather have offline python scripts done well. and spend our time on-line making sure the c++ conversion is easy.

If C++ is not required to do real-time inference, then we don't really need it right? we should just keep everything in python. Is there any problem with this?

Have you tested full sensor going into python for inference live and on-line, yet?

[Iaxama]

It's hard to tell now whether C++ conversion of Python code is always feasible or useful at all. Even though generally speaking C++ is more efficient than Python, the machine learning frameworks are being developed and highly optimized on Python, so the inference might still be better left on Python. However, online event reading might be faster to do on C++, but there would still be a missing part of the pipeline that links that to Python. I see two options there: dusting off the event-driven bindings that are not being maintained in a while or using something like my script.		Bindings	Script
Pros	- Almost as efficient as C++ (check here) - Always up to date with latest C++ code	- Almost ready for deployment - Easier to debug
Cons	- Requires some extra work - Harder to debug	- Need to be maintained separately - In its current state not as efficient as C++

[arrenglover]

yarp::os::BufferedPort are already covered in the bindings of yarp itself. Therefore using a bufferedPort and then decoding in python is the easiest solution. However, I'm unsure if it will be fast enough for live streaming a sensor. I think we should test this to inform the decision.

the vReadPort uses a class called vPortInterface which allocates data directly into events - rather than first allocating a Bottle and then converting to events. That's why we got a speed-up. There is the possibility of understanding how to "bind" that class to also take advantage of it in Python.

There is no actual decode function of events in c++ there is just the "union" trick to read the bits of an int in the correct way to retrieve the correct values. Therefore I don't think we can "bind" the decode functions of the events. Given we already have to maintain python versions of the decoding of the events for bimvee it would make sense to use them for the decoding.

So from what I understand

1. we pursue on-line / live decoding of events from the camera into python
2. we do a test to see the maximum event rate we can achieve reading in this way
3. if we need to, we decide if we want to go with "bind" the vPortInterface, or move to a robust c++ solution for all on-line neural networks.

@Iaxama @marco-monforte - who is in the best position to do # 2.

[Iaxama]

I can do # 2

[Iaxama]

@arrenglover do you have a dataset with high event rate i can use to test?

[marco-monforte]

I don't have a raw events dataset locally, but I can help (like if you want to try on different laptops or similar).

What kind of test do you have in mind? Just reading continuously and see the number of unprocessed queues?

[Iaxama]

more like measuring the time to process a single event under different conditions. If that number is unchanged we have a good estimate of the maximum event rate achievable. Testing on different machines could be useful. I could even use docker to emulate machines with lower CPU.

[arrenglover]

I think looking at 1 event and extrapolating doesn't taking into the network dynamics (events are sent in packets). Using the yarpdataplayer also changes these dynamics as the packet size typically increases. I would only be happy by actually using a camera and showing we read them all in real-time.

[arrenglover]

at the moment I have a camera and aiko has a camera. so If everything is good to go, and there are instructions on how to, I can run a test.

[arrenglover]

@marco-monforte this isn't my highest priority, so I'm not sure when I'll get to test it. Keep it in mind if you are going to get back to the robot and have some hardware of your own you can test with instead.