Python 3 support?

[aginiewicz]

Hello, I'm looking for a replacement for PyFirmata, the only library I use that is requiring Python 2.x I use. I'd like to move to full Python 3 workflow, so I looked for other projects to use StandardFirmata on Arduino from Python. I found PyMata and it looked promising, but now it also looks like Python 2 only library.

Any plans/timeline to update to Python 3? Anything stops the migration?

[MrYsLab]

Thanks for the request. Yours is the third request for Python 3 this month, so it is time for me to start working on this again. I am finishing some other projects and should be able to start looking at this in about 2 to 3 weeks. The last time I looked at porting to Python 3, it wasn't trivial, so I can't give you a timeline as to when it will be complete. Aside from the port, the testing will take time, as well as reworking and testing the example.

I may also add some callback capabilities for data reporting. Is this something that would be of use to you?

[aginiewicz]

Thanks for looking into it. Callbacks sound interesting - everything that helps getting data out and creating or optimizing live plots (with pyplot.ion or alternative) or makes integration code shorter, is of interest to me - as getting data is the thing I do - I actually teach a metrology and data acquisition course using Arduino, so all such things are nice.

[MrYsLab]

Great! I will add callbacks to the list.

[MrYsLab]

@aginiewicz Here is my release plan:

1. Port code to Python 3.
2. Update both the 2 & 3 versions for callbacks
3. As Firmata Configurable matures, see if a plugin scheme can be developed for PyMata to support new devices.

Question for you Andrzej on the callbacks. What type of granularity are you looking for? Do you envision a callback for each updated event as it happens on each pin/device? I am thinking that perhaps a mechanism could be implemented to specify which pins would be reported as part of a callback. For those pins not selected, the data would still automatically be updated as it is now, and the user can then poll when they wish. Any thoughts?

Also, would an MQTT interface be useful to you? I am not very familiar with MQTT as of yet, but perhaps this could be an additional mechanism for specifying callback type granularity. I am going to investigate after I get the porting done.

[aginiewicz]

About MQTT, personally I do not use Ethernet yet - we have 42 Arduinos working and had no cash for Ethernet Shields for them left as we preferred to get more sensors - so we communicate mostly over USB/Serial - it would be interesting to have, but not a priority for me - next big extension of lab setup would be in late 2015 or 2016.

About callback - being able to tell which pins to report is indeed nice option - so you plan something like:

registerCallback({"A0", "13"}, closure)

would be the way to go, where on callback it would be called with

closure(A0=value, 13=value)

But - what if we find change on 13 only - should we then poll the other pins for their values? Or - if we do some sensor which have 3 outputs and we will get 3 callbacks very close to each other - do we somehow filter those two when not all dimensions were updated yet?

While being able to specify more than one pin/device would be nice, it might be quite complicated to get right in generic way. It might need to provide a configurable behaviour for different scenarios.

I believe, that for now, simple interface will be enough, and this could be an advanced feature in next version - for now, just being able to assign callback to single pin, would be enough, and let the user of library deal with logic. Interesting use cases might come up in field use of this.

[MrYsLab]

@aginiewicz I just emailed you a beta copy of PyMata for Python 3.4. Please let me know if you have any questions or problems. Also, please let me know if you are successful using it as well.

I have callbacks working for digital, analog, and encoder pins. I need to test for Sonar and i2c.

The callback parameter is a list The first element of the list is the device mode, digital input, analog input, encoder, sonar or i2c data. The second is the pin number (for i2c devices it will be i2c device address). The third is the updated data value. Callbacks are only called if a value changes so as to not flood the client. Hopefully I will publish all of this sometime this week.

[aginiewicz]

Thanks, I will check it out on Monday!

[MrYsLab]

Great!.

By the way, I now have callbacks working and I hope to publish the whole package soon. A callback can be added to report changes on analog input, digital input, i2c input, encoders, and for digital and analog latches. Adding a callback to a latch, will essentially give you a one-shot. For example, if you set an analog latch to fire when the value for the pin is greater than 500, when that value is achieved, the callback is invoked and the latch is cleared. It can easily be rearmed anywhere in the application program, including the callback function.

Call backs will return the input type, pin number and value. For i2c, instead of pin number the device address is returned. For the latching callbacks, a time stamp of when the event occurred is provided as well.

You can have a single callback function to receive all callbacks, or you can set the granularity to be one callback function per pin.

Want to get rid of the callback, just call set_pin_mode without the callback function specified and the call back is disabled.

[MrYsLab]

@aginiewicz I have been testing with Python 3 and it looks like there may be some serious problems, in that PyMata is reporting data back very slowly. I plan to put a profiler on the code. I have no evidence, but I suspect it is either in PySerial (probably not) or the way I am handling threads. I am using the same test scripts for both Py 2 and 3, and when I run Py 2, performance is excellent. I hope this is an easy fix once I find out where all the time is going ;-).

There is good news - callbacks are working very nicely. Of course with Py 3 I still have a slow down, but at least the callback mechanism has proved out to be working. Again with Py2 things work as expected.

[MrYsLab]

@aginiewicz I ran a profile and it appears that the slow down is in the pyserial library when using python 3. I am submitting a trouble ticket to the author and hopefully there will be a fix. So as a result, even though the PyMata code is now Python2 and Python3 compatible, I won't be releasing it as python3 compatible.

I will be releasing the callback changes for python 2 and when the pyserial problems are resolved, I will re-release for python3.

[aginiewicz]

Thanks for looking into the issue. Can you share a ticket number with this report? I'd be interested in following that ticket :)

[MrYsLab]

@aginiewicz Here is the link to the ticket on sourceforge https://sourceforge.net/p/pyserial/bugs/179/. There are many issues that have gone unanswered, so let's hope I hear back from the author.

[MrYsLab]

@aginiewicz Some additional information - I just installed on Windows 8.1 using python 3.4.2 and I don't see the delays I see in Linux. This is definitely a pyserial problem specific to Linux. If you are using Windows, when I publish the latest files, you should be able to run on Python 3.4.2 without issue.

[MrYsLab]

@aginiewicz I think I found a solution for the Linux slowdown. There is a non-blocking Pyserial option specifically for Linux that seems to work. I will test it out this coming week, and if it all goes well, will update the Pyserial ticket to indicate my fix and will close the ticket. Next is publishing the PyMata code for Python 3.

I think it is strange that the behavior for python 2 and 3 are so different, but the Python folks must have changed the internal threading quite a bit (just a guess on my part).

[MrYsLab]

@aginiewicz I just left my final post for the author of pyserial. Here is what I wrote: "Final update. I will be leaving this ticket open. It appears that Serial.write when executed in Python 3.4 is approximately 10% slower than when executed in Python 2.7. The Serial.write is spent in both the os level write and os level select. For Python 3 the amount of time being spent in select is huge compared to its Python 2 counterpart."

The bottom line is I think using callback provides acceptable performance on 3.4, but a polling mode is unusable.

I left a spreadsheet with profiling data on sourceforge for the author. I don't know if Python 3 in combination with linux is the problem or if pyserial is handling i/o incorrectly for Python 3.

In any case, I will forge ahead with publishing the code and make sure there is a big warning in the readme about using PyMata in a polling mode with Python3. If nothing else, my profiling has shown the efficiency advantages of using callbacks.

[MrYsLab]

This article http://www.reddit.com/r/Python/comments/272bao/python_34_slow_compared_to_27_whats_your_mileage/ supports the argument that Python 3.4 is considerably slower on linux than 2.7. I updated the trouble ticket on sourceforge to have the ticket closed, since I no longer believe that Pyserial is the cause of the slowdown. It appears that Windows does not exhibit the slowdown.

[aginiewicz]

I've runned their tests and indeed, there is small difference, but not as big as some of reported in that post. For me it is like 0.52 vs 0.65 second for python 2 and 3 respectively, maybe it improved slightly in 3.4.2 compared to 3.4.1 which was tested there. On which one you tried this?

edit: ah, I forgot - you mentioned it was 3.4.2 in the sf.net ticket, sorry :) It also matches the "about" 10% you mentined. So yeah, seems it is CPython implementation fault.

I wonder - have you tried it with PyPy? I wonder if it is by design or by implementation for Python 3.x.

[MrYsLab]

I am seeing a 20% increase in the time it takes to do a pyserial write. I am using 3.4.2. In PyMata callback mode, the increase is barely noticeable by eye (I am turning a potentiometer to adjust the brightness of an LED attached to a PWM pin).

However, in polled mode where I continually perform a pyserial write in a tight loop, the delays are seconds in length, and this is unusable. Running the same code in Windows with 3.4.2, there is no noticeable delay

Running the same tight loop code in python 2.7 on Linux , there are no noticeable delays. From my profiling, the amount of time 3.4.2 is spending in a linux select call during writes is very large compared to 2.7.8.

So my conclusion is, if you need to run PyMata on 3.4 use callback mode. If you need it in a polled type environment, try it out. Depending upon the application it may or may not work well.

It appears that the delays in Python 3.4 are application dependent. It took me a while to find the link that discusses the delays, so I am assuming that for most applications, these delays are not noticed. I am hoping that someone on the Python team acknowledges the problems in Linux and fixes them in a later release.

If I can figure out where to report the problem to the Python team I will issue a trouble report.

[MrYsLab]

I just sent a message to the python core-membership team including the results my running cProfile. Let's see if I get better response than I did from pyserial. ;-).

[MrYsLab]

Python 2.01 with python 3 support has been released so I am closing this issue.