python3 port and more

[kmohrf]

Hi,

thank you for neat little project! I almost gave up hope that I would find some kind of voice recognition software that works offline and here it is!

anyway: I’ve forked sopare and started working on a python3 port. You’ll find the repository here.

I already got it running on my Raspberry Pi 3 though it keeps eating RAM and CPU until it dies. There are a few other changes I’ve made and that I would like to do:

Done:

• Basic port
• PEP8 and flake8 compliance

ToDo:

• Split configuration from library code (there’s a lot of application code (terminal output, plots, configuration, etc) intertwined with what I would consider library code (audio recording and processing)
• replace getopt with argparse
• replace deprecated imp module with importlib
• merge some modules
• there was some interesting test code in test :). I think you should look into python -m unittest discover. Generally speaking: there’s too much local state in the test classes and some of the tests feel more like application runtime checks and configuration generation. I’d love to get the tests back working so I can feel safe making other changes.
• add debian packaging

Anyway… I hope this is not overwhelming.

If you’d like we could use this issue for further communication on the state of my changes or I can open up a pull request right away.

[bishoph]

With pleasure :)

Please note development goes into the testing branch and only tested and working code gets branched to the master. I commit only to the master when the code was tested on a test system for several days or weeks, depending on the changes. Long story short: I recommend that you create a new branch for your changes (like python3-dev or alike).

On top of this: I aim for generic Python support. This means SOPARE must run with Python 2 or Python 3. As the target platform is the Raspberry PI Python 2 is currently very important. Please take this in mind when doing changes, testing and creating pull requests.

It seems that we have some different coding styles (like brackets around if conditions) so expect some refactoring when you submit a pull request :)

In terms of test code: Most of the comments on YouTube, GitHub and my own blog are about SOPARE config and mic issues around sound devices. To minimize my effort and get users a useful tool, some of the test cases were written to provide helpful information and to deal with (unrelated) ALSA config stuff.

However, I'm happy that you like SOPARE and you are willing to help. Thx and happy coding!

[kmohrf]

Hi,

I understand your concern for python2 though I’m not really sure if it’s worth the effort as support for it will end in just over a year at the end of 2019. Maybe it’s a good idea if I backport (or forwardport? :smile:) your changes to python3 and keep it updated as a separate branch like you suggested. What do you think?

Regarding the code style: I was hoping that you would adopt those changes as the code is now PEP8 compliant and can easily be validated with flake8. I’ve been on the same road with some of my projects, where I didn’t like the standard formatting someone else defined, but in the end I always adopted it at some point because almost everyone else was compliant with the standard. Just my 2 cents.

Finally about the test code: I didn’t mean to suggest, that the code for testing mic issues or sound devices should be removed. I just think a unit test is not the right place, because those shouldn’t be platform specific. And it kind of makes it hard to automate them or use them to root out bugs during refactoring where I only want to know if I just broke the FFT implementation or stuff like that. For instance: There is a test for the number of CPU cores on the host. It seems to be more of a runtime constraint, that sopare should check in its loop or training mode (I’m not even sure it’s a reasonable constraint because a single-core system will run multi-process or multi-thread code just fine, so it should probably be more of a warning instead?). Do you know where I’m getting at (I’ve been up too long last night…)?

[kmohrf]

By the way: matplotlib and numpy both pledged to drop python2 support by the end of 2019. But I guess Debian will support python 2 for another 3 years with the upcoming buster release next year.

[bishoph]

Let's be as clear as possible: this project has scope and goals. One is the Raspberry PI support. On top of this SOPARE runs for myself on three systems 24/7/365. One system is a Raspberry PI 2 with Jessie. It's important that the software runs without much hassle as intended. Don't get me wrong: Python 3 is the future but AFAIK the default Python version (even for Stretch) is Python 2 on Raspbian. This means dropping Python 2 completely is a "no go" at the current stage.

In terms of brackets for the if conditions: the examples in the PEP8 docu has those brackets like I tend to use them in my code :)

And finally: yes, even a single core system runs SOPARE. But not very well if you use it for production and under load. This is why the requirements exists.

[sumpfralle]

It is great to see progress towards Python 3!

[..] the default Python version (even for Stretch) is Python 2 [..]

Yes, the maintainer teams of the Python modules and applications in Debian decided that /usr/bin/python will point at Python 2 practically forever (or at least after one release without Python 2 was shipped - this means probably after 2025). Thus your statement is not wrong (/usr/bin/python -> Python 2), but it ignores the fact that Python 3 is the heavily recommended Python interpreter everywhere. For example Debian does not allow any new packages that support only Python 2.

Python 3 is available on all distributions for about ten years now. Thus I am not aware of a good reason for keeping a compatibility layer with this end-of-life interpreter.

Please boldly go ahead with Python 3. Thank you for maintaining SOPARE!

[bishoph]

Ok, thank you for the head up and the information. Python 3 is fine. I would like to define the requirement that the development for Python 3 takes place in a separate repository. Once it works and there are no major steps involved in building, compiling and installing required libraries (specially on the Raspberry PI) we can do the merge. Will cross check that in a couple of days as I'm currently on the road.

[HerrAugust]

Is porting to Python 3 (on Raspberry 3) complete?

[bishoph]

No, it's still an ongoing task and an open issue.

[sumpfralle]

@kmohrf: maybe you want to update the changes in your branch and create a merge request in order to foster a discussion of specific details (if necessary)?

[dumblob]

@kmohrf any intention to continue your work?

[kmohrf]

@dumblob yes, but it’s more or less at the end of a very long list of other things I need to get done. Feel free to use my work and use it for whatever you like.

[TobiasReich]

Hey hello, any news about the Python3 port? I hope it is still coming. Sounds exciting but I can't bring it to work with python3.

[sumpfralle]

any news about the Python3 port?

I am not sure, what the current state is, but I would summarize it as follows:

• 87 mentions, that python3 works (but seems to require manual adjustments)

• @kmohrf proposed a set of changes (see https://github.com/kmohrf/sopare), but progress stalled due to requirements for keeping python2-compatibility (that was back in 2018 - before Python2 went end-of-life in January 2020)
  • if anyone is interested in integrating @kmohrf's preparations, then I would propose to fork @kmohrf's repository and turn it into a pull request (probably requires rebase and adjustments) against the testing branch of bishoph's sopare repository

Please correct me, if I misunderstood anything.

[bishoph]

My plan is to start on this issue within the next days and to leverage the testing branch for the Python 3 port. My thoughts from November 2018 are still relevant as the latest Raspbian (Buster) python command is still linked to Python 2. In addition, an update of any running SOPARE version on any operating system should work seamlessly and smoothly. The outcome could be a new Python 3 branch or a common version that runs either with Python 2 or 3 without user interaction.

[sumpfralle]

My plan is to start on this issue within the next days and to leverage the testing branch for the Python 3 port.

This sounds great - thank you!

My thoughts from November 2018 are still relevant as the latest Raspbian (Buster) python command is still linked to Python 2.

I have the feeling, that you misunderstand the relevance of /usr/bin/python - it does not refer to the default python interpreter, it is just a placeholder for keeping backward compatibility. The approach of Debian (and its derivatives) towards /usr/bin/python is well described in the description of the python-is-python2 package. Python2 will still be provided in the upcoming Debian Bullseye release, but only intended for the few packages, which still require it for its build process, it is not intended for running software.

Thus I think, you can move forward boldly and switch to python3 without regarding python2 support, since this interpreter is end-of-life now for almost a year. I am sure, this will save a bit of development effort on your side. Keep up your good work!

[bishoph]

Yeah, I understand fully the end of life scenario. Thus said, I care about users. Like, if I have a tool that's no longer been build and supported I still use it for years. AFAIK the lite versions of Raspbian come with only Python 2 support and require to install Python 3 manually. Lot's of users have no clue about the difference about Python 2 or 3 and don't care at all. Not everyone is a developer and no everyone is installing and using the latest stuff. I simply don't want to "break" SOPARE for the normal user if not absolutely necessary as I know such a "move" comes with the costs of support I have to give for free in my free time...

[bishoph]

Made the commits to the testing branch with support for both Python 2 and 3. Tested Python 3 on the latest Buster release and Python 2 on an old Wheezy release that still runs in my basement without internet connection. So far I did not encounter any issues but the devil could be in the details so please report issues if you find some.

Many thx to kmohrf at this point, even if I did not use your pull request you are the one who should be praised for the initial work.

SOPARE version with Python 3 support >= 1.5.6

[kmohrf]

thanks for the shoutout and nice to see sopare is officially supported on py3 now :blush: i’m looking forward to try it out myself

[bishoph]

As we say in Germany: "Ehre wem Ehre gebührt" :)

Have fun!