litex broken under python3.5

[bunnie]

The use of f-string formatting in on this line breaks litex under python 3.5.2:

https://github.com/enjoy-digital/litex/blob/56aa7897df99d7ad68ea537ab096c3abdc683666/litex/build/lattice/programmer.py#L35

This is from commit 351551a041f737b7b0ac08132a5b356fa8be468b

It looks like f-strings were introduced in python 3.6+

Was it the intent to break environments running with python 3.5? That is, what is official minimum recommended level for python for litex to work.

If the officially supported python is a level higher than python3.5 I'll upgrade my system and re-do the build scripts to check for this compatibility issue. However, I'd rather not be constantly chasing the upgrade curve on python 3 to incorporate the latest features for no reason. Especially considering I don't even use this feature, but it's a build-breaker because of the syntax error it causes.

[mithro]

Python 3.5 was released in 2015-09-13 and the current Python version is 3.8 -- If you are upgrading, jump straight to 3.8

Currently tested versions are 3.6, 3.7 and 3.8 -> https://travis-ci.com/github/enjoy-digital/litex

[bunnie]

So is it official policy that python 3.6>= is required for Litex?

I don't think "old" is a reason on its own to deprecate support. Ideally it would be a reason like, "this feature is crucial to rock solid code" or "there was a security problem and the upstream maintainers aren't fixing it so we've deprecated this version (e.g. like Python2)".

Example: NeTV2 was started in 2016 and is now in maintenance mode. At that time, Python 3.5 was only a year old. I still have to maintain and build the NeTV2 base, and I'd like a reproducible build environment.

Upgrading Python on my development instance may also introduce bugs into NeTV2 and should strictly require a full regression. This is especially true because we had problems with a sensitivity to the ordering of dictionary keys back then, and those orderings change with the Python version.

So while it's an easy matter to upgrade Python for a project in active development, it's a big problem for projects in maintenance mode because of the fragility of Python code. Thus I would prefer we deprecate a Python version only if there's a really good reason to do so...

[mithro]

@bunnie I'm not sure I understand? For projects not in active development why would LiteX be changing?

FYI - It is always recommended you use isolated environments + installs (not based on your system) for each project you have -- this is specially for maintenance mode projects.

[bunnie]

Litex isn't changing. The version of Python3 on the system is.

I've managed to deal with the changing of Litex constantly. What I didn't expect is that Litex would change in a way that would require me to move to a new version of Python3.

[mithro]

@bunnie I'm still confused -- If you are not changing the LiteX version you are using then how did you get the new change which removes Python 3.5 support?

[enjoy-digital]

@bunnie: the main issue with supporting Python3.5 is that developers are now familiar with Python3.6+ features and use them in their contributions. I'm already trying to avoid latest 3.7, 3.8 features and have to refactor some PRs to stick to 3.6; so it's also some efforts for the project to support older Python version. We are currently only testing with 3.6, 3.7 and 3.8 and with the 2020.04 release done today, the minimum tested and recommended is 3.6. It's not planned to switch to 3.7, 3.8 very soon, but if this make things easier for you, that's things we could plan in advance.

For an "old" project you maintain, you should indeed use the version of LiteX that was used at that time + eventual bug/features patches you would do manually, that also means using the same Python version if you strictly want the project to be build in the same conditions, but in this case as @mithro is suggesting, i would recommend setting up a VM or isolated environment that you would just boot to work on this (and this would also ensure others parts have not changed: GCC compiler, etc...)

[mithro]

I'm wondering if we are dropping 3.5 if we should also drop 3.6 and 3.7 at the same time?

My feeling is that for most people moving from Python 3.5 to Python 3.6 is about as much work as moving from Python 3.5 to Python 3.8?

Python 3.8 is a pretty nice release and has quite a few nice features like much better enum, dataclasses and things like that.

[bunnie]

@bunnie I'm still confused -- If you are not changing the LiteX version you are using then how did you get the new change which removes Python 3.5 support?

I have a VM running Ubuntu 16.04 (for legacy Vivado reasons) that uses Python3.5. I built NeTV2 originally in this VM. Also in this VM I'm building Betrusted devices.

The version of Litex for NeTV2 is pinned (this is a feature of lxbuildenv, the tooling that xobs made) separately from the Betrusted stuff. Basically we have a "deps" directory associated with every Litex project that allows us to pin repos to various commits so we can have better reproducibility and we can "freeze" repos.

However, I am using the Python3 interpreter in the system to run the individual builds, and that's 3.5 due to the complicated chain of dependencies going back to an old version of Vivado I'm keeping around to reproduce some builds.

So if Betrusted latest litex all of a sudden needs 3.6 I have to upgrade the system python to that which affects the NeTV2 stuff.

I guess on top of all of this, I'm out of hard drive space on my dev environment so a separate Python env for every project is also not practical for me simply from the perspective of I don't have the gigabytes of free hard drive space to carry that overhead for every project. I've been taking measures to try and fix that constraint, but basically, notching up a Python rev is not a variable I had considered I'd ever had to deal with when setting up my work flow.

[bunnie]

@bunnie: the main issue with supporting Python3.5 is that developers are now familiar with Python3.6+ features and use them in their contributions. I'm already trying to avoid latest 3.7, 3.8 features and have to refactor some PRs to stick to 3.6; so it's also some efforts for the project to support older Python version. We are currently only testing with 3.6, 3.7 and 3.8 and with the 2020.04 release done today, the minimum tested and recommended is 3.6. It's not planned to switch to 3.7, 3.8 very soon, but if this make things easier for you, that's things we could plan in advance.

For an "old" project you maintain, you should indeed use the version of LiteX that was used at that time + eventual bug/features patches you would do manually, that also means using the same Python version if you strictly want the project to be build in the same conditions, but in this case as @mithro is suggesting, i would recommend setting up a VM or isolated environment that you would just boot to work on this (and this would also ensure others parts have not changed: GCC compiler, etc...)

Right, I get that now. I will take measures in the future to snapshot an environment in a separate VM once a product ships. Unfortunately because I knew so little about how unmaintainable Python can be when I started NeTV2, I made all the mistakes and that build is extremely fragile. I originally had this idea that I'd have a VM where all the Litex stuff lives, but it's now turning into a VM-per-product. So, I need to block out a couple solid days to calve that off and do a full regression and pin things, I'll try to schedule that time in the next couple months. I understand clearly your need now to move off of older versions of Python, but I also hope you understand my concern about the risks of upgrading the interpreter.

I see that your README stipulates Python 3.6+. If you plan to change the minimum requirement in the future, I would suggest giving some warning. Please make sure whatever minimum version you pick in the future is at least supported on the active LTS releases for Ubuntu -- this is in part because Vivado trails the Ubuntu release cycle by a good margin in terms of official OS support, and that provides some downward pressure on supporting older releases.

Understanding that the Python version can and will move changes my approach to how I structure my dev (also, it blows my mind that there's a serious programming language that breaks compatibility every release; that's an eye-opener for sure). If you can think of any other core assumptions that I should consider when building a 10-year support plan for my hardware, I'd love to hear them.

[mithro]

@bunnie - I think you might be starting to understand why litex-buildenv uses conda to isolate the Python / LiteX / compilers / everything from the based system.

[bunnie]

Conda has other problems...but yah, it's basically a whole separate VM for every build, one way or the other. :-/ Who has this kind of disk space...

[enjoy-digital]

I think the risk of upgrading the interpreter has to be evaluated the same as upgrading Vivado, GCC, or any other tool involved in the process of generating the gateware or software. Your reasoning with using a VM for LiteX project seems valid, but the fact that you switch to a VM per product to avoid a Python interpreter change seems a bit extreme. It also has to be evaluated, and as you can see, we are able to test LiteX on 3.6 to 3.8 without troubles, so things are already a bit de-risked.

We are still a small team and we are trying to satisfy all requests the best we can, so don't have the capability to do LTS releases as Ubuntu can do and ensure things will not changes in the next 5 years, so we try to be practical. There is you case on one side, there are also others cases, like for example Minerva that requires Python3.6 for nMigen, so since we clearly haven't seen regressions while testing LiteX with Python3.6, the practical choice is to upgrade the dependency, but i can understand your concern and frustration.

So what we can do for now is making sure we only use Python3.6 specific features only when it's required and allow you to continue using Python3.5 for the next few months to organize things on your side now that you are aware of this. Is if fine for you?

But disk space is cheap nowadays and i think as soon as a gateware project is no longer maintained actively, a backup of the environment should probably be done to be able to maintain it in the future (i remember seeing in the industry FPGA bitstreams being patched manually because no one did a backup of the environment and it was no longer possible to build it...).

[mithro]

@bunnie - https://www.amazon.com/dp/B078DPCY3T/ref=twister_B07P5VF5TL?_encoding=UTF8&th=1 -- 1tb is $150 USD

@enjoy-digital -- Having the CI testing the Python versions we support is the best way for us to make sure that it doesn't regress (which is how we actually discovered that we had already broken Python 3.5 support). Trying to improve the test coverage will help here too.

@bunnie - Have more of the things you do covered by test cases in this repository also helps make sure we don't accidentally break things you depend on.

[enjoy-digital]

With https://github.com/enjoy-digital/litex/pull/488 i added back Python3.5 testing in Travis-CI. I'm ok doing efforts trying to keep things compatible with Python3.5 until we have a good reason to no do so, but i would still recommend using Python3.6+ for new users so will keep this in the README/setup.py. In the future, we'll make sure to discuss/warn users about these changes, but users should also be aware that minimum Python requirement could change to benefit from last version of the tools. I think we all learned from the discussion, thanks for the feedback.

[bunnie]

We learned from the NeTV2 experience and everything on Betrusted is now in CI and we can keep up with the environment change, so we are agile enough there.

The 3.5 breakage is mainly a concern for NeTV2. Calving that off to a VM so I can build its bitstreams ten years from now is probably the right way to go. I just thought we had it figured out when we decoupled the dependency of Litex version from the build.

Actually I thought about it last night and given that I'm facing a VM-per-project methodology, I think I can just clone my current VM and put a copy in the freezer and call that the NeTV2 "eternal" environment.

This means you can go ahead and deprecate 3.5 right away.

FWIW: the cost of space isn't just the cost of the cheapest hard drive if you take backups seriously. If you want to be able to build a bitstream image 10 years from now for a given product, the data still has to be there, as well as the computer to interpret it. I have now taken to archiving whole /laptops/ because few OSes can run binaries from 10 years ago without problems.

If you think I'm just being crazy, it's been almost 15 years since we shipped the first Chumby, and I still get support requests for it today. It is a real challenge to support products that weren't designed to become obsolete.

[mithro]

@bunnie - For very long term things like this, I think a full VM is your only real option. For example, I can actually still boot MSDOS 2.0 in a VM today.

@bunnie - I think you should also think about a story around "handing over maintenance to the community". With this model, as long as a device has a community around it, then it continues to get updates and once that community disappears, the updates stop happening. This is sometimes called the "evergreen" model. This is my current plan around the Fomu and related devices.

[bunnie]

@mithro - I'd love it if the community wanted to maintain the NeTV2. I'm definitely taking PRs.

However, I think it would be lame if I didn't at least have a support plan that went beyond "sorry, I can't actually test a patch because I broke my original environment keeping the rest of my toolchain up to date. Maybe someone else can help?"

Also, NeTV2 is different in that the primary customer for it tends to be not developers but ... people who think of it as an appliance. Like a pub owner who installs it for March madness overlays. The ideal behavior for NeTV2 for them is it's like a magical video cable. They pay someone to tweak settings until everything works, and then forget it's there for several years until the TV gets upgraded.

TV formats are really fussy, and it turns out "improvements" can break bug-compatibility with a customer setup a non-trivial portion of the time. I recently had one customer contact me because an update that added auto-res switching broke his setup because he was using a weird resolution mode that I thought nobody used. And the matrix of weird buggy STBs and TVs to test against is pretty big and I'd have to spend a lot of $$ and lab space to put that into a CI framework.

Thus pushing patches to NeTV2 that don't fix a material bug or provide a material feature improvement (like oh, this is the same thing, but built with Python 3.6!) is often not perceived as good support by customers that aren't also developers. Unfortunately regressions are very common in Litex updates -- I'm currently puzzling over new glitches in the Betrusted EC that happened with the latest Litex update.

Finally, one of the challenges of supporting an expensive and complicated product like NeTV2 is that if a community patch bricks a unit, the person who made the patch just goes "oops. Well that's a learning experience." Meanwhile I'm out a serious amount of capital mailing replacements or running damage control with an enraged customer.

Fomu has the advantage that it's cheap enough that if you bricked one for a given customer's use case, you can trivially offer a refund or a replacement; and most users for it are developers, if I'm not mistaken. I'm not rich enough and NeTV2 is not a big enough business to be able to afford that kind of policy. And NeTV2 is expensive enough that people aren't happy if it's broken. I doubt we as a community could agree on a policy of "your patch bricked it, you get to replace it". I mean, this would greatly improve the quality of every patch submitted, but then why would anyone ever submit a patch.

If you had a specific idea for an approach to an "evergreen" strategy that also could accommodate these considerations, I'd love to hear it. But basically my thought is -- there's an "official" NeTV2 build which is the one that the hardware is guaranteed to support. If the hardware doesn't run that build, you can email me and we can talk about an exchange or replacement. It's a slow-moving build with minimal features that's very thoroughly tested and supported for the lifetime of the product.

Then there's the "evergreen community build" (e.g. HDMI2USB, etc.) which you're welcome to run and maybe someday it will become the preferred version because it has so many awesome features, but if updates to that build bricks the hardware or ruins your use case, I can't support that with an exchange or refund.

[mithro]

@bunnie - Having both a;

• a "officially supported" NeTV2 firmware. That is generated from a fixed VM and never upgrades LiteX.
• a "community supported" NeTV2 firmware which follows the evergreen policy.

And then declaring "AlphaMax only provides supports for the reduced functionality official build" and "For support around the community build please see these forums" seems pretty reasonable to me.

I think it should be expected that most people running the "officially supported" NeTV2 firmware would only do a firmware update unless they were hitting a specific bug which had a fix.

[bunnie]

OK, sounds like we're basically on the same page there. I guess is there an HDMI2USB or other project build that's mature enough at this point for me to start pointing people at? I saw a bunch of commits toward that recently but unclear what the current status is.