Default is Python 2, but would be nicer if Python 3

[matthewfeickert]

All the Python files in pandamonium start with

#!/usr/bin/env python

which is defaulting to Python 2 for the runtime. However, Python 2 is dead (long live Python 2) and it would be nicer if Python 3 was the default runtime

#!/usr/bin/env python3

This seems like it will be a sticky issue though as many people in the HEP world are still running Python 2 as most of the software on clusters is still defaulting to that and this shows no signs of stopping soon (sadly). @dguest What is a way forward here to provide Python 3 support without breaking pandamonium on physics clusters stuck on Python 2?

[matthewfeickert]

cc @kratsg @cshimmin who may have additional thoughts here.

[dguest]

Just on a historical note: For a while I supported both python 2 and python 3 branches, but all the merge requests came from people using python 2. I don't know if that's still relevant.

What is relevant is that, as you said, python 2 is going to be the default in ATLAS physics analysis for at least another few years. Defaulting to python 3 is going to confuse a lot of already confused people, and they are the people this library is meant to help.

Normally I'd feel sort of icky about using python 2 "because that's what people use": python 2 clearly needs to die, I don't like writing python 2 code, and if this code was don't anything remotely complicated or had any dependencies I'd be making life needlessly hard by using something so deprecated. But python 2 is not dead yet, this code doesn't need much maintenance, and this code was only initially intended as a demo for how awesome CLI and REST APIs are. I hope anyone doing serious work beyond checking their job status just guts this code and incorporates the parts they need into some bigger framework.

That said, it would be unfortunate if someone couldn't use this code because they only had python 3. Has this happened yet? Even worse would be if someone decided not to update to python 2 because of this script, but that seems even less likely.

I think the turning point will come when python 3 is more common in ATLAS than python 2. Any bets as to when that happens?

[matthewfeickert]

That said, it would be unfortunate if someone couldn't use this code because they only had python 3. Has this happened yet?

Not yet. A lot of this was also fueled by my assumption that this could be a local machine utility (c.f. Issue #24), and becomes far less relevant if it is regulated to being a LXPLUS utility. If local machine utility happens though, then this will become an issue soon as Ubuntu and new versions of macOS will start shipping with Python 3 only (actually that already happened with Ubuntu 20.04).

I hope anyone doing serious work beyond checking their job status just guts this code and incorporates the parts they need into some bigger framework.

This is somewhat fair, but I'm a huge fan of the little utility that does its job well and earns its place in the toolbox. So I'd like to see pandamonium be useable and contributed to for a long time.

I think the turning point will come when python 3 is more common in ATLAS than python 2. Any bets as to when that happens?

I really have no idea, only secret hopes. I think @lukasheinrich is the person who has the best guess at the moment, but I also don't think that he can even give a clear answer on that.

That being said, now that I've understood the issues of Issue #24 better I actually have a way to make the code fully Python 2 and Python 3 compatible that way very simple. I've opened up PR #25 for it once PR #22 is resolved (either yay or nay).

[afortiorama]

This seems like it will be a sticky issue though as many people in the HEP world are still running Python 2 as most of the software on clusters is still defaulting to that and this shows no signs of stopping soon (sadly).

the problem is that python2 is still the default on centos7 sites will not change the default. And there has been no request to do it. ATLAS (and the other experiments) will have to take care of it.

[matthewfeickert]

ATLAS (and the other experiments) will have to take care of it.

Thanks for this info, @afortiorama. It is always helpful to better understand where the work actually is. I also didn't mean to put the "blame" on anyone here, so hopefully it didn't come across as me condemning the good compute site people. :)

[afortiorama]

@matthewfeickert not at all. But it is a problem that has been discussed in ADC for almost 2 years and while ATLAS is slowly phasing out python2 it is still very much in use.

[matthewfeickert]

Note to self, when in the (far) future pandamonium drops Python 2.7 support, the bdist_wheel universal option in setup.cfg

https://github.com/dguest/pandamonium/blob/81b5d5929a729baa139f92e8f8d941f51478a6bf/setup.cfg#L30-L31

should be removed. c.f. https://github.com/scikit-hep/pyhf/pull/1295 for reference.

[rbarrue]

Coming back to this issue, on CentOS7 machines, /usr/bin/python contains the executable of Python2(.7.5), but on EL9 machines, /usr/bin/python contains nothing, even when setting up a Python version (although /usr/bin/python3 exists).

I can change the header of the pandamon script to make it run, but do we want to force it to python3 in the newer versions ? Or just put a comment on the README for people on EL9 machines to change the header ?

[kratsg]

I can change the header of the pandamon script to make it run, but do we want to force it to python3 in the newer versions ? Or just put a comment on the README for people on EL9 machines to change the header ?

which script are you using...? if you install via pip or uv, should be fine.

[rbarrue]

I can change the header of the pandamon script to make it run, but do we want to force it to python3 in the newer versions ? Or just put a comment on the README for people on EL9 machines to change the header ?

which script are you using...? if you install via pip or uv, should be fine.

I have (on LXPLUS) a clone of the repo with its folder added to PATH.

[kratsg]

I have (on LXPLUS) a clone of the repo with its folder added to PATH.

if you clone it, at least install it in editable mode python3 -m pip install -e /path/to/it and it will work as expected.