wq python: _python_function_command may use incorrect python executable

[btovar]

When using PythonTask, we wrap the function calls with:

 command = 'python {wrapper} {function} {args} {out}'.format(...

That python executable comes from the $PATH, and may be a different executable version from the one running the wq manager. If both manger and task have the same environment (say from the same conda tarball), this not an issue. This issue appeared in tests with centos7, where a wq manager runs with python3, but the tasks wrappers use python, which is a python2.

[dthain]

Advice from Python seems not to help... https://peps.python.org/pep-0394/

[dthain]

It seems like we have two different ways to address this:

• Use python3 explicitly, and revisit the issue when python moves to 4.0.
• Continue using python and take additional steps to make sure that the the worker is running in an appropriate environment.
• Something else?

[btovar]

Ensuring the appropriate environment would be an opportunity to use poncho. However, it takes minutes to build an environment and I don't know if we want to slow the tests for that.

If we use python3, then we would need to disable PythonTask for python2. But I'm fine with that, as any application still using python2 is not using the newer PythonTask.

We could append the 2 or 3 to the python command when constructing the wrapper. (We do something like that for the poncho spec in topcoffea). That would work also for python4. I'm inclining to do this option.

[dthain]

Oh, I see, this command line is constructed by Python code, right? Why not just sense the python version on the manager side, and use that to construct the right command line?