search

nodejs / build

Better build and test infra for Node.
506 stars 166 forks source link

Add Python 3 to more build machines #1674

Closed cclauss closed 4 years ago

cclauss commented 5 years ago

Continue the process started in #1644 of putting both Python 2 and Python 3 on various build machines so that we can perform tests and experimental builds on the Python 3. https://github.com/nodejs/node/pull/25759#issuecomment-458146087

/R = @rvagg

cclauss commented 5 years ago

On those build machines where Python 3 is already installed, we need to do: python3 -m pip install --upgrade pipenv # See test failures on nodejs/node#25878

refack commented 5 years ago

Need to add to ansible recipes (https://github.com/nodejs/build/search?p=3&q=python&type=&utf8=%E2%9C%93) Might be a good chance to refactor common prerequisites.

cclauss commented 5 years ago

Can we please get this resolves and closed? 303 daze left...

rvagg commented 5 years ago

some help getting ansible working properly with python3 would be great, I can't run it on my mac anymore for machines that need ccache and have to resort to linux where it uses python2. The line causing problems is https://github.com/nodejs/build/blob/master/ansible/roles/baselayout/tasks/ccache.yml#L13 and there's a similar one in ansible/roles/jenkins-worker/tasks/partials/docker-host.yml. Something about jinja2 processing of "{{ ccache_html_content.stdout | regex_findall('ccache-[0-9]+.[0-9]+(?:.[0-9]+)*.tar.gz') | map('regex_replace', 'ccache-') | map('regex_replace', '.tar.gz') | list | latest_version }}" the chaining of the maps seems to be the problem and I don't have the error at hand. If needed I could probably generate the error again.

refack commented 5 years ago

@rvagg does https://github.com/nodejs/build/pull/1399 help?

(BTW this has the new "npm" like pipenv work flow, git fetch, pipenv install, pipenv shell)

cclauss commented 5 years ago

Should we close this one or is this still an issue?

refack commented 5 years ago

We still need Python3 on more machines.

cclauss commented 5 years ago

Can we track progress in a table similar to the one below?

Operating System Architectures Versions Python 2 Python 3
GNU/Linux x64 kernel >= 3.10, glibc >= 2.17 Tests pass Tests pass
GNU/Linux x64 kernel >= 3.10, musl >= 1.1.19 Tests pass Tests fail
GNU/Linux x86 kernel >= 3.10, glibc >= 2.17 Tests pass Not installed
GNU/Linux arm64 kernel >= 4.5, glibc >= 2.17 Tests pass Not installed
GNU/Linux armv7 kernel >= 4.14, glibc >= 2.24 Tests pass Not installed
GNU/Linux armv6 kernel >= 4.14, glibc >= 2.24 Tests pass Not installed
GNU/Linux ppc64le >=power8 kernel >= 3.13.0, glibc >= 2.19 Tests pass Not installed
GNU/Linux s390x kernel >= 3.10.0, glibc >= 2.17 Tests pass Not installed
Windows x64, x86 (WoW64) >= Windows 7/2008 R2/2012 R2 Tests pass Not installed
Windows x86 (native) >= Windows 7/2008 R2/2012 R2 Tests pass Not installed
Windows arm64 >= Windows 10 Tests pass Not installed
macOS x64 >= 10.12 Tests pass Not installed
SmartOS x64 >= 18 Tests pass Not installed
AIX ppc64be >=power7 >= 7.1 TL05 Tests pass Not installed
FreeBSD x64 >= 11 Tests pass Not installed
sam-github commented 5 years ago

Is https://github.com/nodejs/build/issues/1674#issuecomment-459875569 needed on the AIX machines? If so, after updating them all to have python3 available, https://github.com/nodejs/build/issues/1775#issuecomment-492803274, I could do it.

sam-github commented 5 years ago
sam-github commented 5 years ago

I ran https://github.com/nodejs/build/pull/1800 on test-osuosl-centos7-ppc64_le-1, if it all still looks good tomorrow, I'll update the others.

[centos@test-osuosl-centos7-ppc64--le-1 ~]$ ls -c1 /usr/bin/python*
/usr/bin/python3.6
/usr/bin/python3.6m
/usr/bin/python3
/usr/bin/python36
/usr/bin/python2.7
/usr/bin/python
/usr/bin/python2
[centos@test-osuosl-centos7-ppc64--le-1 ~]$ python3 --version
Python 3.6.8
sam-github commented 5 years ago

I've spent more than an hour and a half looking for s390x RPMs for a python3, no luck yet.

No s390x in:

I'm finding some suggestions that if I use subscription-manager, I might be able to get it. I'm not exactly sure what we would have legal entitlement to (or if it would even work), and we are not currently registered for any subscriptions.

I'm not very familiar with RH-derived distros, I might be missing something obvious.

@nodejs/python or @nodejs/platform-s390, suggestions?

mhdawson commented 5 years ago

@sam-github I suggest you talk to @miladfarca to see if he can get some ideas of who we might ask about python3 for the linuxOne machines.

sam-github commented 5 years ago

Updated second centos-ppc64_le:

% ssh test-osuosl-centos7-ppc64_le-2 python3 --version                                                                              
Python 3.6.8
cclauss commented 5 years ago

What is the Python 2 version on those machines?

We want python2 >= 2.7.9

sam-github commented 5 years ago

@cclauss see https://github.com/nodejs/build/issues/1674#issuecomment-493206073, I'm updating it

sam-github commented 5 years ago

Sorry, misread your comment, you want to know about python2, not python3. @cclauss I'll go look.

sam-github commented 5 years ago

@cclauss 

% ssh test-osuosl-ubuntu1404-ppc64_be-1 python2 --version
Python 2.7.6
% ssh test-osuosl-ubuntu1404-ppc64_le-1 python2 --version
Python 2.7.6
% ssh test-osuosl-aix61-ppc64_be-1 python2 --version
Python 2.7.11
% ssh test-osuosl-centos7-ppc64_le-1 python2 --version
Python 2.7.5
sam-github commented 5 years ago

@cclauss I don't understand why we need to update python2 to a supported version if we have python3. If it was easy, I'd just do it, but its not, so I am pushing back a bit.

Why should we update Python2 if a supported version of Python3 exists? Especially given python2 is EOL on Jan 1, 2020?

cclauss commented 5 years ago

Let's focus first and foremost of getting a Python >= 3.5 on these machines (and in the $PATH). If it was up to me, I would just take out legacy Python completely and step into the future. However, Node.js is not yet compatible with Python 3. This exercise is to set up the tests on Python 3 but it is not yet clear if those tests will pass and build a workable Node.js.

Therefor, Python 2 will be a part of Node.js beyond the 1/1/20 date so given that we will be depending on an unsupported codebase, we would like to uplevel the Python 2 to be >= 2.7.9 to encourage our users to avoid the most egregious attack vectors. If it is too much work to fix legacy Python in the near term, lets focus on upgrading Python as you suggest.

Thanks much for all your hard work.

sam-github commented 5 years ago

see: https://github.com/nodejs/build/pull/1807

sam-github commented 5 years ago

Folks here, and @nodejs/python , we need more clarity on the expectation for python3. At least, I do.

Python3 has to be >= 3.5.0.

When system packages are available, I guess it will be in the system path, as python3.

Is this the intended setup, in other words, is it a requirement that python3 be in the default PATH?

And is this a requirement: https://github.com/nodejs/build/issues/1674#issuecomment-459875569 ?

On test-osuosl-centos7-ppc64--le-1:

$ ./python3 -m pip install --upgrade pipenv
/opt/python-3.7/bin/python3: No module named pip

I don't know much about pip, and nothing about pipenv, so it would help if we had a relatively clear statement of what was required.

Possibly we want node itself to build with python3, if that is even possible. Is there some way I can test that manually? If I have a succinct recipe for what I should do manually on a machine once python3 is installed on it to verify the python3 environment fulfills expectations, that would be helpful.

EDIT: also see https://github.com/nodejs/build/pull/1807#issuecomment-495757049

cclauss commented 5 years ago

pip is the package installer for Python. It is the essential bootstrap for installing packages. Without pip you can not install pyenv or any other Python packages so we need pip. You can try python3 -m ensurepip to install it but the preference would be an install which already includes pip.

python3 --version
python3 -m pip --version
thefourtheye commented 5 years ago

Is this the intended setup, in other words, is it a requirement that python3 be in the default PATH?

If Python 3 is not there in the default PATH, we may have to update our scripts to detect Python 3 as well, I am afraid. Installing it at /usr/local should not create any problems, right?

Possibly we want node itself to build with python3, if that is even possible. Is there some way I can test that manually?

The ultimate goal of this activity is to build and test Node.js with both Python 3 and Python 2.

If I have a succinct recipe for what I should do manually on a machine once python3 is installed on it to verify the python3 environment fulfills expectations, that would be helpful.

As far as I am concerned, we just need Python 3 in the default PATH. pip is optional/good-to-have in my opinion, as we do not install many additional packages to test (I could be wrong here, I haven't checked all the scripts we have). @cclauss Do you think pip is necessary for us to proceed at this point?

cclauss commented 5 years ago

A Python install without pip is a painful thing so we should avoid it if we can.

https://github.com/nodejs/node/blob/master/Makefile#L1316 and below use pip to install flake8 but linting on all platforms is not required.

sam-github commented 5 years ago

For anyone finding their python3 is lacking pip, or ctypes, libffi and zlib (the -dev versions of these) might be missing.

cclauss commented 5 years ago

See the note above (https://github.com/nodejs/build/issues/1674#issuecomment-495783378) about ensurepip.

sam-github commented 5 years ago

@cclauss Perhaps I misunderstood that comment, don't you suggest not using ensurepip? Are you suggesting we should use it? Or suggesting that its an alternative to installing python3's dev dependencies before configuring and building python3?

cclauss commented 5 years ago

Are we done on this one? Can we start testing Python 3.6 on one of the Jenkins test machines? Python 3.7 is pending nodejs/node#29326 but Python 3.6 should be fair game to start testing.

github-actions[bot] commented 4 years ago

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.