nodejs / build

Better build and test infra for Node.
505 stars 165 forks source link

temporary access to build machines for thefourtheye #1775

Closed thefourtheye closed 4 years ago

thefourtheye commented 5 years ago

In order to have our Node.js core build system compatible with Python 2 and Python 3, we need to have both of them in our build machines as the first step.

We are trying to figure out a way to install Python3 in all our build machines. Few of our supported platforms do not have Python 3 in their respective package management systems. To figure out how to install it, I would like to have temporary access to our build machines.

PS: I tried local VMs to do this, but it is very time consuming and I don't have enough resources in my machine.

cc @nodejs/build @refack @cclauss


List of machines which need Python 3 or attention

Machine Python2 Python3 Notes
test-digitalocean-centos5-x86-1 2.6.8 None yum search python3 failed with M2Crypto.SSL.SSLError: tlsv1 alert protocol version
test-digitalocean-freebsd11-x64-1 2.7.16 3.6.8
test-digitalocean-ubuntu1204-x64-1 2.7.3 None apt-get -s install python3 shows python3.2 is available
test-digitalocean-ubuntu1204-x64-2 2.7.3 None apt-get -s install python3 shows python3.2 is available
test-joyent-smartos14-x64-1 2.7.9 None pkgin av \| grep -i python3 shows both 3.3 and 3.4 are available
test-joyent-smartos14-x64-2 2.7.9 None pkgin av \| grep -i python3 shows both 3.3 and 3.4 are available
test-joyent-smartos14-x86-1 2.7.9 None pkgin av \| grep -i python3 shows both 3.3 and 3.4 are available
test-joyent-smartos14-x86-2 2.7.9 None pkgin av \| grep -i python3 shows both 3.3 and 3.4 are available
test-joyent-smartos15-x64-1 2.7.11 None pkgin av \| grep -i python3 shows 3.3, 3.4, and 3.5 are available
test-joyent-smartos15-x64-2 2.7.11 None pkgin av \| grep -i python3 shows 3.3, 3.4, and 3.5 are available
test-joyent-smartos16-x64-1 2.7.11 None pkgin av \| grep -i python3 shows 3.3, 3.4, and 3.5 are available
test-joyent-smartos16-x64-2 2.7.11 None pkgin av \| grep -i python3 shows 3.3, 3.4, and 3.5 are available
test-joyent-smartos17-x64-1 2.7.14 None pkgin av \| grep -i python3 shows 3.4, 3.5, and 3.6 are available
test-joyent-smartos17-x64-2 2.7.14 None pkgin av \| grep -i python3 shows 3.4, 3.5, and 3.6 are available
test-linuxonecc-rhel72-s390x-1 2.7.5 None Might have to check SCL
test-linuxonecc-rhel72-s390x-2 2.7.5 None Might have to check SCL
test-linuxonecc-rhel72-s390x-3 2.7.5 None Might have to check SCL
test-macstadium-macos10.08-x64-1 Network is unreachable
test-macstadium-macos10.08-x64-2 Network is unreachable
test-macstadium-macos10.09-x64-1 Network is unreachable
test-macstadium-macos10.09-x64-2 Network is unreachable
test-macstadium-macos10.09-x64-2 Network is unreachable
test-macstadium-macos10.10-x64-1 Network is unreachable
test-macstadium-macos10.10-x64-2 Network is unreachable
test-macstadium-macos10.11-x64-1 Network is unreachable
test-macstadium-macos10.11-x64-2 Network is unreachable
test-macstadium-macos10.12-x64-1 Network is unreachable
test-macstadium-macos10.12-x64-2 Network is unreachable
test-marist-zos13-s390x-1 2.7.13 None
test-marist-zos13-s390x-2 2.7.13 None
test-mininodes-ubuntu1604-arm64_odroid_c2-1 Key didn't work
test-mininodes-ubuntu1604-arm64_odroid_c2-2 Key didn't work
test-mininodes-ubuntu1604-arm64_odroid_c2-3 Key didn't work
test-osuosl-aix61-ppc64_be-1 2.7.11 None Have to read up on smitty
test-osuosl-aix61-ppc64_be-2 2.7.11 None Have to read up on smitty
test-osuosl-aix61-ppc64_be-3 2.7.11 None Have to read up on smitty
test-packetnet-centos7-arm64-1 2.7.5 None yum search python3 shows Python 36
test-packetnet-centos7-arm64-2 2.7.5 None yum search python3 shows Python 36
test-rackspace-centos7-x64-1 2.7.5 None yum search python3 shows Python 36
test-rackspace-ubuntu1204-x64-1 2.7.3 None apt-cache search python3 shows Python 3.2 Debug build
test-softlayer-centos5-x64-1 2.4.3 None yum search python3 does not have anything
test-softlayer-centos5-x64-2 2.4.3 None yum search python3 does not have anything
test-softlayer-centos6-x64-1 2.6.6 None yum search python3 shows Python 36
test-softlayer-centos6-x64-2 2.6.6 None yum search python3 shows Python 36
test-softlayer-centos7-x64-1 2.7.5 None yum search python3 shows Python 36

Notes:

OS Commands Instructions to Install Python 3
centos-release-6-10.el6.centos.12.3.x86_64 1. rpm -q centos-release (Find OS version) 1. yum install centos-release-scl
2. yum install rh-python36
rvagg commented 5 years ago

@thefourtheye as a TSC member I don't think it'll be too controversial to just give you "test" access to our infra which should give you everything you need, there's a few others in this category already. You'll just need to be aware that:

I'll need to put a gpg key for you into https://github.com/nodejs-private/secrets so you can read build/test where our test ssh key is and some more instructions, so get me a public gpg key I can put in. You'll also need to either read ansible/inventory.yml and use those user@ip combinations directly or use write-config.yml (see https://github.com/nodejs/build/tree/master/ansible#playbooks) to set up your .ssh/config.

If anyone objects to going this route, please speak up!

thefourtheye commented 5 years ago

Thanks @rvagg for the instructions and guidelines. I'll generate a GPG key and share here.

cclauss commented 5 years ago

I am highly supportive of this effort related to https://github.com/nodejs/build/issues/1674#issuecomment-483954275.

mhdawson commented 5 years ago

+1

gireeshpunathil commented 5 years ago

+1

cclauss commented 5 years ago

Any update here?

thefourtheye commented 5 years ago

I had a few questions, so I checked with @rvagg last week. I have shared my public key with him today.

rvagg commented 5 years ago

@thefourtheye I've added your gpg key to our secrets repo, see the readme there for instructions on how to get access to the resources.

Review my comments to Sam when I added him last week, that's a good start: https://github.com/nodejs/build/pull/1787

You'll probably want to get your .ssh/config set up. As per the ansible/README.md, on the # begin: node.js template - # end: node.js template block to put in your .ssh/config. After that run: ansible-playbook playbooks/write-ssh-config.yml to populate it. I've had problems with this on my mac thanks to Python 3 so you're welcome to try and address any nits you find in the process! You should end up with an .ssh/config full of our machines and links to the ssh key you grab from the secrets repo.

sam-github commented 5 years ago

write-ssh-config.yml has worked fine for me, but I'm on linux, and /usr/bin/python is 2.7.

mhdawson commented 5 years ago

@gireeshpunathil, @sam-github could you look at how we get Python 3 onto the AIX machines and pull me in for context on what we did for other packages. I think it's best that we help out on that front.

mhdawson commented 5 years ago

@thefourtheye we've not used smitty in the past on AIX for adding modules. I think it would be best if @sam-github helped out on this front.

mhdawson commented 5 years ago

bull freeware is where we have pulled packages in the past, this is the first place to look: http://www.bullfreeware.com/search.php?package=python3.

It is pretty tricky though (AIX 7.1 is much better) so I'd prefer if we do it on one machine to start.

cclauss commented 5 years ago

https://www.python.org/download/other might be worth looking at also. (Or if it is out of date, contributing back to.)

thefourtheye commented 5 years ago

I managed to compile the list of all the loginable machines with the nodejs_build_test key and the versions of Pythons installed. The actual list can be seen here

cclauss commented 5 years ago

Are we agreed that:

If we install or upgrade a machine’s Python 3, then we will also upgrade its Python 2 if required.

Perhaps we need a fill color version of the spreadsheet: Legend Blue - Upgrade is possible Red - Upgrade is needed but how is unclear Orange - Release candidate version should be replaced

thefourtheye commented 5 years ago

As per Python's support information for different versions (which can be found here),

Branch Schedule Status First release End-of-life Release manager
master PEP 569 features 2019-10-20 2024-10 Łukasz Langa
3.7 PEP 537 bugfix 2018-06-27 2023-06-27 Ned Deily
3.6 PEP 494 security 2016-12-23 2021-12-23 Ned Deily
2.7 PEP 373 bugfix 2010-07-03 2020-01-01 Benjamin Peterson
3.5 PEP 478 security 2015-09-13 2020-09-13 Larry Hastings
3.4 PEP 429 end-of-life 2014-03-16 2019-03-19 Larry Hastings

I agree to all the three points.


As discussed over Slack, I gave write access to the document to you.

thefourtheye commented 5 years ago
rvagg commented 5 years ago

I see 3 problems: 1) freebsd11 machine is dead, not a surprise, it happens occasionally (freebsd rage post omitted), I'll address that. 2) the odroid c2 machines are no longer in use, they should have been removed from the inventory but obviously I missed that step. 3) something weird's up with your access to the macos machines. They use a separate port for SSH access, are you using it from your .ssh/config or manually connecting? I'm not sure what else this could be other than missing port on SSH but I'll have a look when I'm back at a computer (phone posting this one).

rvagg commented 5 years ago

Oh, and you can ignore some of the other machines too: centos5 should be removed from our infra. iirc it was redundant with node 4. Some of the other ones will be redundant with node 6 retiring I think and maybe we can skip them entirely anyway if we're not going to bother "fixing" node 8, and node 10? If we only want full python 3 support then the list of machines is much smaller. Centos6, ubuntu14 and some others all go away.

sam-github commented 5 years ago

I'm spelunking into the ansible files to see if I can figure out how python is installed on AIX. Finding anything in ansible is a nightmare, but I'll do my best. If I can figure that out, I'll see if I can find a python3 equivalent.

cclauss commented 5 years ago

Are these three Ubuntu ansible docker files of interest? #1679

sam-github commented 5 years ago

I installed python3 on test-osuosl-aix61-ppc64_be-2. See https://github.com/nodejs/build/pull/1799. It is the only thing installed into `/opt/bin', so should not disturb existing ci, but out of caution I did only the one machine. If builds are fine in a day or so, I'll do the other ci machines.

Also, I didn't use the documented steps exactly, because I don't have permissions to upload the archive to ci.nodejs.org/downloads, I just downloaded it directly.

# ls -c1 /opt/bin
2to3
2to3-3.7
easy_install-3.7
idle3
idle3.7
pip3
pip3.7
pydoc3
pydoc3.7
python3
python3-config
python3.7
python3.7-config
python3.7m-config
pyvenv
pyvenv-3.7
sam-github commented 5 years ago

https://ci.nodejs.org/computer/test-osuosl-aix61-ppc64_be-2/builds <--- builds look fine, installing python3 on the other two test machines.

rvagg commented 5 years ago

I've forced a restart of the freebsd11 machine and it's still not letting me in, it might be reprovision time for that one

macstadium is odd now that I'm looking at it, I can't get in either so there's something up with our setup there, I'll open a new issue for that

rvagg commented 5 years ago

test-digitalocean-freebsd11-x64-1 is back up again

thefourtheye commented 5 years ago

Thanks @rvagg. I updated the status of test-digitalocean-freebsd11-x64-1 in the table and the spreadsheet.


something weird's up with your access to the macos machines. They use a separate port for SSH access, are you using it from your .ssh/config or manually connecting?

I tried connecting with

➜ ssh -oPort=10009 administrator@test-macstadium-macos10.08-x64-1
ssh: connect to host 207.254.58.162 port 10009: Network is unreachable

I am able to ping that machine though

PING 207.254.58.162 (207.254.58.162): 56 data bytes
64 bytes from 207.254.58.162: icmp_seq=0 ttl=45 time=224.519 ms
64 bytes from 207.254.58.162: icmp_seq=1 ttl=45 time=224.027 ms
64 bytes from 207.254.58.162: icmp_seq=2 ttl=45 time=223.146 ms
64 bytes from 207.254.58.162: icmp_seq=3 ttl=45 time=224.018 ms
64 bytes from 207.254.58.162: icmp_seq=4 ttl=45 time=224.586 ms

But telnet fails. So, the port is not open I guess. Perhaps the SSH Demon is not running?

➜ telnet 207.254.58.162 10009
Trying 207.254.58.162...
telnet: connect to address 207.254.58.162: Network is unreachable
telnet: Unable to connect to remote host

macstadium is odd now that I'm looking at it, I can't get in either so there's something up with our setup there, I'll open a new issue for that

Thank you :-)

rvagg commented 5 years ago

@refack @mhdawson can either of you address the macos concerns here please?

mhdawson commented 5 years ago

See https://github.com/nodejs/build/issues/1810#event-2370170066 it was just an issue with the specific machine (and one other) that you tried to ssh to.

They are behind a host that forwards on to the actual machines. I just as the required port to my ssh config file and all works fine

cclauss commented 5 years ago

@mhdawson Can you please tell us what happens when you do “brew info python” on one or more of these macOS machines? If homebrew is installed on them then these upgrades should be quite straightforward: https://docs.python-guide.org/starting/install3/osx/

rvagg commented 5 years ago
python: stable 3.7.3, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed

Changes to these machines are going to need to go through ansible rather than just being manual, so we maintain repeatability. It looks like brew is all set up in our Ansible configs so maybe a section in roles/baselayout/vars/main.yml for 'macos' to do the work is the way to go?

@nodejs/build anyone with access want to give this a go?

rvagg commented 5 years ago

Here's a full output ftr, it's not consistent:

$ parallel-ssh -i -h /tmp/macos.hosts '/usr/local/bin/brew info python | head -4'
[1] 13:58:34 [SUCCESS] release-macstadium-macos10.10-x64-1
python: stable 3.7.0, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
[2] 13:58:34 [SUCCESS] test-macstadium-macos10.10-x64-2
python: stable 3.7.0, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
[3] 13:58:34 [SUCCESS] test-macstadium-macos10.12-x64-2
python: stable 3.7.0 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/2.7.14 (3,569 files, 49.3MB)
[4] 13:58:34 [SUCCESS] release-macstadium-macos10.11-x64-1
python: stable 3.7.0 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
[5] 13:58:34 [SUCCESS] test-macstadium-macos10.12-x64-1
python: stable 3.7.0 (bottled), devel 3.7.1rc1, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/2.7.14 (3,569 files, 49.3MB)
[6] 13:58:34 [SUCCESS] test-macstadium-macos10.11-x64-1
python: stable 3.7.2, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/2.7.14 (3,570 files, 49.3MB)
[7] 13:58:34 [SUCCESS] test-macstadium-macos10.10-x64-1
python: stable 3.7.3, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
[8] 13:58:35 [SUCCESS] test-macstadium-macos10.11-x64-2
python: stable 3.7.2, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/2.7.14 (3,570 files, 49.3MB)
$ parallel-ssh -i -h /tmp/macos.hosts '/usr/local/bin/brew info python@2 | head -4'
[1] 13:58:03 [SUCCESS] release-macstadium-macos10.11-x64-1
python@2: stable 2.7.15 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
[2] 13:58:03 [SUCCESS] test-macstadium-macos10.10-x64-2
python@2: stable 2.7.15, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (7,512 files, 122.3MB)
[3] 13:58:03 [SUCCESS] test-macstadium-macos10.12-x64-2
python@2: stable 2.7.15 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (4,714 files, 83.4MB)
[4] 13:58:03 [SUCCESS] test-macstadium-macos10.12-x64-1
python@2: stable 2.7.15 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (4,713 files, 83.4MB)
[5] 13:58:04 [SUCCESS] test-macstadium-macos10.11-x64-1
python@2: stable 2.7.15, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (4,713 files, 83.5MB)
[6] 13:58:04 [SUCCESS] test-macstadium-macos10.11-x64-2
python@2: stable 2.7.15, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (4,714 files, 83.5MB)
[7] 13:58:04 [SUCCESS] release-macstadium-macos10.10-x64-1
python@2: stable 2.7.15, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (7,512 files, 122.3MB)
[8] 13:58:04 [SUCCESS] test-macstadium-macos10.10-x64-1
python@2: stable 2.7.16, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (7,512 files, 122.3MB)
cclauss commented 5 years ago

brew install python or brew install python@2 should force a brew update/upgrade to happen before the Python install so my bet is that all the boxes would end up with the same Python version(s).

cclauss commented 5 years ago

Is the list at the top of this PR up-to-date?

rvagg commented 5 years ago

update on the state of macos is in https://github.com/nodejs/build/pull/1836, it's taken a few days to wrangle it under control but it seems to be all consistent now with both versions installed but we need to be explicitly referencing python2 and python3 to be getting them.

cclauss commented 5 years ago

Is the status table at the top of this issue up to date? I am having trouble understanding our progression and #1802 which would have allowed shared editing of such a table is also not progressing. We need an up-to-date status table.

Our only repos that have known (lintable) Python 3 incompatibilities are: nodejs/ffi, nodejs/nan, nodejs/node-auto-test, nodejs/node-chakracore, nodejs/perfetto-nodejs-wip, nodejs/worker. Please let me know if any of these repos are mission critical so that I can focus on them.

If not, then now is the time for us to start testing in earnest Python 3. With 188 days left until Python 2 end of life, I would like us to be pushing hard to be ready. Your thoughts on how we can accelerate?

rvagg commented 5 years ago

@thefourtheye ping re status, can you help us move forward here?

@cclauss re the list of repos, none are critical path.

github-actions[bot] commented 4 years ago

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

Trott commented 4 years ago

We should probably remove the temporary access at this point unless @thefourtheye chimes in to let us know they're still working on this.

richardlau commented 4 years ago

We should probably remove the temporary access at this point unless @thefourtheye chimes in to let us know they're still working on this.

I've gone ahead and removed the gpg key for @thefourtheye. We can add him back if needed.