Closed thefourtheye closed 4 years ago
@thefourtheye as a TSC member I don't think it'll be too controversial to just give you "test" access to our infra which should give you everything you need, there's a few others in this category already. You'll just need to be aware that:
I'll need to put a gpg key for you into https://github.com/nodejs-private/secrets so you can read build/test where our test ssh key is and some more instructions, so get me a public gpg key I can put in. You'll also need to either read ansible/inventory.yml and use those user@ip combinations directly or use write-config.yml (see https://github.com/nodejs/build/tree/master/ansible#playbooks) to set up your .ssh/config.
If anyone objects to going this route, please speak up!
Thanks @rvagg for the instructions and guidelines. I'll generate a GPG key and share here.
I am highly supportive of this effort related to https://github.com/nodejs/build/issues/1674#issuecomment-483954275.
+1
+1
Any update here?
I had a few questions, so I checked with @rvagg last week. I have shared my public key with him today.
@thefourtheye I've added your gpg key to our secrets repo, see the readme there for instructions on how to get access to the resources.
Review my comments to Sam when I added him last week, that's a good start: https://github.com/nodejs/build/pull/1787
You'll probably want to get your .ssh/config set up. As per the ansible/README.md, on the # begin: node.js template
- # end: node.js template
block to put in your .ssh/config. After that run: ansible-playbook playbooks/write-ssh-config.yml
to populate it. I've had problems with this on my mac thanks to Python 3 so you're welcome to try and address any nits you find in the process! You should end up with an .ssh/config full of our machines and links to the ssh key you grab from the secrets repo.
write-ssh-config.yml
has worked fine for me, but I'm on linux, and /usr/bin/python
is 2.7.
@gireeshpunathil, @sam-github could you look at how we get Python 3 onto the AIX machines and pull me in for context on what we did for other packages. I think it's best that we help out on that front.
@thefourtheye we've not used smitty in the past on AIX for adding modules. I think it would be best if @sam-github helped out on this front.
bull freeware is where we have pulled packages in the past, this is the first place to look: http://www.bullfreeware.com/search.php?package=python3.
It is pretty tricky though (AIX 7.1 is much better) so I'd prefer if we do it on one machine to start.
https://www.python.org/download/other might be worth looking at also. (Or if it is out of date, contributing back to.)
I managed to compile the list of all the loginable machines with the nodejs_build_test
key and the versions of Pythons installed. The actual list can be seen here
Are we agreed that:
If we install or upgrade a machine’s Python 3, then we will also upgrade its Python 2 if required.
Perhaps we need a fill color version of the spreadsheet: Legend Blue - Upgrade is possible Red - Upgrade is needed but how is unclear Orange - Release candidate version should be replaced
As per Python's support information for different versions (which can be found here),
Branch | Schedule | Status | First release | End-of-life | Release manager |
---|---|---|---|---|---|
master | PEP 569 | features | 2019-10-20 | 2024-10 | Łukasz Langa |
3.7 | PEP 537 | bugfix | 2018-06-27 | 2023-06-27 | Ned Deily |
3.6 | PEP 494 | security | 2016-12-23 | 2021-12-23 | Ned Deily |
2.7 | PEP 373 | bugfix | 2010-07-03 | 2020-01-01 | Benjamin Peterson |
3.5 | PEP 478 | security | 2015-09-13 | 2020-09-13 | Larry Hastings |
3.4 | PEP 429 | end-of-life | 2014-03-16 | 2019-03-19 | Larry Hastings |
I agree to all the three points.
As discussed over Slack, I gave write access to the document to you.
I could use some help with the Ansible stuff.
Also ping @nodejs/platform-aix and @sam-github (as per @mhdawson's suggestion) for AIX help.
@rvagg What shall we do about the machines which were not accessible. There is a table in the original post with machines which do not have Python 3.x also machines which are not accessible.
I see 3 problems: 1) freebsd11 machine is dead, not a surprise, it happens occasionally (freebsd rage post omitted), I'll address that. 2) the odroid c2 machines are no longer in use, they should have been removed from the inventory but obviously I missed that step. 3) something weird's up with your access to the macos machines. They use a separate port for SSH access, are you using it from your .ssh/config or manually connecting? I'm not sure what else this could be other than missing port on SSH but I'll have a look when I'm back at a computer (phone posting this one).
Oh, and you can ignore some of the other machines too: centos5 should be removed from our infra. iirc it was redundant with node 4. Some of the other ones will be redundant with node 6 retiring I think and maybe we can skip them entirely anyway if we're not going to bother "fixing" node 8, and node 10? If we only want full python 3 support then the list of machines is much smaller. Centos6, ubuntu14 and some others all go away.
I'm spelunking into the ansible files to see if I can figure out how python is installed on AIX. Finding anything in ansible is a nightmare, but I'll do my best. If I can figure that out, I'll see if I can find a python3 equivalent.
Are these three Ubuntu ansible docker files of interest? #1679
I installed python3 on test-osuosl-aix61-ppc64_be-2
. See https://github.com/nodejs/build/pull/1799. It is the only thing installed into `/opt/bin', so should not disturb existing ci, but out of caution I did only the one machine. If builds are fine in a day or so, I'll do the other ci machines.
Also, I didn't use the documented steps exactly, because I don't have permissions to upload the archive to ci.nodejs.org/downloads, I just downloaded it directly.
# ls -c1 /opt/bin
2to3
2to3-3.7
easy_install-3.7
idle3
idle3.7
pip3
pip3.7
pydoc3
pydoc3.7
python3
python3-config
python3.7
python3.7-config
python3.7m-config
pyvenv
pyvenv-3.7
https://ci.nodejs.org/computer/test-osuosl-aix61-ppc64_be-2/builds <--- builds look fine, installing python3 on the other two test machines.
I've forced a restart of the freebsd11 machine and it's still not letting me in, it might be reprovision time for that one
macstadium is odd now that I'm looking at it, I can't get in either so there's something up with our setup there, I'll open a new issue for that
test-digitalocean-freebsd11-x64-1 is back up again
Thanks @rvagg. I updated the status of test-digitalocean-freebsd11-x64-1
in the table and the spreadsheet.
something weird's up with your access to the macos machines. They use a separate port for SSH access, are you using it from your .ssh/config or manually connecting?
I tried connecting with
➜ ssh -oPort=10009 administrator@test-macstadium-macos10.08-x64-1
ssh: connect to host 207.254.58.162 port 10009: Network is unreachable
I am able to ping
that machine though
PING 207.254.58.162 (207.254.58.162): 56 data bytes
64 bytes from 207.254.58.162: icmp_seq=0 ttl=45 time=224.519 ms
64 bytes from 207.254.58.162: icmp_seq=1 ttl=45 time=224.027 ms
64 bytes from 207.254.58.162: icmp_seq=2 ttl=45 time=223.146 ms
64 bytes from 207.254.58.162: icmp_seq=3 ttl=45 time=224.018 ms
64 bytes from 207.254.58.162: icmp_seq=4 ttl=45 time=224.586 ms
But telnet
fails. So, the port is not open I guess. Perhaps the SSH Demon is not running?
➜ telnet 207.254.58.162 10009
Trying 207.254.58.162...
telnet: connect to address 207.254.58.162: Network is unreachable
telnet: Unable to connect to remote host
macstadium is odd now that I'm looking at it, I can't get in either so there's something up with our setup there, I'll open a new issue for that
Thank you :-)
@refack @mhdawson can either of you address the macos concerns here please?
See https://github.com/nodejs/build/issues/1810#event-2370170066 it was just an issue with the specific machine (and one other) that you tried to ssh to.
They are behind a host that forwards on to the actual machines. I just as the required port to my ssh config file and all works fine
@mhdawson Can you please tell us what happens when you do “brew info python” on one or more of these macOS machines? If homebrew is installed on them then these upgrades should be quite straightforward: https://docs.python-guide.org/starting/install3/osx/
python: stable 3.7.3, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
Changes to these machines are going to need to go through ansible rather than just being manual, so we maintain repeatability. It looks like brew is all set up in our Ansible configs so maybe a section in roles/baselayout/vars/main.yml for 'macos' to do the work is the way to go?
@nodejs/build anyone with access want to give this a go?
Here's a full output ftr, it's not consistent:
$ parallel-ssh -i -h /tmp/macos.hosts '/usr/local/bin/brew info python | head -4'
[1] 13:58:34 [SUCCESS] release-macstadium-macos10.10-x64-1
python: stable 3.7.0, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
[2] 13:58:34 [SUCCESS] test-macstadium-macos10.10-x64-2
python: stable 3.7.0, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
[3] 13:58:34 [SUCCESS] test-macstadium-macos10.12-x64-2
python: stable 3.7.0 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/2.7.14 (3,569 files, 49.3MB)
[4] 13:58:34 [SUCCESS] release-macstadium-macos10.11-x64-1
python: stable 3.7.0 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
[5] 13:58:34 [SUCCESS] test-macstadium-macos10.12-x64-1
python: stable 3.7.0 (bottled), devel 3.7.1rc1, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/2.7.14 (3,569 files, 49.3MB)
[6] 13:58:34 [SUCCESS] test-macstadium-macos10.11-x64-1
python: stable 3.7.2, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/2.7.14 (3,570 files, 49.3MB)
[7] 13:58:34 [SUCCESS] test-macstadium-macos10.10-x64-1
python: stable 3.7.3, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
[8] 13:58:35 [SUCCESS] test-macstadium-macos10.11-x64-2
python: stable 3.7.2, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/2.7.14 (3,570 files, 49.3MB)
$ parallel-ssh -i -h /tmp/macos.hosts '/usr/local/bin/brew info python@2 | head -4'
[1] 13:58:03 [SUCCESS] release-macstadium-macos10.11-x64-1
python@2: stable 2.7.15 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
Not installed
[2] 13:58:03 [SUCCESS] test-macstadium-macos10.10-x64-2
python@2: stable 2.7.15, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (7,512 files, 122.3MB)
[3] 13:58:03 [SUCCESS] test-macstadium-macos10.12-x64-2
python@2: stable 2.7.15 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (4,714 files, 83.4MB)
[4] 13:58:03 [SUCCESS] test-macstadium-macos10.12-x64-1
python@2: stable 2.7.15 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (4,713 files, 83.4MB)
[5] 13:58:04 [SUCCESS] test-macstadium-macos10.11-x64-1
python@2: stable 2.7.15, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (4,713 files, 83.5MB)
[6] 13:58:04 [SUCCESS] test-macstadium-macos10.11-x64-2
python@2: stable 2.7.15, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (4,714 files, 83.5MB)
[7] 13:58:04 [SUCCESS] release-macstadium-macos10.10-x64-1
python@2: stable 2.7.15, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (7,512 files, 122.3MB)
[8] 13:58:04 [SUCCESS] test-macstadium-macos10.10-x64-1
python@2: stable 2.7.16, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.15 (7,512 files, 122.3MB)
brew install python or brew install python@2 should force a brew update/upgrade to happen before the Python install so my bet is that all the boxes would end up with the same Python version(s).
Is the list at the top of this PR up-to-date?
update on the state of macos is in https://github.com/nodejs/build/pull/1836, it's taken a few days to wrangle it under control but it seems to be all consistent now with both versions installed but we need to be explicitly referencing python2
and python3
to be getting them.
Is the status table at the top of this issue up to date? I am having trouble understanding our progression and #1802 which would have allowed shared editing of such a table is also not progressing. We need an up-to-date status table.
Our only repos that have known (lintable) Python 3 incompatibilities are: nodejs/ffi, nodejs/nan, nodejs/node-auto-test, nodejs/node-chakracore, nodejs/perfetto-nodejs-wip, nodejs/worker. Please let me know if any of these repos are mission critical so that I can focus on them.
If not, then now is the time for us to start testing in earnest Python 3. With 188 days left until Python 2 end of life, I would like us to be pushing hard to be ready. Your thoughts on how we can accelerate?
@thefourtheye ping re status, can you help us move forward here?
@cclauss re the list of repos, none are critical path.
This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.
We should probably remove the temporary access at this point unless @thefourtheye chimes in to let us know they're still working on this.
We should probably remove the temporary access at this point unless @thefourtheye chimes in to let us know they're still working on this.
I've gone ahead and removed the gpg key for @thefourtheye. We can add him back if needed.
In order to have our Node.js core build system compatible with Python 2 and Python 3, we need to have both of them in our build machines as the first step.
We are trying to figure out a way to install Python3 in all our build machines. Few of our supported platforms do not have Python 3 in their respective package management systems. To figure out how to install it, I would like to have temporary access to our build machines.
PS: I tried local VMs to do this, but it is very time consuming and I don't have enough resources in my machine.
cc @nodejs/build @refack @cclauss
List of machines which need Python 3 or attention
yum search python3
failed withM2Crypto.SSL.SSLError: tlsv1 alert protocol version
apt-get -s install python3
shows python3.2 is availableapt-get -s install python3
shows python3.2 is availablepkgin av \| grep -i python3
shows both 3.3 and 3.4 are availablepkgin av \| grep -i python3
shows both 3.3 and 3.4 are availablepkgin av \| grep -i python3
shows both 3.3 and 3.4 are availablepkgin av \| grep -i python3
shows both 3.3 and 3.4 are availablepkgin av \| grep -i python3
shows 3.3, 3.4, and 3.5 are availablepkgin av \| grep -i python3
shows 3.3, 3.4, and 3.5 are availablepkgin av \| grep -i python3
shows 3.3, 3.4, and 3.5 are availablepkgin av \| grep -i python3
shows 3.3, 3.4, and 3.5 are availablepkgin av \| grep -i python3
shows 3.4, 3.5, and 3.6 are availablepkgin av \| grep -i python3
shows 3.4, 3.5, and 3.6 are availablesmitty
smitty
smitty
yum search python3
shows Python 36yum search python3
shows Python 36yum search python3
shows Python 36apt-cache search python3
shows Python 3.2 Debug buildyum search python3
does not have anythingyum search python3
does not have anythingyum search python3
shows Python 36yum search python3
shows Python 36yum search python3
shows Python 36Notes:
centos-release-6-10.el6.centos.12.3.x86_64
rpm -q centos-release
(Find OS version)yum install centos-release-scl
2.
yum install rh-python36