Support for Python 3

[cclauss]

See ~#1150~, ~#1335~, ~#1336~, and ~https://travis-ci.com/nodejs/node/builds/79706150~ (~nodejs/node#21942~)

• Node Version:
• Platform:
• Compiler:
• Module:

Verbose output (from npm or node-gyp):

``` ```

[richardlau]

Historically the blocker to this has been the gyp dependency and normally changes should be upstreamed there unless we're planning on forking gyp in which case we would want our own gyp project so that we can share the changes with the main Node.js project.

cc @nodejs/gyp

[mthacker801]

I cannot successfully install Angular on my mac. I keep getting gyp error and when I look at gyp-mac-tool it says: gyp verb command configure [] gyp verb check python checking for Python executable "python2" in the PATH gyp verb which failed Error: not found: python2 gyp verb which failed at getNotFoundError (/Users/thackerfam/AngularProj/angular-cli/node_modules/which/which.js:13:12)

I have had python 3 installed forever and do not want python 2 as it can cause havoc to have both installed as they are not compatible with one another. If this is the reason for my inability to use Angular, what should I do?

[cclauss]

On your Mac, what happens when you do:

• python2 --version
• /usr/bin/python --version
• /usr/bin/python2 --version
• /usr/bin/python2.7 --version

I would highly recommend Homebrew as essential if you are doing real development on a Mac.

Brew allows you to easily install both Python 2 and Python3 with: brew install python python@2

[cclauss]

A few Python 2 issues on gyp:

$ python2 -m flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./gyptest.py:61:47: F821 undefined name 'opts'
    extra_path = [os.path.abspath(p) for p in opts.path]  # <-- 'opts' should be changed to 'args'
                                              ^
./pylib/gyp/generator/make.py:1637:50: F821 undefined name 'target'
      print "WARNING: no output for", self.type, target  # <-- 'target' should be changes to 'self.target'
                                                 ^

[SatishMHiremath]

I have faced and fixed this issue relates node-sass and python error.Try installing latest angular cli command with the steps below

Set the system variable path as well: C:\Users\windowsusername\AppData\Roaming\npm

C:\Users\windowsusername\AppData\Roaming\npm\node_modules@angular\cli\bin;

C:\Program Files\nodejs;

Then start using angular cli command prompt using: angular client project folder path>npm install -g @latest

angular client project folder path>npm install -g node-gyp

[stanleyxu2005]

It's a bit embraced that the original gyp project relies on python 2, and after couple of years, the patch is getting too big so that no people is dare to review and merge it. Need to give the gyp community more feedback to move this forward.

[AdamMajer]

Is there any interest in this? Python 2 is starting to get removed from distributions at this point.

[timoxley]

Python 2 is starting to get removed from distributions at this point.

Correct e.g. brew is now installing/upgrading to Python 3 by default.

The following just worked for me as I already had a python2.7 executable installed via brew:

npm config set python python2.7

However, if you don't have a python2.7 executable, I suspect you can do something like:

brew install python@2
# follow the instructions printed after installing `python@2` to get python 2 on your PATH 
# e.g. echo 'export PATH="/usr/local/opt/python@2/bin:$PATH"' >> ~/.bashrc
npm config set python python2.7

[NetOpWibby]

Tim's method should go in the README. I use ZSH and there was a message mentioned Xcode when I installed Python 2 so my process was as follows:

xcode-select --install

brew install python@2
echo 'export PATH="/usr/local/opt/python@2/bin:$PATH"' >> ~/.zshrc
npm config set python python2.7

[ioquatix]

For some reason, npm config set python python2.7 stopped working for me. Even after setting it, I get the error that node-gyp is trying to run python 3.6

[cclauss]

What does python2.7 --version return on your machine?

[ioquatix]

> python2.7 --version
Python 2.7.14

[ioquatix]

Here is the full error

^_^ > apm install      
Installing modules ✗
> node-pty@0.7.4 install /Users/samuel/.atom/packages/script-runner/node_modules/node-pty
> node scripts/install.js

script-runner@3.0.0 /Users/samuel/.atom/packages/script-runner
├── etch@0.14.0 
├── resize-observer-polyfill@1.5.0 
├─┬ shell-environment@0.3.1 
│ └─┬ os-locale@2.1.0 
│   ├─┬ execa@0.7.0 
│   │ ├─┬ cross-spawn@5.1.0 
│   │ │ ├─┬ lru-cache@4.1.2 
│   │ │ │ ├── pseudomap@1.0.2 
│   │ │ │ └── yallist@2.1.2 
│   │ │ ├─┬ shebang-command@1.2.0 
│   │ │ │ └── shebang-regex@1.0.0 
│   │ │ └─┬ which@1.3.0 
│   │ │   └── isexe@2.0.0 
│   │ ├── get-stream@3.0.0 
│   │ ├─┬ npm-run-path@2.0.2 
│   │ │ └── path-key@2.0.1 
│   │ ├── p-finally@1.0.0 
│   │ ├── signal-exit@3.0.2 
│   │ └── strip-eof@1.0.0 
│   ├─┬ lcid@1.0.0 
│   │ └── invert-kv@1.0.0 
│   └─┬ mem@1.1.0 
│     └── mimic-fn@1.2.0 
├── shellwords@0.1.1 
├─┬ temp-write@3.4.0 
│ ├── graceful-fs@4.1.11 
│ ├── is-stream@1.1.0 
│ ├── make-dir@1.2.0 
│ ├── pify@3.0.0 
│ ├── temp-dir@1.0.0 
│ └── uuid@3.2.1 
└── xterm@3.3.0 

gyp ERR! configure error 
gyp ERR! stack Error: Python executable "/Applications/Atom.app/Contents/Resources/app/apm/bin/python-interceptor.sh" is v3.6.4, which is not supported by gyp.
gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & < 3.0.0.
gyp ERR! stack     at failPythonVersion (/Applications/Atom.app/Contents/Resources/app/apm/node_modules/node-gyp/lib/configure.js:454:14)
gyp ERR! stack     at /Applications/Atom.app/Contents/Resources/app/apm/node_modules/node-gyp/lib/configure.js:443:9
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:197:7)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at maybeClose (internal/child_process.js:877:16)
gyp ERR! stack     at Socket.<anonymous> (internal/child_process.js:334:11)
gyp ERR! stack     at emitOne (events.js:96:13)
gyp ERR! stack     at Socket.emit (events.js:188:7)
gyp ERR! stack     at Pipe._handle.close [as _onclose] (net.js:498:12)
gyp ERR! System Darwin 17.5.0
gyp ERR! command "/Applications/Atom.app/Contents/Resources/app/apm/bin/node" "/Applications/Atom.app/Contents/Resources/app/apm/node_modules/.bin/node-gyp" "rebuild"
gyp ERR! cwd /Users/samuel/.atom/packages/script-runner/node_modules/node-pty
gyp ERR! node -v v6.9.5
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok 
npm ERR! Darwin 17.5.0
npm ERR! argv "/Applications/Atom.app/Contents/Resources/app/apm/bin/node" "/Applications/Atom.app/Contents/Resources/app/apm/node_modules/npm/bin/npm-cli.js" "--globalconfig" "/Users/samuel/.atom/.apm/.apmrc" "--userconfig" "/Users/samuel/.atom/.apmrc" "install" "--runtime=electron" "--target=1.7.11" "--arch=x64"
npm ERR! node v6.9.5
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE

npm ERR! node-pty@0.7.4 install: `node scripts/install.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the node-pty@0.7.4 install script 'node scripts/install.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the node-pty package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node scripts/install.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs node-pty
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls node-pty
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/samuel/.atom/packages/script-runner/npm-debug.log

Here is my config

 > apm config list
; cli configs
globalconfig = "/Users/samuel/.atom/.apm/.apmrc"
user-agent = "npm/3.10.10 node/v6.9.5 darwin x64"
userconfig = "/Users/samuel/.atom/.apmrc"

; environment configs
node-gyp = "/Applications/Atom.app/Contents/Resources/app/apm/bin/../node_modules/.bin/node-gyp"
python = "/Applications/Atom.app/Contents/Resources/app/apm/bin/python-interceptor.sh"

; userconfig /Users/samuel/.atom/.apmrc

; globalconfig /Users/samuel/.atom/.apm/.apmrc
cache = "/Users/samuel/.atom/.apm"
progress = false

; node bin location = /Applications/Atom.app/Contents/Resources/app/apm/bin/node
; cwd = /Users/samuel/.atom/packages/script-runner
; HOME = /Users/samuel/.atom/.node-gyp
; "npm config ls -l" to show all defaults.

^_^ > npm config list
; cli configs
user-agent = "npm/3.10.10 node/v8.10.0 darwin x64"

; userconfig /Users/samuel/.npmrc
init.author.email = "samuel.williams@oriontransfer.co.nz"
init.author.name = "Samuel Williams"
init.author.url = "http://www.codeotaku.com"
python = "python2.7"

; node bin location = /opt/local/bin/node
; cwd = /Users/samuel/.atom/packages/script-runner
; HOME = /Users/samuel
; "npm config ls -l" to show all defaults.

Hmm, I'm sure I also had apm config set python=python2.7. Maybe that's why it's not working.

[ioquatix]

That's odd, maybe it's a bug in apm.

> apm config set python python2.7
> apm config list                
; cli configs
globalconfig = "/Users/samuel/.atom/.apm/.apmrc"
user-agent = "npm/3.10.10 node/v6.9.5 darwin x64"
userconfig = "/Users/samuel/.atom/.apmrc"

; environment configs
node-gyp = "/Applications/Atom.app/Contents/Resources/app/apm/bin/../node_modules/.bin/node-gyp"
python = "/Applications/Atom.app/Contents/Resources/app/apm/bin/python-interceptor.sh"

; userconfig /Users/samuel/.atom/.apmrc

; globalconfig /Users/samuel/.atom/.apm/.apmrc
cache = "/Users/samuel/.atom/.apm"
progress = false

; node bin location = /Applications/Atom.app/Contents/Resources/app/apm/bin/node
; cwd = /Users/samuel/.atom/packages/script-runner
; HOME = /Users/samuel/.atom/.node-gyp
; "npm config ls -l" to show all defaults.

Even thought I set it, it's not changed in the config!?

[raininja]

@ioquatix what does "apm config get" show?

[raijin@ishtar][~]%apm config get
; cli configs
globalconfig = "/home/raijin/.atom/.apm/.apmrc"
user-agent = "npm/3.10.10 node/v6.9.5 linux x64"
userconfig = "/home/raijin/.atom/.apmrc"

; environment configs
node-gyp = "/usr/share/atom/resources/app/apm/bin/../node_modules/.bin/node-gyp"

; userconfig /home/raijin/.atom/.apmrc
python = "python2.7"

; globalconfig /home/raijin/.atom/.apm/.apmrc
cache = "/home/raijin/.atom/.apm"
progress = false

; node bin location = /usr/share/atom/resources/app/apm/bin/node
; cwd = /home/raijin
; HOME = /home/raijin/.atom/.node-gyp
; "npm config ls -l" to show all defaults.

[ioquatix]

koyoko% apm config get
; cli configs
globalconfig = "/home/samuel/.atom/.apm/.apmrc"
metrics-registry = "https://registry.npmjs.org/"
scope = ""
user-agent = "npm/6.0.0 node/v10.0.0 linux x64"
userconfig = "/home/samuel/.atom/.apmrc"

; environment configs
python = "/usr/lib/node_modules/atom-package-manager/bin/python-interceptor.sh"

; project config /home/samuel/.npmrc
strict-ssl = true

; globalconfig /home/samuel/.atom/.apm/.apmrc
cache = "/home/samuel/.atom/.apm"
progress = false

; node bin location = /usr/bin/node
; cwd = /home/samuel
; HOME = /home/samuel/.atom/.node-gyp
; "npm config ls -l" to show all defaults.

koyoko% apm config set python foo
koyoko% apm config get           
; cli configs
globalconfig = "/home/samuel/.atom/.apm/.apmrc"
metrics-registry = "https://registry.npmjs.org/"
scope = ""
user-agent = "npm/6.0.0 node/v10.0.0 linux x64"
userconfig = "/home/samuel/.atom/.apmrc"

; environment configs
python = "/usr/lib/node_modules/atom-package-manager/bin/python-interceptor.sh"

; project config /home/samuel/.npmrc
strict-ssl = true

; userconfig /home/samuel/.atom/.apmrc

; globalconfig /home/samuel/.atom/.apm/.apmrc
cache = "/home/samuel/.atom/.apm"
progress = false

; node bin location = /usr/bin/node
; cwd = /home/samuel
; HOME = /home/samuel/.atom/.node-gyp
; "npm config ls -l" to show all defaults.

[brennon]

It's one thing to retain support for Python 2 (though many projects are dropping it altogether). It's another thing entirely to not support Python 3. Please consider this. Python 3 is a decade old, after all... :smile:

[ostrolucky]

star this https://bugs.chromium.org/p/gyp/issues/detail?id=36 to help resolve this issue

[raininja]

starred, not sure why 10 years have passed and we are still dealing with this!

[Croydon]

GYP is dead. Google moved on to GN.

[ioquatix]

What is GN?

[Croydon]

https://chromium.googlesource.com/chromium/src/tools/gn/#what-is-gn

[raininja]

Yea but this thread https://groups.google.com/forum/#!topic/gyp-developer/U66xtisrYRM says it's just a matter of Dirk landing and reviewing the patch.

[raininja]

My question is this: why can't we just fork gyp and add the patch and go with that?

[cclauss]

There is no reason why not. That is the way of open source. If no maintainer steps forward with a cogent plan that fits your needs, you can always hit the fork button.

[cclauss]

Perhaps @refack can help to keep things progressing on this repo.

[conradwt]

Is there any movement on node-gyp compatible with Python 3.6+ as it does make any sense to downgrade to Python 2.7 on my system? Is the issue with the gyp src? If this is the case, what does one need to do resolve the issue?

[ghost]

Let's contribute to the deno project.

https://github.com/ry/deno

[cclauss]

They too seem to be focused on legacy Python at a time when EOL is only about 500 days away.

[NetOpWibby]

No such luck, more like 2020 https://stackoverflow.com/a/23062904/1167646

[cclauss]

Do the math dude! http://pythonclock.org

On 2018-08-19 there will be 500 days until Python 2 end of life.

https://github.com/cclauss/Ten-lines-or-less/blob/master/daze_left.py

[NetOpWibby]

...I forgot what year it was.

[KiYugadgeter]

Should it use six to keep compatibility?

[andi1984]

Is there any movement on node-gyp compatible with Python 3.6+ as it does make any sense to downgrade to Python 2.7 on my system? Is the issue with the gyp src? If this is the case, what does one need to do resolve the issue?

Did we get an answer on that? Please let us know what the underlying issue is, so that we can work on allowing having Python 3 used. We do not have to remove support for Python 2, but we should definitely have support for Python 3.

[refack]

So https://github.com/nodejs/node-gyp/pull/1335 landed. Now we need to fix the python detection logic in the JS part

[fenchu]

Any action here: with nodejs 10.13LTS you still ned python2:

C:\Users\morten.sensio\AppData\Roaming\npm\node_modules\appium\node_modules\heapdump>if not defined npm_config_node_gyp
(node "C:\dist\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp
.js" rebuild )  else (node "C:\dist\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
gyp ERR! configure error
gyp ERR! stack Error: Command failed: D:\python\Python371\python.EXE -c import sys; print "%s.%s.%s" % sys.version_info[:3];

if you are mainly on python2.7.x or higher most frequent python3 features have been backported like:

D:\python\Python371\python.EXE -c "import sys; print('%s.%s.%s' % sys.version_info[:3])"
3.7.1
D:\python\Python2715\python.EXE -c "import sys; print('%s.%s.%s' % sys.version_info[:3])"
2.7.15

[cclauss]

That code is now at https://github.com/nodejs/node-gyp/commit/a6990ada1f1147cb6da7ceaa0bb7c8333c9c7ae6#diff-7ad6309831a48a234780b58baca65710R374

[methane]

FYI, Python 3 support is committed on gyp repository: https://github.com/chromium/gyp/commits/master

[cclauss]

@methane @dpranke On the https://github.com/chromium/gyp repo, flake8 still spots at least two Python 3 syntax errors E999 and 9 instances of F821 undefined names including several issues for Python builtins cmp() and buffer() that were removed in Python 3. https://travis-ci.org/chromium/gyp/jobs/464221586#L473

[dpranke]

Well, I won't swear that the Python3 support is perfect but it should be pretty close once a couple more minor patches land, at least.

[cclauss]

Really nice work! chromium/gyp has done it... https://travis-ci.org/chromium/gyp/builds/466532175

[cclauss]

Where are we on this issue folks?? PRs are no longer getting reviewed? How do we make progress once again?

[morenoh149]

Not sure I see I fix above or not. I have this issue opened where I can't install the module on osx https://github.com/ethereumjs/helpeth/issues/25 should node-gyp be updated in the project?

[rvagg]

Discussing this and related issues https://github.com/nodejs/TSC/issues/642, if anyone here is passionate about python & node then you might want to join in on a working group or strategic initiative we may start to figure out an approach to moving ourselves beyond this mess. node-gyp is such a critical piece of Node ecosystem infrastructure that touching it carries a lot of risk, hence some of the hesitancy moving forward too fast on any of this. It's going to be extremely sensitive to breakage and edge cases and this repo already gets more support issues than we can handle.

[cclauss]

I am happy to remain engaged in the effort but would appreciate help. Where there are effective JavaScript solutions, we should transition to those.

node-gyp is such a critical piece of Node ecosystem infrastructure

One year from now, there will be no more updates -- not even security updates -- to legacy Python. The Node ecosystem infrastructure is too strategic and widely used to be based on unsupported code so by 1/1/2020, we really want no Python 2 in this codebase. If there are acceptable JavaScript substitutes then we should switch to them but this codebase really should not rely on Python 2. The sunsetting of Python 2 has already been delayed five years so that community is now quite emphatic on the no more updates policy. This is fair given that Python 3 predates Node.js.

[lalomartins]

I think you're 10 years too late to be talking about “moving forward too fast”.

[cclauss]

Over a year ago #1336 attempted to directly link pull requests to automated tests so that contributors and reviewers had ✅ or ❌ to give them a sanity check on PRs. Can we please get something similar working on this repo?

Some tactical steps (going down a path of porting what we have, instead of replacing):

• [ ] Get nodejs/build linting machines running both Python 2 and Python 3
  • [x] ~nodejs/build#1639~
  • [x] ~nodejs/build#1644~
  • [x] ~nodejs/node#25118~ replaced by make lint-py PYTHON=python3
  • [ ] nodejs/build#1631
  • [ ] nodejs/node#24512
• [ ] Get Travis CI kicking off automated tests on this repo (#1336 or similar)
• [ ] Get nodejs/node-gyp to be Python 3 syntax compatible
  • [ ] #1611
  • [ ] #1612
  • [ ] #1613
  • [ ] #1614
  • [ ] #1615

One important way that everyone can help is to review the above PRs.

[cclauss]

Any progress on these issues?? I opened this issue more than a year ago and in less than one year, support for Python 2 expires.

[Krinkle]

https://pythonclock.org Python 2.7 will reach end-of-life after December 31, 2019.

python2.7 is not installed on Debian 9 Stretch by default it seems, which means unless sysadmins install it explicitly, Node.js projects involving node-gyp somewhere in their dependencies currently fail to install properly.

[goddenrich]

We are transitioning to not supporting python2 anymore. This is the only blocker stopping us from doing this. Im really looking forward to when node-gyp can use python3