[Docs] Chocolatey VS installation steps and project-local msvs_version for Windows (#on-windows)

scscgit commented 2 years ago

I'd like to suggest some DX improvements to README.md documentation of an initial Windows setup, which is presented to the users under error log by linking to https://github.com/nodejs/node-gyp#on-windows when they attempt to install any npm project that has node-gyp as a dependency.

The "Install tools and configuration manually" steps are ambiguous and basically just point users to a home page, having them search for binaries with correct versions by themselves. It doesn't even mention if it's okay to use the latest versions in a backward-compatible way or not.

My suggestion is to add an optional alternative in the form of Chocolatey scripts, which will also help with our automation. For example, by trial and error, I came up with this set of packages:

choco install python3 visualstudio2022community visualstudio2022buildtools visualcpp-build-tools -y
(Node can be also installed as choco install nodejs-lts -y)

I'm not sure what's the optimal or minimal set of dependencies, so I'm asking for this to be tested and reviewed, especially to cover any other edge case scenarios that can occur. (I'm not a regular user of node-gyp, so I probably won't be able to help you.)

Additionally, the npm config set msvs_version 2017 command recommends us to set this variable globally for our machine, which doesn't make much sense to me if we're doing this as part of a project setup (e.g. you download and install a random open-source library). My suggestion is to change the default recommendation to store this variable in .npmrc under project root directory, making it possible to commit this into project repository together with any other project setup instructions. This is supported by a --location switch as follows:

~npm config set msvs_version 2017 --location project~

For a reference, here is a typical error message during the project setup troubleshooting, which is also related to

https://github.com/nodejs/node-gyp/issues/802

npm ERR! gyp info find Python using Python version 3.10.4 found at "C:\Python310\python.exe"
npm ERR! gyp ERR! find VS
npm ERR! gyp ERR! find VS msvs_version not set from command line or npm config
npm ERR! gyp ERR! find VS VCINSTALLDIR not set, not running in VS Command Prompt
npm ERR! gyp ERR! find VS checking VS2022 (17.1.32319.34) found at:
npm ERR! gyp ERR! find VS "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools"
npm ERR! gyp ERR! find VS - found "Visual Studio C++ core features"
npm ERR! gyp ERR! find VS - missing any VC++ toolset
npm ERR! gyp ERR! find VS checking VS2022 (17.1.32319.34) found at:
npm ERR! gyp ERR! find VS "C:\Program Files\Microsoft Visual Studio\2022\Community"
npm ERR! gyp ERR! find VS - found "Visual Studio C++ core features"
npm ERR! gyp ERR! find VS - missing any VC++ toolset
npm ERR! gyp ERR! find VS checking VS2017 (15.9.28307.1831) found at:
npm ERR! gyp ERR! find VS "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools"
npm ERR! gyp ERR! find VS - found "Visual Studio C++ core features"
npm ERR! gyp ERR! find VS - found VC++ toolset: v141
npm ERR! gyp ERR! find VS - missing any Windows SDK
npm ERR! gyp ERR! find VS could not find a version of Visual Studio 2017 or newer to use
npm ERR! gyp ERR! find VS looking for Visual Studio 2015
npm ERR! gyp ERR! find VS - not found
npm ERR! gyp ERR! find VS not looking for VS2013 as it is only supported up to Node.js 8
npm ERR! gyp ERR! find VS
npm ERR! gyp ERR! find VS **************************************************************
npm ERR! gyp ERR! find VS You need to install the latest version of Visual Studio
npm ERR! gyp ERR! find VS including the "Desktop development with C++" workload.
npm ERR! gyp ERR! find VS For more information consult the documentation at:
npm ERR! gyp ERR! find VS https://github.com/nodejs/node-gyp#on-windows
npm ERR! gyp ERR! find VS **************************************************************

cclauss commented 2 years ago

~My caution would be that every optional alternative creates support complexity. So I would be happier if we could eliminate an alternative for every alternative that gets added. See~ #2061

~Many JavaScript developers do not bother updating their Python so we have been well served by migrating to the well-tested auto-updating Python installation from the Microsoft store.~

scscgit commented 2 years ago

Wow, we already had PR with a nice fine-tuned command like choco install visualstudio2017buildtools --params "--includeRecommended --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.VC.140", yet it was rejected without any hints as to its existence? Nowadays an automated/scriptable installation is a must-have if it exists, I mean, it's nice that (with the existing documentation) your "support burden" is limited to a scope of keeping readme up-to-date with Microsoft website updates, but really it's just a lie, because as long as the steps are ambiguous even a little bit, any support necessarily consists of replying to people who must open new issue threads, which is a waste of time for both parties when it could've been avoided. We don't collectively save any more time by ignoring a situation when the choco package suddenly becomes incompatible, quite the opposite, there just won't be any track record of the change. Same applies to any workarounds that could be automated but aren't.

(Ambiguity also includes situations with different combinations of versions installed where we need to know their compatibility beforehand, and to be able to clean it up afterwards, or when we aren't able/willing to randomly experiment with installing tools onto personal/work computers that we aren't absolutely certain are required. You also can't consistently automate clicking on checkboxes of installation wizard, nor would you want to sacrifice setup reproducibility.)

I'm aware that if you add a list of choco packages, you'll need to keep them up-to-date, but that's exactly the point - we need someone to maintain an explicit/minimal list of dependencies whenever the situation changes in future anyway. We're in this situation only because of Microsoft's poor decisions that made the installation process overly complicated and error-prone, which forces us to move the complexity onto any onboarding documentation of our projects, so we all inherit this technical debt anyway: if it isn't documented by the library, we need to document the transitive dependencies ourselves, and if we aren't experts at history of Microsoft VS build tools, we can only "guess" and hope we haven't forgotten anything critical. Nowadays, pretty much every other open-source npm library online fails to build when strictly following their documentation, it's as if we were returning to C++/Makefile ages where package managers didn't exist...

As a source of truth, the goal here is to specify the minimal set of dependencies (or their tested version ranges). Similar to that ambiguous link to a Microsoft portal, choco packages are also just another specific download method for the same specific (recommended) binaries, so even with two download options, their documentation stays singular. In theory, the mapping of binaries to choco packages could be done at any level, but Microsoft doesn't leave us with any options other than directly listing all the packages as far as I know.

Regarding Python, their package is pretty much straightforward, so I don't really care either way, it can stay as it is. Just note that unlike On Unix, On macOS or Configuring Python Dependency, the On Windows section for some reason doesn't include the version ranges. This makes the Source of Truth basically equivalent to the Microsoft Store package (rather than making it just one way to install it), despite how we may already have different Python installed. The only reason why I'd suggest including it as part of (an optional) choco script is that we could eventually have a "quick-fix" script that installs everything, for the laziest of us (and for any virtual machine setup) :)

However, I STRONGLY DISAGREE with the "now that Microsoft has invested to put the current Python into the Windows AppStore" sentiment. Microsoft Store is trash, always has been a trash, and we should be never required to use it - especially not for the reason of background auto-updates, which is a feature some of us consider unwanted. I'd much rather see a build assertion with error message "please update your Python" :)
- But feel free to add a command-line solution to install the Microsoft Store Python app ;) It's not like Microsoft makes it easy to automate working with appx, and their winget sucks anyway.

iki commented 2 years ago

@scscgit Thanks for the command, saved my day. Agree that automated install is useful and luckily quite common nowadays.

choco install visualstudio2017buildtools --params "--includeRecommended --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.VC.140"

To me it seems safe to document it, as it is equal to manual install. Also, th chocolatey package issues can be handled at the package repo (maybe that could be mentioned too).

cclauss commented 2 years ago

If it will lower the number of open Windows issues then I am all for it.

https://github.com/nodejs/node-gyp/labels/Windows

lokinmodar commented 2 years ago

This MUST be included in the docs urgently

DenisBessa commented 1 year ago

This would improve massively the DX

karlhorky commented 7 months ago

Now that the 2022 version is out (Visual C++ build tools workload for Visual Studio 2022 Build Tools 1.0.0) does anyone eg. @scscgit have a suggestion including this new version?

One nice benefit about 2022 version is that it no longer requires configuration of npm config set msvs_version

Our current version:

Our Windows System Setup guide for our students learning JavaScript and TypeScript and installing npm packages currently includes the following step:

choco install python visualstudio2022-workload-vctools -y

(Python 3 is also commonly used to build packages)

But happy to hear if we're missing something here!

karlhorky commented 7 months ago

Interesting, while @Eprince-hub and I were trying to use visualstudio2022-workload-vctools install libpg-query, there seems to be an error message about "missing any Windows SDK":

cc @jberezanski (maintainer of the Chocolatey package), in case you would have any tips here

Victor@Victor MINGW64 ~/upleveled/next-js-repo-example-2023 (main)
$ pnpm add libpg-query@16.1.0
Packages: +602
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Progress: resolved 611, reused 602, downloaded 0, added 0, done
node_modules/.pnpm/libpg-query@16.1.0/node_modules/libpg-query: Running install script, failed in 6.5s
.../node_modules/libpg-query install$ node-pre-gyp install --fallback-to-build
│ node-pre-gyp info it worked if it ends with ok
│ node-pre-gyp info using node-pre-gyp@1.0.11
│ node-pre-gyp info using node@19.8.1 | win32 | x64
│ node-pre-gyp info check checked for "C:\Users\Victor\upleveled\next-js-repo-example-2023\node_modules\.pnpm\libpg-query@16.1.0\node_modules\libpg-query\build\Release\queryparser.node" (not found)
│ node-pre-gyp http GET https://supabase-public-artifacts-bucket.s3.amazonaws.com/libpg-query-node/queryparser-v16.1.0-node-v111-win32-x64.tar.gz
│ node-pre-gyp ERR! install response status 404 Not Found on https://supabase-public-artifacts-bucket.s3.amazonaws.com/libpg-query-node/queryparser-v16.1.0-node-v111-win32-x64.tar.gz 
│ node-pre-gyp WARN Pre-built binaries not installable for libpg-query@16.1.0 and node@19.8.1 (node-v111 ABI, unknown) (falling back to source compile with node-gyp) 
│ node-pre-gyp WARN Hit error response status 404 Not Found on https://supabase-public-artifacts-bucket.s3.amazonaws.com/libpg-query-node/queryparser-v16.1.0-node-v111-win32-x64.tar.gz 
│ gyp info it worked if it ends with ok
│ gyp info using node-gyp@9.4.1
│ gyp info using node@19.8.1 | win32 | x64
│ gyp info ok 
│ gyp info it worked if it ends with ok
│ gyp info using node-gyp@9.4.1
│ gyp info using node@19.8.1 | win32 | x64
│ gyp info find Python using Python version 3.12.2 found at "C:\Python312\python.exe"
│ gyp ERR! find VS 
│ gyp ERR! find VS msvs_version was set from command line or npm config
│ gyp ERR! find VS - looking for Visual Studio version 2022
│ gyp ERR! find VS VCINSTALLDIR not set, not running in VS Command Prompt
│ gyp ERR! find VS checking VS2022 (17.9.34723.18) found at:
│ gyp ERR! find VS "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools"
│ gyp ERR! find VS - found "Visual Studio C++ core features"
│ gyp ERR! find VS - found VC++ toolset: v143
│ gyp ERR! find VS - missing any Windows SDK
│ gyp ERR! find VS could not find a version of Visual Studio 2017 or newer to use
│ gyp ERR! find VS not looking for VS2015 as it is only supported up to Node.js 18
│ gyp ERR! find VS not looking for VS2013 as it is only supported up to Node.js 8
│ gyp ERR! find VS 
│ gyp ERR! find VS valid versions for msvs_version:
│ gyp ERR! find VS 
│ gyp ERR! find VS **************************************************************
│ gyp ERR! find VS You need to install the latest version of Visual Studio
│ gyp ERR! find VS including the "Desktop development with C++" workload.
│ gyp ERR! find VS For more information consult the documentation at:
│ gyp ERR! find VS https://github.com/nodejs/node-gyp#on-windows
│ gyp ERR! find VS **************************************************************
│ gyp ERR! find VS
│ gyp ERR! configure error
│ gyp ERR! stack Error: Could not find any Visual Studio installation to use
│ gyp ERR! stack     at VisualStudioFinder.fail (C:\Users\Victor\AppData\Local\node\corepack\pnpm\8.15.5\dist\node_modules\node-gyp\lib\find-visualstudio.js:122:47)
│ gyp ERR! stack     at C:\Users\Victor\AppData\Local\node\corepack\pnpm\8.15.5\dist\node_modules\node-gyp\lib\find-visualstudio.js:75:16
│ gyp ERR! stack     at VisualStudioFinder.findVisualStudio2013 (C:\Users\Victor\AppData\Local\node\corepack\pnpm\8.15.5\dist\node_modules\node-gyp\lib\find-visualstudio.js:380:14)
│ gyp ERR! stack     at C:\Users\Victor\AppData\Local\node\corepack\pnpm\8.15.5\dist\node_modules\node-gyp\lib\find-visualstudio.js:71:14
│ gyp ERR! stack     at VisualStudioFinder.findVisualStudio2015 (C:\Users\Victor\AppData\Local\node\corepack\pnpm\8.15.5\dist\node_modules\node-gyp\lib\find-visualstudio.js:364:14)
│ gyp ERR! stack     at C:\Users\Victor\AppData\Local\node\corepack\pnpm\8.15.5\dist\node_modules\node-gyp\lib\find-visualstudio.js:67:12
│ gyp ERR! stack     at VisualStudioFinder.parseData (C:\Users\Victor\AppData\Local\node\corepack\pnpm\8.15.5\dist\node_modules\node-gyp\lib\find-visualstudio.js:237:5)
│ gyp ERR! stack     at C:\Users\Victor\AppData\Local\node\corepack\pnpm\8.15.5\dist\node_modules\node-gyp\lib\find-visualstudio.js:143:14
│ gyp ERR! stack     at ChildProcess.exithandler (node:child_process:411:7)
│ gyp ERR! stack     at ChildProcess.emit (node:events:512:28)
│ gyp ERR! System Windows_NT 10.0.22631
│ gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\Victor\\AppData\\Local\\node\\corepack\\pnpm\\8.15.5\\dist\\node_modules\\node-gyp\\bin\\node-gyp.js" "configure" "--fallback-to-build" …  
│ gyp ERR! cwd C:\Users\Victor\upleveled\next-js-repo-example-2023\node_modules\.pnpm\libpg-query@16.1.0\node_modules\libpg-query
│ gyp ERR! node -v v19.8.1
│ gyp ERR! node-gyp -v v9.4.1
│ gyp ERR! not ok
│ node-pre-gyp ERR! build error
│ node-pre-gyp ERR! stack Error: Failed to execute 'C:\Program Files\nodejs\node.exe C:\Users\Victor\AppData\Local\node\corepack\pnpm\8.15.5\dist\node_modules\node-gyp\bin\node-gyp.js configure --fallback-…  
│ node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (C:\Users\Victor\upleveled\next-js-repo-example-2023\node_modules\.pnpm\@mapbox+node-pre-gyp@1.0.11\node_modules\@mapbox\node-pre-gyp\lib\util\comp…  
│ Failed to execute 'C:\Program Files\nodejs\node.exe C:\Users\Victor\AppData\Local\node\corepack\pnpm\8.15.5\dist\node_modules\node-gyp\bin\node-gyp.js configure --fallback-to-build --module=C:\Users\Vict…  
│ node-pre-gyp ERR! stack     at ChildProcess.emit (node:events:512:28)
│ node-pre-gyp ERR! stack     at maybeClose (node:internal/child_process:1098:16)
│ node-pre-gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:304:5)
│ node-pre-gyp ERR! System Windows_NT 10.0.22631
│ node-pre-gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\Victor\\upleveled\\next-js-repo-example-2023\\node_modules\\.pnpm\\@mapbox+node-pre-gyp@1.0.11\\node_modules\\@mapbox\\node-pre…  
│ node-pre-gyp ERR! cwd C:\Users\Victor\upleveled\next-js-repo-example-2023\node_modules\.pnpm\libpg-query@16.1.0\node_modules\libpg-query
│ node-pre-gyp ERR! node -v v19.8.1
│ node-pre-gyp ERR! node-pre-gyp -v v1.0.11
│ node-pre-gyp ERR! not ok
└─ Failed in 6.5s at C:\Users\Victor\upleveled\next-js-repo-example-2023\node_modules\.pnpm\libpg-query@16.1.0\node_modules\libpg-query

cclauss commented 7 months ago

You are running node-gyp v9 on Python 3.12 when only node-gyp >= v10 supports Py3.12.

2869

npm config set msvs_version

npm ERR! msvs_version is not a valid npm option

karlhorky commented 7 months ago

You are running node-gyp v9 on Python 3.12 when only node-gyp >= v10 supports Py3.12.

Ok nice thanks! I'll see what we can do to upgrade node-gyp or downgrade Python.

npm config set msvs_version

Hm, @gengjiawen mentioned it is no longer required to configure npm config set msvs_version

cclauss commented 7 months ago

https://github.com/nodejs/node-gyp/blob/main/docs/Force-npm-to-use-global-node-gyp.md#windows

But... of course, you will get:

npm ERR! node_gyp is not a valid npm option

I stand by https://github.com/nodejs/node-gyp/issues/2636#issuecomment-1203996325

Please submit a PR that modernizes the installation of node-gyp on Windows but please don't provide too many options.

Given that modern npm config no longer allows setting msvs_version and node_gyp, we should review:

@owl-from-hogvarts Do you have input on how we should upgrade our install docs?

jberezanski commented 7 months ago

Hello everyone,

│ gyp ERR! find VS "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools" │ gyp ERR! find VS - found "Visual Studio C++ core features" │ gyp ERR! find VS - found VC++ toolset: v143 │ gyp ERR! find VS - missing any Windows SDK

By default, the visualstudio2022-workload-vctools Chocolatey package should install all components marked as Required or Recommended (see the component list for the workload). This does include the Windows 11 SDK version 10.0.22621.0 (component id: Microsoft.VisualStudio.Component.Windows11SDK.22621).

You can verify this by querying the installed Visual Studio Build Tools using the VSSetup powershell module (like find-visualstudio.js does):

Install-Module -Name VSSetup -Force
Get-VSSetupInstance | Select-Object -ExpandProperty Packages | Select-Object -ExpandProperty Id | Sort-Object | Select-String 'Windows.+SDK'

If the SDK component is indeed shown in the output of the above command, then the find-visualstudio.js script should be investigated as to why it does not detect the SDK correctly (the "silly" logs should be enlightening).

karlhorky commented 7 months ago

@jberezanski thanks for the debugging details, those steps look like they could be pretty helpful!

By default, the visualstudio2022-workload-vctools Chocolatey package should install all components marked as Required or Recommended (see the component list for the workload). This does include the Windows 11 SDK version 10.0.22621.0 (component id: Microsoft.VisualStudio.Component.Windows11SDK.22621).

So this leads me to believe that the Windows SDK components may be "optional components" of visualstudio2022-workload-vctools, which can be installed with the --package-parameters "--includeOptional" command below:

choco install visualstudio2022-workload-vctools --package-parameters "--includeOptional"

We did try this (uninstalled visualstudio2022-workload-vctools first, and then installed it again with the flag above), and it didn't help us to move past the "missing any Windows SDK" error message. (but it's possible we maybe did something incorrectly)

Or maybe I'm misinterpreting what you said above - maybe the Windows SDK component(s) are not optional components for this visualstudio2022-workload-vctools package at all?

If this is the case, do you have a suggestion for a different Chocolatey package (eg. maybe one of your packages) that would also have the Windows SDK component(s) needed for node-gyp? (ideally just a single package without arguments such as --package-params, but if it needs to be multiple packages and/or arguments, then that would also be acceptable).

Background / Next Steps:

I'm trying to come up with a reproducible, scriptable Chocolatey setup command for getting the Windows dependencies required for node-gyp set up on a fresh Windows 10/11 installation
Once I can confirm the correctness of my setup command, I'm prepared to also submit a PR to the node-gyp project to document this Chocolatey installation command publicly for the node-gyp community.

Eprince-hub commented 7 months ago

I did a fresh installation of Windows 11 on my Windows machine. I followed the installation guide on the UpLeveled System Setup Guide , which uses choco to install the visualstudio2022-workload-vctools without any other options, flags or extra package. Just running choco install visualstudio2022-workload-vctools, and I was able to continue my libpg-query installation without any more issue

CC @karlhorky CC @cclauss

@cclauss I want to know if this is enough already to make a PR or is there any other test that needs to be done?

cclauss commented 7 months ago

I think we are ready for a PR but I am not a Windows user so others will need to review that PR.

cclauss commented 7 months ago

Could some Windows users please review the pull request #3008?

3008

karlhorky commented 7 months ago

Thanks for the PR and reviews @Eprince-hub @StefanStojanovic @lukekarrys @cclauss 🎉

Great that Chocolatey installation is documented in the node-gyp docs!

silence48 commented 6 months ago

Thank you whoever did this.

nodejs / node-gyp

[Docs] Chocolatey VS installation steps and project-local msvs_version for Windows (#on-windows) #2636

2869

3008