Closed luisrudge closed 8 years ago
One other thing I'd like to point out; a vast amount of users of Node.js come from a web background. Node.js is the epitome of the melding between native and web technologies. Gyp is seen as "the problem" when a lot of the times it's the build configuration within the modules is to blame - or, code is written but not tested (yes, web people, native code does actually need to be tested on each platform before it's considered "cross platform"!)
Things like that should be emphasized - the web world and the native world are enormously different.
@Qix- IMO, the developer typing npm install <package>
shouldn't need to care if the package is native or not. This is a problem for the package manager to solve. Unfortunately, npm choose to ignore the real issue and put the burden on the developers.
@luisrudge that's in an ideal world, though. I would love to see that. My point of native code requiring testing on different platforms still stands. Developers don't do this before pushing to NPM.
It's not NPM's job to fix that, and Gyp can only do so much to mitigate that. Javascript developers aren't end-users, they're developers. They should at least have a basic understanding of what is going on underneath their own applications, IMO.
@Qix-
that's in an ideal world, though. I would love to see that. My point of native code requiring testing on different platforms still stands. Developers don't do this before pushing to NPM. It's not NPM's job to fix that, and Gyp can only do so much to mitigate that.
Yes. The point stands. It should be tested, but my point is that node-gyp shouldn't even exist. npm should distribute the native packages itself and if it failed to build in a certain environment, don't allow developers with the same environment to download the package.
Javascript developers aren't end-users, they're developers. They should at least have a basic understanding of what is going on underneath their own applications, IMO.
Basic is a very subjective word. IMO, if I install node-saas, I know it converts saas to css. That's the understanding a developer has to have. I couldn't care less if it does with c++ bindings or javascript. Same thing with socket.io and other the other packages that depend on node-gyp.
npm should distribute the native packages itself
There is a lot of extra overhead on that. Windows users would have to have the right MSVCRT installed. And then you have to instruct users not to build in debug mode, because we all know they would (Why is this package crashing saying "Missing MSVCRTD120.dll"?
). And then users of OSX would have to make sure the code worked for them (there are a lot of compilation differences between each version, especially as of late). And then Linux users would have to compile a version for many, many different flavors of linux.
It's not feasible. That would kill the native Node scene and reduce productivity. Unless, that is, NPM wants to create its own cross-platform compilation service that compiles your code for a butt-ton of platforms and versions. I'll let the NPM developers decide the feasibility of that. :+1:
That's the understanding a developer has to have. I couldn't care less if it does with c++ bindings or javascript.
You should, though. You should understand what's going on underneath. That kind of mentality is a direct influencer in continued security concerns and bloat within NPM modules.
That's not to say you should know exactly how a library utilizes such technologies. That's a large part of the reason why you're using a library and not writing it yourself.
The point to understanding what's going on underneath is to know what an error message means and to know how to fix it. For instance, seeing that there is a syntax error in a released module indicates there is either a compiler version mismatch or that the original developer wrote code for a non-standard compiler/extension and to open a ticket only then. If it's a dependency issue, such as a missing library, you'd save yourself time by just installing the dependency yourself and then opening a ticket with the solution to either include it in the bootstrapping step or to just include it as a requirement in the documentation.
Either way, the lack of understanding of basic native operations is what is fueling a lot of problems with Node and the Node suite (NPM, node-gyp, etc.) - Node is simply a small, cross platform Javascript environment that wraps POSIX or otherwise unix-y system calls in slightly higher level constructs. That's how it got popular, and now that web developers have caught on to its usefulness, the percentage of users that actually understand the API's origins and how Node is structured (including what it actually does) is decreasing steadily.
But that's more off-topic than I think is appropriate for this thread.
By the way I like that node-gyp exists. I don't have to worry about configuring CMake or some other build system to build against Node and then make whatever a .node
is - and then call those steps in a cross-platform manner in an install hook - for each of my repositories that uses native code. I don't personally prefer the style/manner that configuration files are formatted in, but that's purely taste. Node-gyp works well for what it does.
It's great to know that you know everything about node and how it's made and what it is from inside. But I don't care and I'm sure 95% of the developers that are not writing a node app don't care too. If it's hard for npm to do this for linux, because of the huge amount of distros, then fix this for windows users, since most of the issues related with node, on Windows, are related to this nonsense.
I'm not trying to turn it into a pissing match here. My point is that in order to solve problems you should understand what you're developing with. If an individual can't at least appreciate that, then I severely question their ability to write code, much less solve problems concerning cross-platform compatibility.
@Qix-, I made an accurate statement that most people are "just trolling" and blindly making comments without reading the thread. That was my observation.
If there is a specific problem related to topic (some X package is not installing on Windows vY given I have all the prerequisites installed), lets talk about it, find solution, feel "happy" about it and move on.
Keeping it "professional" means refraining oneself from being personal as well as sticking to the topic at hand (e.g. it makes zero sense to discuss cmake vs. gyp on this thread, especially when you have no idea what you are talking about).
Alright; I'm not going to cause drama with you.
The point is, there should be a resource that outlines how the build system for Node works. I think that's fairly agreed upon.
I agree with that :) Somewhere, somehow, we need better docs for this kinds of errors.
Using the standalone C++ build tools I can't get socket.io to build it keeps throwing this error:
$ npm install --save socket.io
npm WARN package.json @ No repository field.
npm WARN package.json @ No license field.
> utf-8-validate@1.2.1 install D:\projects\nodejs\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate
> node-gyp rebuild
\
D:\projects\nodejs\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate>if not defined npm_config_node_gyp (node "D:\installs\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node rebuild )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppBuild.targets(366,5): warning MSB8003: Could not find WindowsSDKDir variable from the registry. TargetFrameworkVersion or PlatformToolset may be set to an invalid version number. [D:\projects\nodejs\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate\build\validation.vcxproj]
validation.cc
c:\users\mtpultz\.node-gyp\4.2.2\include\node\uv-win.h(32): fatal error C1083: Cannot open include file: 'winsock2.h': No such file or directory [D:\projects\nodejs\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules \utf-8-validate\build\validation.vcxproj]
gyp ERR! build error
gyp ERR! stack Error: `C:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe` failed with exit code: 1
gyp ERR! stack at ChildProcess.onExit (D:\installs\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:270:23)
gyp ERR! stack at emitTwo (events.js:87:13)
gyp ERR! stack at ChildProcess.emit (events.js:172:7)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Windows_NT 10.0.10240
gyp ERR! command "D:\\installs\\nodejs\\node.exe" "D:\\installs\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd D:\projects\nodejs\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate
gyp ERR! node -v v4.2.2
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok
> bufferutil@1.2.1 install D:\projects\nodejs\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil
> node-gyp rebuild
Any suggestions?
These kinds of errors are exactly what I'm talking about.
Could not find WindowsSDKDir variable from the registry. TargetFrameworkVersion or PlatformToolset may be set to an invalid version number.
There is the error. Hint: look for warning:
or error:
. Often times errors are accompanied by warnings that tell you what's really wrong.
Install the Windows SDK, or re-install MSVS.
@Qix- Why do I have to install MSVC when they just provided the C++ build tools and I have them installed with Python 2.7 and followed the instructions provided by @mousetraps, which seemed to work for some people, but maybe they weren't running Windows 10. I just want to compile not use their IDE or even go near .NET programming, which I know absolutely nothing about. I tried your other suggestion and downloaded and ran the Windows SDK and it said I was up to date already.
The solution I'm using now is just attaching the project to a virtualbox like Homestead in my case as I'm using Laravel + Redis + ExpressJS in this application, then just SSH in and run npm. Way less hassle then buggering around for hours downloading/installing/uninstalling/downloading/reinstalling/etc.
Sorry, should have read "MSVS" (MS Visual Studio). I haven't (yet) installed the independent build tools by Microsoft yet, but I wouldn't be surprised if they didn't include the SDK. Plus, seeing as how Windows 10 is fairly new I wouldn't be surprised if there was some other obscure error.
Did you download/install the Windows SDK? @mousetraps do the build tools include the windows SDK?
@mtpultz, we have a solution to the exact problem by @ricmed205 above:
I finally got it working checking BOTH windows 10 SDKs and windows 8 SDK (I'm on win10) when installing the 2015 build tools. It seems that somehow the windows 10 SDKs is built on top of the windows 8 SDKs OR the windows 10 SDKs package has a bug that fails to add the correspondent registry values, and my system is simple using the windows 8 SDKs (when I added it later).
As @mousetraps mentioned that SDKs are included, here is the blog post http://blogs.msdn.com/b/vcblog/archive/2015/11/02/announcing-visual-c-build-tools-2015-standalone-c-tools-for-build-environments.aspx which details on what is included and what is marked optional etc.
if you rerun the build tools setup and select both 8,1 and 10 SDKs, it will install the missing one.
Thanks @jasonwilliams200OK that did it.
@jasonwilliams200OK As mousetraps pointed here, latest nodejs is not needed (v0.x, v4, v5 works). I work with v4 myself.
@luisrudge I would add the following remarks in the fix summary:
Honestly, what really prevented/frightened me from using a node-gyp dependent module (after days frustrated with why I couldn't install selenium-webdriver, just as any other module, and researching/learning then what node-gyp was) was the size of the MSVS/Build Tools install. Why on earth would I need several GBs of code to work with at most MB size programs (JS scripts, nodejs, python, npm modules...). "Is python needed to work with sockets.io module? Mmm Ok, I'm not quite sure why this is the case, but, let's see: download/install~30MB? easy, let's do it. Next, download/install a Gbs thing. What?? I must be doing something wrong or there SHOULD be a better way somewhere else. Hell, photoshop/dreamweaver are ~1GB and the whole windows OS is ~15GB..."
Can someone explain why all those GBs are needed? Also, I see in the Build Tools: 800 MB of available hard disk space, but I got installed ~4.5Gbs? which I went an entire day downloading.
Sorry for the rant. Hopefully, the Q&A will supplement the docs to quickly/better explain the newbies like me and be more confident about the whole nodejs/windows experience.
So here it is. I have tested these steps multiple times on (each time) newly created Windows 7 SP1 VMs with IE8.
Scenario: I have a brand new Windows system, what to do?
- [Only if it is Windows 7 then and you don't already] Download .NET Framework 4.5.1 and run the installer.
- Download Build Tools 2015, run the installer, select
Custom
, clickNext
and check bothSDKs
, clickNext
and thenInstall
.- Download and install latest node.js.
- Download and install Python 2.7.x.
- Open
cmd
and runnpm config set msvs_version 2015 --global
.Hopefully it is a temporary registry fix and they will fix it before Build Tools get out of Technical Preview status. Out of 4 times it created that registry value once.
- Start > Run > cmd (and press Ctrl+Shift+Enter to Run as Administrator):
For 64-bit Windows
:: copy paste this command and press enter
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\SxS\VS7 /v 14.0 /t REG_EXPAND_SZ /d ^%programfiles(x86)^%"\Microsoft Visual Studio 14.0\\"
For 32-bit Windows
:: copy paste this command and press enter
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\SxS\VS7 /v 14.0 /t REG_EXPAND_SZ /d ^%programfiles^%"\Microsoft Visual Studio 14.0\\"
Now install the package like npm install selenium-webdriver
and it should work!
My installation is not brand new, I had some related stuff installed and now I am getting
MSB4019
orMSB8020
.
This means somehow you have incorrect version of msbuild
in PATH
. To fix that, put C:\Program Files (x86)\MSBuild\14.0\Bin;
at the beginning of PATH
under System Environment Variables. To do it quickly (without making permanent change), run SET PATH=%ProgramFiles(x86)%\MSBuild\14.0\Bin;%PATH%
in command prompt in case of 64-bit Windows or %ProgramFiles%\MSBuild\14.0\Bin;
in case of 32-bit Windows. Then run your intended npm install
command.
Finally, what to expect?
Expect every package which requires C/C++ code compilation at the time of installation to succeed on Windows, exactly as they compile at installation time on Unix-like operating systems, except those which are not supporting Windows at all (yet?) such as node-gd
. For such packages, you can find their repository URL on package's http://npmjs.org/ page and find the Windows support status in their issue tracker.
@ricmed205, I know the latest version of node.js is not a requirement here, but if you are to install right now, fetch the latest instead of some old one. Otherwise it should work on all supported (by package) versions of node.js (and conversely v8 and whatnot).
@jasonwilliams200OK
Hopefully it is a temporary registry fix
Perhaps even a default .reg
file for the C:
drive would be useful instead of having people mess with their registries? Just a thought. Good writeup, though!
Should there be an optional way to distribute native modules that don't involve a prebuild script? And any sane module will state what it depends on, if it isn't preinstalled. Most native modules do that already.
Highly frustrating stuff. node-gyp is unable to find the header files from cpp.
@rishantagarwal you'll be hard-pressed to find a build system that can do that reliably and efficiently. That's not node-gyp's fault.
@jasonwilliams200OK ,
I think
Open cmd and run npm config msvs_version 2015 --global.
should actually be
Open cmd and run npm config set msvs_version 2015 --global.
@teobugslayer, you are right, fixed above.
@jasonwilliams200OK did you try a REG_EXPAND_SZ
registry key type w/ %programfiles(x86)%
? That could allow skipping the explanation about c:
. Also instead of manually doing it with regedit, this instruction could probably be restated with a 1 line reg
command.
@fredericosilva, good idea, updated.
4 days trying only to do a npm install. Download more than 6gb. Thank you Windows and everything else for this waste of time,
Yet with these errors
@lehno We all understand the frustration some people are having, but as evidenced by this thread there are clearly many people working to make the Node ecosystem better for Windows. Why not keep things on a more positive note and support those who are spending a lot of time trying to make things better for other people? The Microsoft build tools situation is something that has been beaten to death many times over at this point.
Why on earth would I need several GBs of code to work with at most MB size programs (JS scripts, nodejs, python, npm modules...).
Several GB of stuff is to compile C/C++ code (using C/C++ compilers and SDKs).
Python is required for gyp (the underlying C/C++ build system technology for node-gyp
).
This is not for any npm package which relies on pure JavaScript code but only those which are known as "npm packages with native or C/C++ modules".
It's Generalization vs. Specialization; keeping things in proportion makes it easier to improve science and technology!
I think the number of steps needed to setup the build environment or the size of it should not be addressed here because it's simply nothing that node-gyp can change. Please don't misunderstand the title as an invitation to rant about windows here. It just bloats this thread and leads to nothing. If you still have problems following the mentioned steps you can ask for help here. @jasonwilliams200OK you are doing a great job btw.
Registry keys often get messed up when there are multiple versions, or some incomplete versions installed. In such cases using VS native tools command prompt
usually works since it sets correct environment variables using C:\Program Files (x86)\Microsoft Visual Studio <version>\VC\vcvarsall.bat
Would it be possible for node-gyp to call this files before starting gyp?
In cases when this file can not be found, error report from node-gyp about not being able to find sdk at the expected path, would be much easier to understand and resolve than "C:\Microsoft.Cpp.Default.props"
not found errors from the compiler.
@nightwing, I agree with you, generally this is how it is. However, in this case vcvarsall.bat
is throwing error from a more internal script (VC\bin\vcvars32.bat
) which tries to deduce to version from the very registry key. The line of code which is setting registry in the aforementioned steps are exactly fixing that for Build Tools 2015 and that is exactly what it looks like when we install full VS.
Also note that gyp
calls vcvarsall.bat
for us, so we don't have to call it explicitly or use Visual Studio Command Prompt.
This is a build tools bug in Technical Preview
stage. We can expect it to fix by RTM (which I think might be aligned with VS2015 Update 1 release on 30th Nov. 2015).
Thank you, @mousetraps - your comment helped, node-gyp works now on Windows 7-x64. Before installing VC++ Build Tools Technical Preview Additionally, I had to install MS Visual Studio 2015 Express with custom installation Windows 8.1 SDK checked, but I don't know, if this is necessary for others. Also I had to manually install the latest npm version:
npm install -g npm
npm cache clean
And I had to globally install the latest version of node-gyp:
npm install node-gyp@latest -g
Now I have this working environment without any node-gyp errors, if i do npm install
:
Windows 7-x64
node 4.2.2-x64
python 2.7.10-x64
npm 3.4.1
node-gyp 3.1.0
If you have trouble with this messages: “warning MSB8003: Could not find WindowsSDKDir variable from the registry.TargetFrameworkVersion or PlatformToolset may be set to an invalid version number.”
The problem may occurs even if you had installed windows sdk, or Visual C++ Build Tools 2015 (Pre-release) has been installed. I had spend almost 3 days. :(
Here comes the resolution:
{ 'target_defaults': { 'type': 'loadable_module', 'win_delay_load_hook': 'true', 'product_prefix': '',
'include_dirs': [
'<(node_root_dir)/include/node',
'<(node_root_dir)/src',
'<(node_root_dir)/deps/uv/include',
'<(node_root_dir)/deps/v8/include',
'C:/Program Files (x86)/Windows Kits/10/Include/10.0.10240.0/um',
'C:/Program Files (x86)/Windows Kits/10/Include/10.0.10240.0/shared',
'C:/Program Files (x86)/Windows Kits/10/Include/10.0.10240.0/ucrt',
'C:/Program Files (x86)/Windows Kits/10/Include/10.0.10240.0/winrt'
],
…… The warning message still there. But all the windows headers need by the package you are building will be found.
@mousetraps
@jianglinchun, this is due to the fact that in Windows 10, the SDK location has been changed and the upstream gyp (http://code.google.com/p/gyp/) has yet to address this.
Therefore, the current (bit non-hacky) recommendation is to select both Windows 8.1 and Windows 10 SDKs when installing Build Tools 2015. This way the software perhaps actually builds against Windows 8.1 SDK on Windows 10.
See the related Universal CRT announcement here: http://blogs.msdn.com/b/vcblog/archive/2015/03/03/introducing-the-universal-crt.aspx.
Thank you @jasonwilliams200OK. I do believe this problem will be solved. :) If gentlyRm have an option not to nuking the package will help a lot. :)
So I'm thinking we should just update the README Installation instructions with @mousetraps' instructions in https://github.com/nodejs/node-gyp/issues/629#issuecomment-153196245, and then we can close this issue.
Hey @TooTallNate we have an issue open over on node/node
:smile:
@TooTallNate @TheAlphaNerd yep, it's coming... got caught up with other things this week was just a little too optimistic on my timeline for sending out a PR. :-/
@mousetraps What about iso-image with build tools?
@Sogl we're looking into ISOs for the RTM release. As far as I understand, there's no technical / strategic / legal reason we didn't release one... it's mostly that things have been pretty hectic around here gearing up for VSConnect and Update 1, and there were more critical things to focus on at that point in time.
Additionally, I had to install MS Visual Studio 2015 Express with Windows 8 SDK checked
@stevenpietzsch - interesting... did you try selecting the windows 8.1 SDK in the build tools installer?
@jianglinchun, this is due to the fact that in Windows 10, the SDK location has been changed and the upstream gyp (http://code.google.com/p/gyp/) has yet to address this.
Therefore, the current (bit non-hacky) recommendation is to select both Windows 8.1 and Windows 10 SDKs when installing Build Tools 2015. This way the software perhaps actually builds against Windows 8.1 SDK on Windows 10.
@jasonwilliams200OK - is there an issue open in gyp on this (can't find one)? Also a more minimal recommendation would be to select Windows 8.1 SDK on Windows 10, right? (which I thought was the default in the c++ build tools anyways, which is why I didn't mention it in the original instructions, but maybe I was mistaken)
EDIT: Ahh, never mind... it appears that both are required for now... I didn't realize that the default install was actually different from the default options selected when you run through the installer.
Just as a humorous aside for anyone following this thread, if, like me you were curious about what node-gyp is, then without the least irony you will find it described here: https://github.com/nodejs/node-gyp like this:
node-gyp is a cross-platform command-line tool ..............takes away the pain of dealing with the various differences in build platforms.
Ahh... so painless...
Question for everyone who's configuration using the C++ Build Tools failed and they had to install Windows SDKs separately and whatnot...
How many of you chose to do the custom install? Was there anyone who ran into an issue after selecting the default install?
I've been trying to investigate some of these edge cases that people who were running into issues w/ the build SKU ran into so that I can update the instructions appropriately, and my hunch is that the custom install is where all of the issues are arising. The interesting thing about the build sku is that the default install does not do a full install. Rather, it installs a minimal set of dependencies from the Windows 8.1 and Windows 10 SDKs, and I've successfully been able to make things work using the default install of the Build Tools on Windows 7, Windows 8.1, and Windows 10.
@mousetraps, I always selected custom install. Will try with default installation option later. About the registry hack, is it also not required with default installation (or will it be fixed in RTM)?
@dukedougal You mean writing an impossible program? Building an impossible shape? Building a contradiction of the laws of the universe? You mean, this sentence is false? :stuck_out_tongue_winking_eye:
(Thankfully, I normally use Linux, but I feel your pain - try learning anything .NET with a vested interest and find out the primary package manager, NuGet, is only just now considering porting to your OS.)
@mousetraps my own recent failed attempts were using the custom installation as well, having selected both the Windows 10 and Windows 8.1 SDKs.
Curiously, running the C++ Build Tools installer again and selecting the "modify" action always displays the Windows 8.1 SDK as unselected. At this point the installer does not permit me to continue with the modification unless I check the Windows 8.1 SDK and/or uncheck the Windows 10 SDK. Checking the Windows 8.1 SDK indicates an additional 112 MB will be used, and in proceeding with the action the installer completes the "modification" in under a minute - however running the installer and choosing "modify" once again brings me back to where I started; the 8.1 SDK unchecked, and checking it apparently being interpreted as a modification to my Build Tools installation. This may simply be a misleading UI quirk or a conflict in my environment...
I will try again tomorrow with the default installation. Failing that, I will try again after removing extraneous VS components.
It all started with this conversation.
As I said, I'm pretty much a newbie and I don't even know what node-gyp does, but what I DO know is that it's a pain to run Windows and install a project with node-gyp as a dependency. Some numbers:
It's bizarre that a nodejs developer needs to install visual studio (even the express edition) in order to be able to work on Windows. Even more bizarre is that I, as a .NET developer with full visual studio installed in my machine still can't download a bunch of projects that use this project.
With that being said, I'd like to know from all of you what should we do to make windows users fall in the pit of success? What are the alternatives? Does Microsoft can solve this in any way?
This issue is fixed (kinda)
@mousetraps pointed that Microsoft is shipping the C++ build tools separately now. Check here
Some updates on this gigantic thread
Absolutely non-official statement that @Microsoft is aware of this issue and seeing what can be done
@mousetraps said that this thread "is high on people's radar" at @Microsoft
Very good Q&A about noge-gyp issues on Windows and why they happen
@dekarguy took the time to answer some questions about what we all are going through