search

addaleax / lzma-native

Node.js interface to the native liblzma compression library (.xz file format, among others)
MIT License
105 stars 37 forks source link

Cannot load on Windows #13

Closed ak-mcooper closed 8 years ago

ak-mcooper commented 8 years ago

Install: npm install lzma-native following line (in my code or in node cli) causes crash: var lzma = require('lzma-native') got the following output:

Error: %1 is not a valid Win32 application. \node_modules\lzma-native\binding\lzma_native.node at Error (native) at Object.Module._extensions..node (module.js:440:18) at Module.load (module.js:357:32) at Function.Module._load (module.js:314:12) at Module.require (module.js:367:17) at require (internal/module.js:16:19) at \node_modules\lzma-native\index.js:31:14 at Object. (\node_modules\lzma-native\index.js:422:3) at Module._compile (module.js:413:34) at Object.Module._extensions..js (module.js:422:10)

Currently running on: node v5.10.0 node-gyp v3.3.1 node-pre-gyp v0.6.26

OS: Win10x64

addaleax commented 8 years ago

@SnapKrakPop Thanks for the report! Are you running multiple versions of Node.js and/or have renamed the executable (is this nw.js or something like that)? Does this work for you with npm install lzma-native --build-from-source?

ak-mcooper commented 8 years ago

First off, thanks for the quick response; very impressive =)

I'm attempting to run your module in electron. Bet even before getting to electron, I've tried the --build-from-source option, and it compiles fine, but running it from the node command line gives the same results. I'll be investigating some more today and I'll let you know if I come up with anything else.

addaleax commented 8 years ago

It is a known issue with Node.js addons that it can be hard to get them to work with nw.js, electron, etc. because Node.js API versions may not match and, on Windows, the main executable’s name has changed.

If you want, I can try and see whether I can create pre-built libraries for electron, but until then, you may want to use a tool like http://developers.ironsrc.com/rename-import-dll/: rid.exe node_modules/lzma-native/binding/lzma_native.node node.exe electron.exe may work. There’s also http://electron.atom.io/docs/v0.37.6/tutorial/using-native-node-modules/, but I have no experience with those steps myself.

ak-mcooper commented 8 years ago

After further investigating, I've opened up the Visual Studio project created by npm install lzma-native --build-from-source I tried to recompile them, and I get the following:

------ Rebuild All started: Project: liblzma, Configuration: Release Win32 ------ 1> build 1> Microsoft (R) Library Manager Version 14.00.23506.0 1> Copyright (C) Microsoft Corporation. All rights reserved. 1> 1> Creating library E:\Dev\npm-test\node_modules\lzma-native\deps\bin_i686\lzma.lib and object E:\Dev\npm-test\node_modules\lzma-native\deps\bin_i686\lzma.exp 1> A subdirectory or file E:\Dev\npm-test\node_modules\lzma-native\binding already exists. 1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" exited with code 1.

It seems that a build step is failing, and because of that, one or more libs are missing. Am I correct, or way off track...?

addaleax commented 8 years ago

Huh, that’s weird… Does that mean npm install lzma-native --build-from-source fails for you? If so, there a npm-debug.log you could maybe upload?

Maybe a3018c95167a fixes this problem, but unfortunately I don’t have a Windows machine at hand right now (I’ll wait for the AppVeyor built though). Does that help?

ak-mcooper commented 8 years ago

I manually deleted the directory then rebuilt and everything passed with ease. However, same results in node command line.

addaleax commented 8 years ago

Does npm install addaleax/lzma-native#a3018c95167a work for you? If so, I’ll publish that as an update.

ak-mcooper commented 8 years ago

It only solves the "already existing directory" problem, but still same results.

addaleax commented 8 years ago

Mh, did you try anything recommended in http://electron.atom.io/docs/v0.37.6/tutorial/using-native-node-modules/? That is probably the closest you can get to supporting native modules in electron, because npm cannot really know for what the modules are being used except when told explicitly…

ak-mcooper commented 8 years ago

Yup, I've tried that. There are still some issues that I have to work out. But before I worry about Electron, I have to get lzma-native to load in Node.js, which at the moment does not.

Side note: is it possible that the issue is in the fact that lzma-native.node itself includes lzma.lib?

addaleax commented 8 years ago

Ah, sorry, didn’t catch that bit – how does it fail with Node.js itself? I don’t think you’ve posted any kind of error message from this, did you?

And well, lzma-native.node should contain everything from lzma.lib, yes… that’s intentional, lzma.lib contains everything necessary to link the contained liblzma C library into the addon. Is there any particular reason you think that may be a problem?

addaleax commented 8 years ago

@SnapKrakPop Could it be you deleted the comment you just made? If you delete the lzma-native/ folder and install it from scratch, does that work? (It should download the native library via node-pre-gyp, so it should work without compilation).

The lzma.lib (and the deps/liblzma-….a) are necessary because this module includes the liblzma library itself. Amongst the reasons for that is that requiring another lib like liblzma to be installed is even more of a hustle when installing, especially on Windows :smile:

Really appreciate you digging through this with me, btw. I’d love to get to the ground of this.

ak-mcooper commented 8 years ago

From node command line:

E:\Dev\npm-test>node

var lzma = require('lzma-native') var lzma = require('lzma-native') Error: %1 is not a valid Win32 application. \?\E:\Dev\npm-test\node_modules\lzma-native\binding\lzma_native.node at Error (native) at Object.Module._extensions..node (module.js:440:18) at Module.load (module.js:357:32) at Function.Module._load (module.js:314:12) at Module.require (module.js:367:17) at require (internal/module.js:16:19) at E:\Dev\npm-test\node_modules\lzma-native\index.js:31:14 at Object. (E:\Dev\npm-test\node_modules\lzma-native\index.js:422:3) at Module._compile (module.js:413:34) at Object.Module._extensions..js (module.js:422:10)

As for the inclusion of lzma.lib, I ask because I've created my own native module, but it doesn't include another .lib on the side. I'm just trying to identify possible causes why node is reporting your module as "not a valid win32 application"...

addaleax commented 8 years ago

That just doesn’t make any sense… :( if you have the experience to, could you check that binding\lzma_native.node is a 64-bit dll?

ak-mcooper commented 8 years ago

I opened the dll with depends.exe and both liblzma.dll and lzma_native.node are reported as x86

addaleax commented 8 years ago

Huh… you said you are x64, right? Is that the same with your own node addon, too? It could of course be that node-gyp or node-pre-gyp don’t get the architecture right for some reason

ak-mcooper commented 8 years ago

I've actually tried both with --arch= argument, but no luck

addaleax commented 8 years ago

Hmm – there should be a liblzma.dll in bindings/. Do they match? Is your Node.js executable the right arch?

If you’re getting frustrated, you should feel free to check out one of the alternatives to this project that I listed in the readme. The other native projects are quite cool, too, and while they don’t have as many features and don’t support downloading pre-compiled binaries, they have a less fancy installation process.

ak-mcooper commented 8 years ago

It seems we're both at a loss for words =( Truely, thank you for all your time. I'll see if I can eventually get to the bottom of this. Cheers

addaleax commented 8 years ago

Yeah, thanks… there has been an issue for Windows support before (#5), and we ultimately got it working. On the traffic graphs it’s still one of the more popular pages, so there may be more people with your problem. :/ I’ll make sure to check this out the next time I’m on Windows, but that may be a while.

jviotti commented 8 years ago

Hi, I'm hitting this same issue in Appveyor, and can reproduce from a Windows 10 VM, when trying to make use of this module in a plain NodeJS project.

Looks like it fails due to python not being available:

gyp ERR! configure error
gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
gyp ERR! stack     at failNoPython (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:401:14)
gyp ERR! stack     at C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:356:11
gyp ERR! stack     at FSReqWrap.oncomplete (fs.js:82:15)
gyp ERR! System Windows_NT 10.0.10240
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "configure" "--fallback-to-build" "--module=C:\\Users\\IEUser\\Projects\\resin-image-write\\node_modules\\lzma-native\\binding\\lzma_native.node" "--module_name=lzma_native" "--module_path=C:\\Users\\IEUser\\Projects\\resin-image-write\\node_modules\\lzma-native\\binding"
gyp ERR! cwd C:\Users\IEUser\Projects\resin-image-write\node_modules\lzma-native
gyp ERR! node -v v4.4.2
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute 'node-gyp.cmd configure --fallback-to-build --module=C:\Users\IEUser\Projects\resin-image-write\node_modules\lzma-native\binding\lzma_native.node --module_name=lzma_native --module_path=C:\Users\IEUser\Projects\resin-image-write\node_modules\lzma-native\binding' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (C:\Users\IEUser\Projects\resin-image-write\node_modules\lzma-native\node_modules\node-pre-gyp\lib\util\compile.js:83:29)
node-pre-gyp ERR! stack     at emitTwo (events.js:87:13)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:827:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
node-pre-gyp ERR! System Windows_NT 10.0.10240
node-pre-gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\IEUser\\Projects\\resin-image-write\\node_modules\\lzma-native\\node_modules\\node-pre-gyp\\bin\\node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd C:\Users\IEUser\Projects\resin-image-write\node_modules\lzma-native
node-pre-gyp ERR! node -v v4.4.2
node-pre-gyp ERR! node-pre-gyp -v v0.6.26
node-pre-gyp ERR! not ok
Failed to execute 'node-gyp.cmd configure --fallback-to-build --module=C:\Users\IEUser\Projects\resin-image-write\node_modules\lzma-native\binding\lzma_native.node --module_name=lzma_native --module_path=C:\Users\IEUser\Projects\resin-image-write\node_modules\lzma-native\binding' (1)
npm ERR! Windows_NT 10.0.10240
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "lzma-native" "--build-from-source"
npm ERR! node v4.4.2
npm ERR! npm  v2.15.0
npm ERR! code ELIFECYCLE

npm ERR! lzma-native@1.2.1 install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the lzma-native@1.2.1 install script 'node-pre-gyp install --fallback-to-build'.
npm ERR! This is most likely a problem with the lzma-native package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-pre-gyp install --fallback-to-build
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs lzma-native
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!
npm ERR!     npm owner ls lzma-native
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\IEUser\Projects\resin-image-write\npm-debug.log

C:\Users\IEUser\Projects\resin-image-write>npm install lzma-native --build-from-source
jviotti commented 8 years ago

Even with Python, the compilation carries on successfully, but I get the same error:

Error: %1 is not a valid Win32 application.
\\?\C:\Users\IEUser\Projects\resin-image-write\node_modules\lzma-native\binding\lzma_native.node
  at Error (native)
  at Object.Module._extensions..node (module.js:434:18)
  at Module.load (C:\Users\IEUser\Projects\resin-image-write\node_modules\coffee-script\lib\coffee-script\register.js:45:36)
  at Function.Module._load (module.js:300:12)
  at Module.require (module.js:353:17)
  at require (internal/module.js:12:17)
  at C:\Users\IEUser\Projects\resin-image-write\node_modules\lzma-native\index.js:31:14
  at Object.<anonymous> (C:\Users\IEUser\Projects\resin-image-write\node_modules\lzma-native\index.js:502:3)
  at Module._compile (module.js:409:26)
  at Object.Module._extensions..js (module.js:416:10)
  at Module.load (C:\Users\IEUser\Projects\resin-image-write\node_modules\coffee-script\lib\coffee-script\register.js:45:36)
  at Function.Module._load (module.js:300:12)
  at Module.require (module.js:353:17)
  at require (internal/module.js:12:17)
  at Object.<anonymous> (C:\Users\IEUser\Projects\resin-image-write\tests\e2e.coffee:4:8)
  at Object.<anonymous> (C:\Users\IEUser\Projects\resin-image-write\tests\e2e.coffee:1:1)
  at Module._compile (module.js:409:26)
  at Object.exports.run (C:\Users\IEUser\Projects\resin-image-write\node_modules\coffee-script\lib\coffee-script\coffee-script.js:119:23)
  at compileScript (C:\Users\IEUser\Projects\resin-image-write\node_modules\coffee-script\lib\coffee-script\command.js:208:29)
  at compilePath (C:\Users\IEUser\Projects\resin-image-write\node_modules\coffee-script\lib\coffee-script\command.js:161:14)
  at Object.exports.run (C:\Users\IEUser\Projects\resin-image-write\node_modules\coffee-script\lib\coffee-script\command.js:96:21)
  at Object.<anonymous> (C:\Users\IEUser\Projects\resin-image-write\node_modules\coffee-script\bin\coffee:7:41)
  at Module._compile (module.js:409:26)
  at Object.Module._extensions..js (module.js:416:10)
  at Module.load (module.js:343:32)
  at Function.Module._load (module.js:300:12)
  at Function.Module.runMain (module.js:441:10)
  at startup (node.js:139:18)
  at node.js:968:3
addaleax commented 8 years ago

Yeah, sorry, for building native modules Python is currently required, and there’s pretty much no way around that right now. Except downloading prebuilt ones, but that seems to fail right now too.

That second one… that is from resin-io-modules/resin-image-write#36, right? Thank you, that could be really useful, because it means that I can reproduce this at least in some way! :)

jviotti commented 8 years ago

@addaleax

That second one… that is from resin-io-modules/resin-image-write#36, right? Thank you, that could be really useful, because it means that I can reproduce this at least in some way! :)

Right, feel free to create a testing PR to that module from a fork so you can trigger Appveyor builds at will . Let me know if there is anything else I can help with.

jviotti commented 8 years ago

The module is currently using lzma-native just for testing purposes to ensure a transform stream is working correctly. I'll replace with gz + Node's zlib just to get the PR merged, but happy to re-test anything if you like.

addaleax commented 8 years ago

Meh, this is all such a pain on Windows. There’s definitely something wrong with the precompiled binaries (which doesn’t get catched when testing them directly after building… ôO).

If you have a build environment set up, you can get this to work with passing --build-from-source to npm install, at least for AppVeyor that works. I’m still looking into what exactly goes wrong.

jviotti commented 8 years ago

Cool, I'm downloading Visual Studio. I'll let you know how it goes as soon as it completes.

jviotti commented 8 years ago

Sorry for the delay, same results as you. --buid-from-source works fine here as well.

addaleax commented 8 years ago

Nevermind! I’m just happy it’s working for some now, Windows can be a real pain… :(

jviotti commented 8 years ago

Yeah, check this out:

$ cd node_modules/lzma-native/binding
$ file *
liblzma.dll:      PE32 executable for MS Windows (DLL) (console) Intel 80386 32-bit
lzma_native.node: PE32 executable for MS Windows (DLL) (GUI) Intel 80386 32-bit

Looks like its fetching 32bit builds, while I'm running a 64bit Windows VM.

addaleax commented 8 years ago

@jviotti That sounds like it could be something, and I had this in mind a bit… it would make sense, but when testing off your repo, I’ve told AppVeyor to check both x86 and x64. And it always failed the same way…

Does toggling --target_arch=ia32 vs --target_arch=x64 when npm installing do anything for you? :/

jviotti commented 8 years ago

It installs 32bit versions regarding of the --target_arch. Maybe something is wrong with the node-pre-gyp configuration? Are 64bit builds being published to S3?

addaleax commented 8 years ago

Hm, there is definitely something weird going on with the arch settings. What version of file are you using? Mine (file-5.22) doesn’t really tell whether it’s 64 or 32 bit. But when I download the 64-bit and 32-bit versions, there is no difference in the file size, and that sounds like a really bad sign

jviotti commented 8 years ago

I'm using file from OS X 10.11.4 (can access Windows files from outside the VM).

$ file -v
file-5.04
addaleax commented 8 years ago

@SnapKrakPop @jviotti I just published a new version, could you check if the problem still exists in lzma-native@1.4.0? It should have been taken care of, and it works for me in a VM, but I’d like to have some external confirmation.

jviotti commented 8 years ago

Hi @addaleax ,

I can confirm it works great in Windows 10 x64. Thanks a lot!

ak-mcooper commented 8 years ago

@jviotti Thanks so much for testing. I would have liked to validate it, but I'm super busy at work (impending release).

@addaleax You are the best! Thanks for your endless efforts in supporting the Windows community =)

addaleax commented 8 years ago

Sure – I’m going to close this then, feel free to reopen or create another issue if there are more problems. It’s always cool to hear people are using my libraries to create cool software!