search

jestjs / jest

Delightful JavaScript Testing.
https://jestjs.io
MIT License
44.22k stars 6.46k forks source link

Cannot load native module when running under jest #7480

Closed elsbrock closed 2 years ago

elsbrock commented 5 years ago

🐛 Bug Report

I can load a native module in a regular script just fine, but I cannot do the same in a spec being executed by jest.

To Reproduce

Steps to reproduce the behavior:

Install native module like this: 

$ npm config set @sap:registry https://npm.sap.com
$ npm install @sap/hana-client

Reproducible example:

$ cat package.json
{
  "name": "foo",
  "version": "1.0.0",
  "description": "",
  "main": "foobar.spec.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@sap/hana-client": "^2.3.123",
    "jest": "^23.6.0"
  }
}
$ cat foobar.spec.js
require("@sap/hana-client");
$ node -v
v8.12.0
$ DEBUG=*
$ node foobar.spec.js
  @sap/hana-client:index Starting index.js +0ms
  @sap/hana-client:index Attempting to load Hana node-hdbcapi driver +3ms
  @sap/hana-client:index ... Trying user-built copy... +0ms
  @sap/hana-client:index ... Looking for user-built copy in /Users/else/code/foo/node_modules/@sap/hana-client/build/Release/hana-client.node ...  +0ms
  @sap/hana-client:index Not found. +0ms
  @sap/hana-client:index ... Trying prebuilt copy... +0ms
  @sap/hana-client:index ... Looking for prebuilt copy in /Users/else/code/foo/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/hana-client_v8.node ...  +0ms
  @sap/hana-client:index Loaded. +50ms
  @sap/hana-client:index Success. +0ms
$ echo $?
0
$ node_modules/.bin/jest --runInBand
  @sap/hana-client:index Starting index.js +0ms
  @sap/hana-client:index Attempting to load Hana node-hdbcapi driver +1ms
  @sap/hana-client:index ... Trying user-built copy... +0ms
  @sap/hana-client:index ... Looking for user-built copy in /Users/else/code/foo/node_modules/@sap/hana-client/build/Release/hana-client.node ...  +0ms
  @sap/hana-client:index Not found. +0ms
  @sap/hana-client:index ... Trying prebuilt copy... +0ms
  @sap/hana-client:index ... Looking for prebuilt copy in /Users/else/code/foo/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/hana-client_v8.node ...  +0ms
  @sap/hana-client:index Failed to load DBCAPI. +2ms
  @sap/hana-client:index Could not load: Prebuilt copy did not satisfy requirements. +0ms
  @sap/hana-client:index Could not load modules for Platform: 'darwin', Process Arch: 'x64', and Version: 'v8.12.0' +0ms
 FAIL  ./foobar.spec.js
  ● Test suite failed to run

    Failed to load DBCAPI.

      at /Users/else/code/foo/node_modules/jest-runtime/build/index.js.requireModule (../node_modules/jest-runtime/build/index.js:372:31)
      at /Users/else/code/foo/node_modules/@sap/hana-client/lib/index.js.Object.<anonymous> (node_modules/@sap/hana-client/lib/index.js:127:14)

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        1.022s
Ran all test suites.

Expected behavior

No error related to module loading

Link to repl or repo (highly encouraged)

https://transfer.sh/%28/JOeLK/example.tar.gz%29.tar.gz

Run npx envinfo --preset jest

Paste the results here:

$ npx envinfo --preset jest
npx: installed 1 in 3.457s

  System:
    OS: macOS 10.14
    CPU: (4) x64 Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
  Binaries:
    Node: 8.12.0 - ~/.nvm/versions/node/v8.12.0/bin/node
    Yarn: 1.10.1 - ~/.nvm/versions/node/v8.12.0/bin/yarn
    npm: 6.4.1 - ~/.nvm/versions/node/v8.12.0/bin/npm
  npmPackages:
    jest: ^23.6.0 => 23.6.0
rickhanlonii commented 5 years ago

Hey @else, a few questions

grosto commented 5 years ago

I have managed to track down this bug.

@sap/hana-client's entrypoint is setting process.env['DBCAPI_API_DLL']. The native module checks for this env value and throws if it's not set. It runs as expected in normal node process, but Jest is running all of its tests in separate vms, which use a copies of process.env. If I change the code to use the real process.env, the bug disappears. I have no idea how vm or native module require work under the hood, but it seems that native module, which was required from vm, is reading from process.env of the main process instead of vm's one. Will try to look deeper into the topic, but if someone knows more about it, any help is appreciated. Anyways, for now setting env variable manually will solve your problem.

export DBCAPI_API_DLL="path/to/project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/libdbcapiHDB.dylib"
elsbrock commented 5 years ago

Thanks Giorgi, I can confirm that indeed makes it work!

If I change the code to use the real process.env, the bug disappears.

What do you mean by that?

@rickhanlonii regarding your questions:

grosto commented 5 years ago

What do you mean by that?

I was just explained how I debugged. It's not an actual solution for this problem.

elsbrock commented 5 years ago

I know, but I was trying to understand what you did.

grosto commented 5 years ago

Oh my bad.

https://github.com/facebook/jest/blob/master/packages/jest-util/src/createProcessObject.js#L19 I changed a return statement of this function with return process.env

elsbrock commented 5 years ago

Anything I can do to help get this fixed?

grosto commented 5 years ago

Just to clarify, I am not actively working on this issue. I think it would be more suitable for someone who has more knowledge in native modules and vm module.

elsbrock commented 5 years ago

Am aware, thanks 👍

nablaoperator commented 5 years ago

Hi, We have experienced the same issue. Hope a solution will come soon!

elsbrock commented 5 years ago

Anything I can do to help get this fixed?

wwweidi commented 5 years ago

Hi,

we have a very similar issue that can be reproduced with this repository: https://github.com/wwweidi/jest-jwt/ on all platforms (MAC,WIN,LINUX)

Ciao Stefan

elsbrock commented 5 years ago

You can workaround this by setting the environment variable again in the Jest config.

ThielenB commented 5 years ago

We were having the same issues combining the SAP Hana client library with Jest.

Setting environment variables was no solution for us, since we are working on different systems (Windows, Linux, Mac). The following steps solved it for us:

  1. set the global setup parameter in the jest config file: 
    globalSetup: "<rootDir>/sapHana.config.js",
  2. create a config file (in this example: sapHana.config.js in the root folder) which sets the environment variable: 
    
var path = require('path');

module.exports = async () => { var extensions = { 'darwin': 'dylib', 'linux': 'so', 'win32': 'dll' };

// Look for prebuilt binary and DBCAPI based on platform
var pb_subdir = null;
if (process.platform === 'linux') {
    if (process.arch === 'x64') {
        pb_subdir = 'linuxx86_64-gcc48';
    } else if (process.arch.toLowerCase().indexOf('ppc') != -1 && os.endianness() === 'LE') {
        pb_subdir = 'linuxppc64le-gcc48';
    } else {
        pb_subdir = 'linuxppc64-gcc48';
    }
} else if (process.platform === 'win32') {
    pb_subdir = 'ntamd64-msvc2010';
} else if (process.platform === 'darwin') {
    pb_subdir = 'darwinintel64-xcode7';
}

var modpath = path.dirname(require.resolve("@sap/hana-client/README.md"));
var pb_path = path.join(modpath, 'prebuilt', pb_subdir);
var dbcapi = process.env['DBCAPI_API_DLL'] || path.join(pb_path, 'libdbcapiHDB.' + extensions[process.platform]);

process.env['DBCAPI_API_DLL'] = dbcapi;

};



Please note that this was tested with version 2.4.126 of @sap/hana-client and things might change with newer versions.
elsbrock commented 5 years ago

Thanks for sharing, IMO it's much easier though to just require("@sap/hana-client/build.js"); in your jest config.

sassman commented 4 years ago

It seems the issue still exists. It fails even when hard coding the DBCAPI_API_DLL and loading the hana-client like below.

process.env['DBCAPI_API_DLL'] = "/path_to_my_project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/libdbcapiHDB.dylib";
require('/path_to_my_project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/hana-client_v10.node');

seems that the native module loading done thru jest, behaves different than not via jest.

The stack trace is as below:

Error: Failed to load DBCAPI.
    at internal/modules/cjs/loader.js.Module._extensions..node (internal/modules/cjs/loader.js:807:18)
    at internal/modules/cjs/loader.js.Module.load (internal/modules/cjs/loader.js:653:32)
    at internal/modules/cjs/loader.js.tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at internal/modules/cjs/loader.js.Module._load (internal/modules/cjs/loader.js:585:3)
    at internal/modules/cjs/loader.js.Module.require (internal/modules/cjs/loader.js:692:17)
    at internal/modules/cjs/helpers.js.require (internal/modules/cjs/helpers.js:25:18)
    at /path_to_my_project/node_modules/jest-runtime/build/index.js._loadModule (/path_to_my_project/node_modules/jest-runtime/build/index.js:673:29)
    at /path_to_my_project/node_modules/jest-runtime/build/index.js.requireModule (/path_to_my_project/node_modules/jest-runtime/build/index.js:536:10)
    at /path_to_my_project/node_modules/jest-runtime/build/index.js.requireModuleOrMock (/path_to_my_project/node_modules/jest-runtime/build/index.js:699:21)
    at /path_to_my_project/node_modules/@sap/hana-client/lib/index.js.Object.<anonymous> (/path_to_my_project/node_modules/@sap/hana-client/lib/index.js:112:14)

Is there anything under the scenes that would explain why the native (nodejs) require for native modules would behave different via jest? Or there are more variables required to be set for loading that hana-client module.

grosto commented 4 years ago

Is there anything under the scenes that would explain why the native (nodejs) require for native modules would behave different via jest?

The problem here is with process.env.

Every test file is run inside VM module. But VM context does not have Node globals(require, process and etc), so Jest modifies these globals and then injects them in. VM and these modifications are needed to ensure all test files run in isolation. a.test.js should not affect the outcome of b.test.js.

process.env is one of the globals which is modified. Jest makes a new object from the prototype of process.env and reimplements the behaviour. Problem is that process.env is a bit special since Node stores env variables in C++ core and passes it down to child processes and native bindings if needed.

So if you do this in normal Node process

process.env['DBCAPI_API_DLL'] = "/path_to_my_project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/libdbcapiHDB.dylib";

All the child processes and native bindings can see this value in their Environment Variables. This is how hana-client works. It read this value from Environment Variables and then tries to load the file.

The reason it does not work with Jest is because Jest's process.env is just a Javascript object. When you set a value on it, this update only exists in Javascript's context. Child processes and Native bindings cannot see it. Since hana-client cannot see this value it fails. But if you do it outside of test context, like in global setup it should work.

I don't think this issue is solvable from Jest's codebase, you will have to use the workarounds described above. Nodejs only has one environment per process. if you run your code in VM module, you get a new Javascript Context but you cannot create new environment without creating a new process.

sassman commented 4 years ago

Every test file is run inside VM module. But VM context does not have Node globals(require, process and etc), so Jest modifies these globals and then injects them in. VM and these modifications are needed to ensure all test files run in isolation. a.test.js should not affect the outcome of b.test.js.

IMHO the VM that you described should be created the from the node process at the very moment the test enters the describe (the very outer) callback, so that everything that did modify the process / environment etc. is kept for the whole test execution time but only modification from within the describe blocks are not affecting any other calls.

But in any case testing like this would give false confidence on side effects that are in the code and altering the environment. For example imaging function a modifies the process.env in some way function b might be affected because it uses this same variable. In production this code might break. But tested with jest it is all fine.

Anyways I switched away from jest using now nyc / mocha / chai and all these problems are gone.

BenoitRanque commented 4 years ago

Edit: I just found this. Should help a bunch.

Hello. My issue does not involve jest, but @sap/hana-client is giving me trouble and I've not had much luck finding people using it, so here goes.

I'm trying to get this library to work in a container, with no luck. I'm getting this exact same error. I've set the DBCAPI_API_DLL env variable in my container, and I can confirm the value is there. However, something else fails I guess, since I get this error: Failed to load DBCAPI.

Full error here. I've been able to confirm the env variable is set, and the folder does contain the files it should. I don't know what to do.

DBI API PATH /usr/app/node_modules/@sap/hana-client/prebuilt/linuxx86_64-gcc48/libdbcapiHDB.so
PREBUILT FOLDER CONTENTS:
-- hana-client.node
-- hana-client_v10.node
-- hana-client_v12.node
-- hana-client_v4.node
-- hana-client_v8.node
-- libdbcapiHDB.so
2020-03-05T23:43:08.366Z @sap/hana-client:index Starting index.js
2020-03-05T23:43:08.367Z @sap/hana-client:index Checking for existence of /usr/app/node_modules/@sap/hana-client/prebuilt/linuxx86_64-gcc48/hana-client_v12.node
2020-03-05T23:43:08.367Z @sap/hana-client:index Attempting to load Hana node-hdbcapi driver
2020-03-05T23:43:08.367Z @sap/hana-client:index ... Trying user-built copy...
2020-03-05T23:43:08.367Z @sap/hana-client:index ... Looking for user-built copy in /usr/app/node_modules/@sap/hana-client/build/Release/hana-client.node ...
2020-03-05T23:43:08.368Z @sap/hana-client:index Not found.
2020-03-05T23:43:08.368Z @sap/hana-client:index ... Trying prebuilt copy...
2020-03-05T23:43:08.368Z @sap/hana-client:index ... Looking for prebuilt copy in /usr/app/node_modules/@sap/hana-client/prebuilt/linuxx86_64-gcc48/hana-client_v12.node ...
2020-03-05T23:43:08.369Z @sap/hana-client:index Failed to load DBCAPI.
2020-03-05T23:43:08.369Z @sap/hana-client:index Could not load: Prebuilt copy did not satisfy requirements.
2020-03-05T23:43:08.369Z @sap/hana-client:index Could not load modules for Platform: 'linux', Process Arch:
'x64', and Version: 'v12.16.1'

/usr/app/node_modules/@sap/hana-client/lib/index.js:123
        throw ex;
        ^
{
  code: -20005,
  message: 'Failed to load DBCAPI.',
  sqlState: 'HY000',
  stack: 'Error: Failed to load DBCAPI.\n' +
    '    at internal/modules/cjs/loader.js.Module._extensions..node (internal/modules/cjs/loader.js:1208:18)\n' +
    '    at internal/modules/cjs/loader.js.Module.load (internal/modules/cjs/loader.js:1002:32)\n' +
    '    at internal/modules/cjs/loader.js.Module._load (internal/modules/cjs/loader.js:901:14)\n' +
    '    at internal/modules/cjs/loader.js.Module.require (internal/modules/cjs/loader.js:1044:19)\n' +
    '    at internal/modules/cjs/helpers.js.require (internal/modules/cjs/helpers.js:77:18)\n' +
    '    at /usr/app/node_modules/@sap/hana-client/lib/index.js (/usr/app/node_modules/@sap/hana-client/lib/index.js:115:14)\n' +
    '    at internal/modules/cjs/loader.js.Module._compile (internal/modules/cjs/loader.js:1158:30)\n' +
    '    at internal/modules/cjs/loader.js.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)\n' +
    '    at internal/modules/cjs/loader.js.Module.load (internal/modules/cjs/loader.js:1002:32)\n' +
    '    at internal/modules/cjs/loader.js.Module._load (internal/modules/cjs/loader.js:901:14)\n'
}

Again, apologies for posting a non jest issue on this repo, I'm willing to remove this if inapropriate

Cheers!

asrikanth27 commented 3 years ago

@FourSpotProject thanks for the os specific config. It works fine, the only issue I faced is the version that hana-client expects when running jest. Somehow one version works fine when the project is run, but while running jest hana-client expects a different version for the env variable DBCAPI_API_DLL as shown below:

Invalid libdbcapiHDB version. Expected 'libdbcapiHDB 2.05.104.1595620917', but found 'libdbcapiHDB 2.06.054.1598302850.

This was resolved by installing the respective version for hana-client, setting the libdbcapiHDB files into a separate folder, and directing it from the config. Then this variable is used irrespective of the version required in the original project. But even then the reason for such an error is unknown.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

github-actions[bot] commented 2 years ago

This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

github-actions[bot] commented 2 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.