Closed elsbrock closed 2 years ago
Hey @else, a few questions
I have managed to track down this bug.
@sap/hana-client
's entrypoint is setting process.env['DBCAPI_API_DLL']
. The native module checks for this env value and throws if it's not set. It runs as expected in normal node process, but Jest is running all of its tests in separate vm
s, which use a copies of process.env
. If I change the code to use the real process.env
, the bug disappears.
I have no idea how vm
or native module require
work under the hood, but it seems that native module, which was required from vm
, is reading from process.env
of the main process instead of vm
's one. Will try to look deeper into the topic, but if someone knows more about it, any help is appreciated.
Anyways, for now setting env variable manually will solve your problem.
export DBCAPI_API_DLL="path/to/project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/libdbcapiHDB.dylib"
Thanks Giorgi, I can confirm that indeed makes it work!
If I change the code to use the real process.env, the bug disappears.
What do you mean by that?
@rickhanlonii regarding your questions:
What do you mean by that?
I was just explained how I debugged. It's not an actual solution for this problem.
I know, but I was trying to understand what you did.
Oh my bad.
https://github.com/facebook/jest/blob/master/packages/jest-util/src/createProcessObject.js#L19
I changed a return statement of this function with return process.env
Anything I can do to help get this fixed?
Just to clarify, I am not actively working on this issue. I think it would be more suitable for someone who has more knowledge in native modules and vm
module.
Am aware, thanks 👍
Hi, We have experienced the same issue. Hope a solution will come soon!
Anything I can do to help get this fixed?
Hi,
we have a very similar issue that can be reproduced with this repository: https://github.com/wwweidi/jest-jwt/ on all platforms (MAC,WIN,LINUX)
Ciao Stefan
You can workaround this by setting the environment variable again in the Jest config.
We were having the same issues combining the SAP Hana client library with Jest.
Setting environment variables was no solution for us, since we are working on different systems (Windows, Linux, Mac). The following steps solved it for us:
globalSetup: "<rootDir>/sapHana.config.js",
var path = require('path');
module.exports = async () => { var extensions = { 'darwin': 'dylib', 'linux': 'so', 'win32': 'dll' };
// Look for prebuilt binary and DBCAPI based on platform
var pb_subdir = null;
if (process.platform === 'linux') {
if (process.arch === 'x64') {
pb_subdir = 'linuxx86_64-gcc48';
} else if (process.arch.toLowerCase().indexOf('ppc') != -1 && os.endianness() === 'LE') {
pb_subdir = 'linuxppc64le-gcc48';
} else {
pb_subdir = 'linuxppc64-gcc48';
}
} else if (process.platform === 'win32') {
pb_subdir = 'ntamd64-msvc2010';
} else if (process.platform === 'darwin') {
pb_subdir = 'darwinintel64-xcode7';
}
var modpath = path.dirname(require.resolve("@sap/hana-client/README.md"));
var pb_path = path.join(modpath, 'prebuilt', pb_subdir);
var dbcapi = process.env['DBCAPI_API_DLL'] || path.join(pb_path, 'libdbcapiHDB.' + extensions[process.platform]);
process.env['DBCAPI_API_DLL'] = dbcapi;
};
Please note that this was tested with version 2.4.126 of @sap/hana-client and things might change with newer versions.
Thanks for sharing, IMO it's much easier though to just require("@sap/hana-client/build.js");
in your jest config.
It seems the issue still exists.
It fails even when hard coding the DBCAPI_API_DLL
and loading the hana-client like below.
process.env['DBCAPI_API_DLL'] = "/path_to_my_project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/libdbcapiHDB.dylib";
require('/path_to_my_project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/hana-client_v10.node');
seems that the native module loading done thru jest, behaves different than not via jest.
The stack trace is as below:
Error: Failed to load DBCAPI.
at internal/modules/cjs/loader.js.Module._extensions..node (internal/modules/cjs/loader.js:807:18)
at internal/modules/cjs/loader.js.Module.load (internal/modules/cjs/loader.js:653:32)
at internal/modules/cjs/loader.js.tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at internal/modules/cjs/loader.js.Module._load (internal/modules/cjs/loader.js:585:3)
at internal/modules/cjs/loader.js.Module.require (internal/modules/cjs/loader.js:692:17)
at internal/modules/cjs/helpers.js.require (internal/modules/cjs/helpers.js:25:18)
at /path_to_my_project/node_modules/jest-runtime/build/index.js._loadModule (/path_to_my_project/node_modules/jest-runtime/build/index.js:673:29)
at /path_to_my_project/node_modules/jest-runtime/build/index.js.requireModule (/path_to_my_project/node_modules/jest-runtime/build/index.js:536:10)
at /path_to_my_project/node_modules/jest-runtime/build/index.js.requireModuleOrMock (/path_to_my_project/node_modules/jest-runtime/build/index.js:699:21)
at /path_to_my_project/node_modules/@sap/hana-client/lib/index.js.Object.<anonymous> (/path_to_my_project/node_modules/@sap/hana-client/lib/index.js:112:14)
Is there anything under the scenes that would explain why the native (nodejs) require for native modules would behave different via jest? Or there are more variables required to be set for loading that hana-client module.
Is there anything under the scenes that would explain why the native (nodejs) require for native modules would behave different via jest?
The problem here is with process.env
.
Every test file is run inside VM module. But VM context does not have Node globals(require, process and etc), so Jest modifies these globals and then injects them in. VM and these modifications are needed to ensure all test files run in isolation. a.test.js
should not affect the outcome of b.test.js
.
process.env
is one of the globals which is modified. Jest makes a new object from the prototype of process.env
and reimplements the behaviour. Problem is that process.env
is a bit special since Node stores env variables in C++ core and passes it down to child processes and native bindings if needed.
So if you do this in normal Node process
process.env['DBCAPI_API_DLL'] = "/path_to_my_project/node_modules/@sap/hana-client/prebuilt/darwinintel64-xcode7/libdbcapiHDB.dylib";
All the child processes and native bindings can see this value in their Environment Variables.
This is how hana-client
works. It read this value from Environment Variables and then tries to load the file.
The reason it does not work with Jest is because Jest's process.env
is just a Javascript object. When you set a value on it, this update only exists in Javascript's context. Child processes and Native bindings cannot see it. Since hana-client
cannot see this value it fails. But if you do it outside of test context, like in global setup it should work.
I don't think this issue is solvable from Jest's codebase, you will have to use the workarounds described above. Nodejs only has one environment per process. if you run your code in VM module, you get a new Javascript Context but you cannot create new environment without creating a new process.
Every test file is run inside VM module. But VM context does not have Node globals(require, process and etc), so Jest modifies these globals and then injects them in. VM and these modifications are needed to ensure all test files run in isolation. a.test.js should not affect the outcome of b.test.js.
IMHO the VM that you described should be created the from the node process at the very moment the test enters the describe
(the very outer) callback, so that everything that did modify the process / environment etc. is kept for the whole test execution time but only modification from within the describe
blocks are not affecting any other calls.
But in any case testing like this would give false confidence on side effects that are in the code and altering the environment. For example imaging function a modifies the process.env
in some way function b might be affected because it uses this same variable. In production this code might break. But tested with jest it is all fine.
Anyways I switched away from jest using now nyc / mocha / chai and all these problems are gone.
Edit: I just found this. Should help a bunch.
Hello. My issue does not involve jest, but @sap/hana-client
is giving me trouble and I've not had much luck finding people using it, so here goes.
I'm trying to get this library to work in a container, with no luck. I'm getting this exact same error. I've set the DBCAPI_API_DLL env variable in my container, and I can confirm the value is there.
However, something else fails I guess, since I get this error: Failed to load DBCAPI.
Full error here. I've been able to confirm the env variable is set, and the folder does contain the files it should. I don't know what to do.
DBI API PATH /usr/app/node_modules/@sap/hana-client/prebuilt/linuxx86_64-gcc48/libdbcapiHDB.so
PREBUILT FOLDER CONTENTS:
-- hana-client.node
-- hana-client_v10.node
-- hana-client_v12.node
-- hana-client_v4.node
-- hana-client_v8.node
-- libdbcapiHDB.so
2020-03-05T23:43:08.366Z @sap/hana-client:index Starting index.js
2020-03-05T23:43:08.367Z @sap/hana-client:index Checking for existence of /usr/app/node_modules/@sap/hana-client/prebuilt/linuxx86_64-gcc48/hana-client_v12.node
2020-03-05T23:43:08.367Z @sap/hana-client:index Attempting to load Hana node-hdbcapi driver
2020-03-05T23:43:08.367Z @sap/hana-client:index ... Trying user-built copy...
2020-03-05T23:43:08.367Z @sap/hana-client:index ... Looking for user-built copy in /usr/app/node_modules/@sap/hana-client/build/Release/hana-client.node ...
2020-03-05T23:43:08.368Z @sap/hana-client:index Not found.
2020-03-05T23:43:08.368Z @sap/hana-client:index ... Trying prebuilt copy...
2020-03-05T23:43:08.368Z @sap/hana-client:index ... Looking for prebuilt copy in /usr/app/node_modules/@sap/hana-client/prebuilt/linuxx86_64-gcc48/hana-client_v12.node ...
2020-03-05T23:43:08.369Z @sap/hana-client:index Failed to load DBCAPI.
2020-03-05T23:43:08.369Z @sap/hana-client:index Could not load: Prebuilt copy did not satisfy requirements.
2020-03-05T23:43:08.369Z @sap/hana-client:index Could not load modules for Platform: 'linux', Process Arch:
'x64', and Version: 'v12.16.1'
/usr/app/node_modules/@sap/hana-client/lib/index.js:123
throw ex;
^
{
code: -20005,
message: 'Failed to load DBCAPI.',
sqlState: 'HY000',
stack: 'Error: Failed to load DBCAPI.\n' +
' at internal/modules/cjs/loader.js.Module._extensions..node (internal/modules/cjs/loader.js:1208:18)\n' +
' at internal/modules/cjs/loader.js.Module.load (internal/modules/cjs/loader.js:1002:32)\n' +
' at internal/modules/cjs/loader.js.Module._load (internal/modules/cjs/loader.js:901:14)\n' +
' at internal/modules/cjs/loader.js.Module.require (internal/modules/cjs/loader.js:1044:19)\n' +
' at internal/modules/cjs/helpers.js.require (internal/modules/cjs/helpers.js:77:18)\n' +
' at /usr/app/node_modules/@sap/hana-client/lib/index.js (/usr/app/node_modules/@sap/hana-client/lib/index.js:115:14)\n' +
' at internal/modules/cjs/loader.js.Module._compile (internal/modules/cjs/loader.js:1158:30)\n' +
' at internal/modules/cjs/loader.js.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)\n' +
' at internal/modules/cjs/loader.js.Module.load (internal/modules/cjs/loader.js:1002:32)\n' +
' at internal/modules/cjs/loader.js.Module._load (internal/modules/cjs/loader.js:901:14)\n'
}
Again, apologies for posting a non jest issue on this repo, I'm willing to remove this if inapropriate
Cheers!
@FourSpotProject thanks for the os specific config. It works fine, the only issue I faced is the version that hana-client expects when running jest. Somehow one version works fine when the project is run, but while running jest hana-client expects a different version for the env variable DBCAPI_API_DLL as shown below:
Invalid libdbcapiHDB version. Expected 'libdbcapiHDB 2.05.104.1595620917', but found 'libdbcapiHDB 2.06.054.1598302850.
This was resolved by installing the respective version for hana-client, setting the libdbcapiHDB files into a separate folder, and directing it from the config. Then this variable is used irrespective of the version required in the original project. But even then the reason for such an error is unknown.
This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.
This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.
🐛 Bug Report
I can load a native module in a regular script just fine, but I cannot do the same in a spec being executed by jest.
To Reproduce
Steps to reproduce the behavior:
Install native module like this:
Reproducible example:
Expected behavior
No error related to module loading
Link to repl or repo (highly encouraged)
https://transfer.sh/%28/JOeLK/example.tar.gz%29.tar.gz
Run
npx envinfo --preset jest
Paste the results here: