Open gregid opened 3 years ago
Hi,
I faced the same issue and dug into the code to find that the following error is thrown:
TypeError: httpClient.get is not a function
I noticed that httpClient is a node proxy and for some reason the proxy does not work with jest >= 25. Jest 24.9.0 is fine.
There is probably a compatibility issue somewhere with esm library
Hi @gregid, @Nebulis -- you're absolutely right, the issue turns out to be with the esm
library; it has problems working with Jest (and some React Native bundlers).
We're investigating a way to move away from it for this reason.
@Nebulis @dmitrizagidulin thanks for looking into this!
Hi,
Would you consider moving back and not use an ESM only library? So far this change broke almost everywhere in our codebase:
I know that many people in the community (me included) are excited about ESM. I even tried to update our libraries and support ESM only. But turns out it was a mistake and the community is not ready for that.
I wrote about every issue I encountered with ESM in our libraries => https://github.com/Open-Attestation/open-attestation/pull/197
The community is simply not ready for ESM only libraries.
Right now, I consider forking your library and get rid of the issue. But to be fair if I can avoid the burden of maintaining a fork, I would be more than happy.
I'm running into the same error, but in my case this is definitely caused because of redirects and not an issue related to ESM or packaging. I'm posting this here first, just in case, but if the OP is indeed caused by packaging, this is a separate issue.
Having said that, I'm running into the error when the node documentLoader encounters a redirect. This is because ky
, or ultimately fetch
, is called with redirect set to manual mode. This causes an opaqueredirect
response when a redirect is encountered; notably, this response has a status of zero and no body or headers. Specifically, there is no Location
header, which the redirect handling in documentLoader seems to depend on. I can only speculate, but to me it looks like a wrong assumption in the document loader's redirect handling about the 'redirect manual' parameter.
Edit:
Just to confirm, if I set redirect to follow
the error is fixed (but bypassing the redirect code in document loader, since ky/fetch follows the redirect).
Is there any update for this issue? Any workarounds or fixes? jsonld
versions < 5 have vulnerabilities related to xmldom
dependencies. At the same time, we cannot update jsonld
to versions >= 5 because of this issue.
Steps to reproduce in our case:
const td = {
"id":"urn:simple",
"@context":"https://www.w3.org/2022/wot/td/v1.1",
"title":"MyLampThing",
"description":"Valid TD copied from the spec's first example",
"securityDefinitions":{
"basic_sc":{
"scheme":"basic",
"in":"header"
},
"basic_scd":{
"scheme":"basic",
"in":"header"
}
},
"security":[
"basic_sc"
],
"properties":{
"status":{
"type":"string",
"forms":[
{
"href":"https://mylamp.example.com/status"
}
]
}
},
"actions":{
"toggle":{
"forms":[
{
"href":"https://mylamp.example.com/toggle"
}
]
},
"toggled":{
"forms":[
{
"href":"https://mylamp.example.com/toggle"
}
]
}
},
"events":{
"overheating":{
"data":{
"type":"string"
},
"forms":[
{
"href":"https://mylamp.example.com/oh",
"subprotocol":"longpoll"
}
]
}
}
}
Run jsonld.toRDF(td, {format: 'application/nquads'})
And now the interesting thing - step 2 runs successfully in a "usual" node environment but fails with jest
(we use version 28.1.3
). The error we get is
> jsonld.InvalidUrl: Dereferencing a URL did not result in a valid JSON-LD object. Possible causes are an inaccessible URL perhaps due to a same-origin policy (ensure the server uses CORS if you are using client-side JavaScript), too many redirects, a non-JSON response, or more than one HTTP Link Header was provided for a remote context.
Is there any update for this issue? Any workarounds or fixes?
jsonld
versions < 5 have vulnerabilities related toxmldom
dependencies. At the same time, we cannot updatejsonld
to versions >= 5 because of this issue.Steps to reproduce in our case:
- Take this W3C Thing Description (which is essentially a JSON-LD document)
const td = { "id":"urn:simple", "@context":"https://www.w3.org/2022/wot/td/v1.1", "title":"MyLampThing", "description":"Valid TD copied from the spec's first example", "securityDefinitions":{ "basic_sc":{ "scheme":"basic", "in":"header" }, "basic_scd":{ "scheme":"basic", "in":"header" } }, "security":[ "basic_sc" ], "properties":{ "status":{ "type":"string", "forms":[ { "href":"https://mylamp.example.com/status" } ] } }, "actions":{ "toggle":{ "forms":[ { "href":"https://mylamp.example.com/toggle" } ] }, "toggled":{ "forms":[ { "href":"https://mylamp.example.com/toggle" } ] } }, "events":{ "overheating":{ "data":{ "type":"string" }, "forms":[ { "href":"https://mylamp.example.com/oh", "subprotocol":"longpoll" } ] } } }
- Run
jsonld.toRDF(td, {format: 'application/nquads'})
- And now the interesting thing - step 2 runs successfully in a "usual" node environment but fails with
jest
(we use version28.1.3
). The error we get is> jsonld.InvalidUrl: Dereferencing a URL did not result in a valid JSON-LD object. Possible causes are an inaccessible URL perhaps due to a same-origin policy (ensure the server uses CORS if you are using client-side JavaScript), too many redirects, a non-JSON response, or more than one HTTP Link Header was provided for a remote context.
I didn't notice that xmldom
-related vulnerability issues were already solved in jsonld
version 4, so we've updated to that one (which works just fine). However, this jest
-related issue still refrains us from updating to versions >= 5.
One problem seems to be in lib/documentLoaders/node.js
, at the bottom where _fetch()
catches errors and sets a cause: e
. That error is somehow dropped when it pops up the stack. So you don't even see the real problem, just the wrapper. I didn't figure that issue out yet, but if you just print the error there, then Jest tells you what is wrong:
Error: You need to run with a version of node that supports ES Modules in the VM API. See https://jestjs.io/docs/ecmascript-modules
at invariant (/.../node_modules/jest-runtime/build/index.js:2448:11)
at importModuleDynamically (/.../node_modules/jest-runtime/build/index.js:1683:11)
...
I didn't read https://jestjs.io/docs/ecmascript-modules beyond the suggestion to try a node option, and with that the original test code above worked fine (adjust to your local testing setup):
NODE_OPTIONS=--experimental-vm-modules npm t
Seems the error chain is too deep and just prints out some deep details as [Object]
. You can see the error in the test catch handler by using something like this instead:
console.error(require('util').inspect(error, {depth: 100}));
This can be reproduced by altering the node-document-loader-tests.js:83
and add .status(304)
to the response:
_app.get('/data/:id', _store, (req, res) => {
res.status(304).json(_data[req.params.id].content);
});
To me the issue is with the handling of http status codes in lib/documentLoaders/node.js:127
where a 304 is handled similarly as a 302 ... but a 304 means 'not modified on the server side' (see mozilla http 304 documentation)
This 'bug' (feature?) leads to an issue with loading the json-ld context of the ed25519-signature-2020 because this returns a 304 as well 😏
@marcvanandel I think your comment may be mixing issues? The other chat here is more related to ESM usage. You might want to create a new issue just for redirection issues.
It does look like the default node document loader could use some improvements to better handle different result codes. I'm not sure how 304 would ever be returned since the loader is not sending cache related headers.
I think my comment above might have been lost in the noise here, but there is definitely an issue with the document loader that is not related to ESM usage. As is, redirects will simply not work.
I've created #499 for the handling of 304s better
One problem seems to be in
lib/documentLoaders/node.js
, at the bottom where_fetch()
catches errors and sets acause: e
. That error is somehow dropped when it pops up the stack. So you don't even see the real problem, just the wrapper. I didn't figure that issue out yet, but if you just print the error there, then Jest tells you what is wrong:Error: You need to run with a version of node that supports ES Modules in the VM API. See https://jestjs.io/docs/ecmascript-modules at invariant (/.../node_modules/jest-runtime/build/index.js:2448:11) at importModuleDynamically (/.../node_modules/jest-runtime/build/index.js:1683:11) ...
I didn't read https://jestjs.io/docs/ecmascript-modules beyond the suggestion to try a node option, and with that the original test code above worked fine (adjust to your local testing setup):
NODE_OPTIONS=--experimental-vm-modules npm t
Hi there,
Any updates on this? Were you able to find a working setup with jest and jsonld? We started having this issue (jest and ESM) when switching from jsonld 5.2.0 to 8.1.0. We had another issue with jest when using node v18.14.2 (because of the jsonld 5.2.0 dependency node-fetch 3.0.0-beta.9).
I remember that I faced an issue with jsonld before but I am not sure if this could be related.
Any updates or explanations are highly appreciated. Thanks a lot!
After upgrading to jsonld >= 5.0.0 (away from using
request
library anywhere) I've got the following error:for the following
jest
test:This test passes in
jsonld
versions3.x
and4.x
*
The comment in https://github.com/digitalbazaar/jsonld.js/blob/master/lib/documentLoaders/node.js#L180 suggests that it now considers redirects as errors. If that's the case it limits the usability of theIt seems this is just a comment onjsonld.documentLoaders.node()
ky
not the loader itself as the rest of the code suggest it should deal with redirects.*
with some caveats related to the multiple use ofrequest
injest
: immers-space/activitypub-express/issues/40 which was the main motivator for upgrade