wikijs cannot be bundled in

[Pomax]

trying to add this to some code that gets bundled complains that "3:15-22 require function is used in a way in which dependencies cannot be statically extracted" meaning your code's probably doing something like this:

somevar = "...";
if (...) { somevar = "......."; }
...
var Dependency = require(somevar);

instead of this:

var Dependency = false;
if (someval===val1) Dependency = require('thing1');
else if (someval===val2) Dependency = require('thing2');
else if (someval===val2) Dependency = require('thing3');

While it might look less clean, this actually makes the code far easier to both bundle and debug, as all requirements are static, and thus traceable.

[dijs]

I will look into this. Thanks.

[dijs]

I am not seeing any conditional requires in my code at all... @Pomax If you can dig in and find the exact line of code it cannot compile, that would be great. In the meantime, I will work on updating the babel version in hopes that a new version will compile it without bundling issues.

[dijs]

I updated the compiler, please try to use the master branch for your project

[Pomax]

let me give that a try

[Pomax]

Hm, this is very odd. If I add a package entry for master, using:

  "dependencies": {
    "arctic-redpoll": "^1.0.0",
    "cli-color": "^1.1.0",
    "inquirer": "^1.0.2",
    "md5": "^2.1.0",
    "pirates": "^2.1.1",
    "socket.io": "^1.4.5",
    "wikijs": "https://github.com/dijs/wiki/tarball/master"
  },

then a require('wikijs') yields:

C:\Users\Mike\Documents\Git projects\projects\mahjong>node
> var Wiki = require('wikijs');
Error: Cannot find module 'wikijs'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at repl:1:12
    at REPLServer.defaultEval (repl.js:252:27)
    at bound (domain.js:287:14)
    at REPLServer.runBound [as eval] (domain.js:300:12)
    at REPLServer.<anonymous> (repl.js:417:12)
    at emitOne (events.js:95:20)

despite the fact that there is a node_modules/wikijs dir.

[Pomax]

ah hang on, I know why - master does not have a dist dir. One npm run build later and the module resolves. The webpack bundling does now throw up a ton of errors though, seemingly related to the har module

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tough-cookie/lib/cookie.js
Module not found: Error: Cannot resolve module 'net' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\tough-cookie\lib
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tough-cookie/lib/cookie.js 32:10-24

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tough-cookie/package.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\tough-cookie\package.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "_args": [
|     [
|       "tough-cookie@~2.2.0",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tough-cookie/lib/cookie.js 38:14-40

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/mime-db/db.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\mime-db\db.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "application/1d-interleaved-parityfec": {
|     "source": "iana"
|   },
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/mime-db/index.js 11:17-37

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/forever-agent/index.js
Module not found: Error: Cannot resolve module 'net' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\forever-agent
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/forever-agent/index.js 6:10-24

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/forever-agent/index.js
Module not found: Error: Cannot resolve module 'tls' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\forever-agent
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/forever-agent/index.js 7:10-24

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/request/lib/har.js
Module not found: Error: Cannot resolve module 'fs' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\request\lib
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/request/lib/har.js 3:9-22

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/cache.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\cache.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "properties": {
|     "beforeRequest": {
|       "$ref": "#cacheEntry"
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 4:9-32

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/cacheEntry.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\cacheEntry.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "oneOf": [{
|     "type": "object",
|     "optional": true,
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 5:14-42

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/content.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\content.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "size",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 6:11-36

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/cookie.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\cookie.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "name",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 7:10-34

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/creator.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\creator.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "name",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 8:11-36

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/entry.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\entry.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "optional": true,
|   "required": [
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 9:9-32

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/har.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\har.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "log"
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 10:7-28

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/log.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\log.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "version",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 11:7-28

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/page.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\page.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "optional": true,
|   "required": [
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 12:8-30

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/pageTimings.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\pageTimings.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "properties": {
|     "onContentLoad": {
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 13:15-44

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/postData.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\postData.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "optional": true,
|   "required": [
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 14:12-38

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/record.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\record.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "name",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 15:10-34

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/request.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\request.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "method",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 16:11-36

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/response.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\response.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "status",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 17:12-38

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/timings.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\timings.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "required": [
|     "send",
|     "wait",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 18:11-36

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tunnel-agent/index.js
Module not found: Error: Cannot resolve module 'net' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\tunnel-agent
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tunnel-agent/index.js 3:10-24

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tunnel-agent/index.js
Module not found: Error: Cannot resolve module 'tls' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\tunnel-agent
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tunnel-agent/index.js 4:10-24

[dijs]

Ah right, you might have to run 'npm run build' manually in that directory On Sat, May 21, 2016 at 11:10 AM Mike Kamermans notifications@github.com wrote:

ah hang on, I know why - master does not have a dist dir.

— You are receiving this because you commented.

Reply to this email directly or view it on GitHub https://github.com/dijs/wiki/issues/16#issuecomment-220786133

[Pomax]

yeah, did that, which generated the dist/wikijs file, but then requiring that in leads to the error log shown.

[dijs]

Okay. Many of those errors are webpack errors related to not having loaders for those file types.

[Pomax]

Hmm, that may be, but without the require('wikijs') (and just that, not even anything else) there are no webpack errors, and everything bundles perfectly fine. Do you know whether the dist/wiki.js lib still relies on lots of other things that require specific webpack loaders before things'll bundle correctly?

[dijs]

These are the dependencies:

"babel-polyfill": "^6.9.0",
"isomorphic-fetch": "^2.2.1",
"underscore": "^1.8.3"

Those should all be browser compatible...

[Pomax]

yeah those shouldn't really be a problem... alright, I created an mcve for webpack (without babel, eslint, etc) that shows this off as minimal as I can figure.

package.json:

{
  "name": "wikitest",
  "version": "1.0.0",
  "description": "",
  "main": "test.js",
  "scripts": {
    "test": "webpack"
  },
  "author": "",
  "license": "PD",
  "dependencies": {
    "webpack": "^1.12.9",
    "wikijs": "^0.2.0"
  }
}

With a simple test.js:

var Wiki = require('wikijs');

And a pretty simple webpack config:

var webpack = require('webpack');
module.exports = {
  entry:  "./test.js",
  target: "web",
  output: {
    path: ".",
    filename: "bundle.js"
  }
};

creating these three files, running npm install and then npm test yields the same kind of errors that I was seeing in my regular project.

[dijs]

Okay thanks for the test, I will do the same On Sun, May 22, 2016 at 9:11 PM Mike Kamermans notifications@github.com wrote:

yeah those shouldn't really be a problem... alright, I created an mcve for webpack (without babel, eslint, etc) that shows this off as minimal as I can figure.

package.json:

{ "name": "wikitest", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "webpack" }, "author": "", "license": "PD", "dependencies": { "webpack": "^1.12.9", "wikijs": "^0.2.0" } }

With a simple test.js:

var Wiki = require('wikijs');

And a pretty simple webpack config:

var webpack = require('webpack'); module.exports = { entry: "./test.js", target: "web", output: { path: ".", filename: "bundle.js" } };

creating these three files, running npm install and then npm test yields the same kind of errors that I was seeing in my regular project.

— You are receiving this because you commented.

Reply to this email directly or view it on GitHub https://github.com/dijs/wiki/issues/16#issuecomment-220873667

[dijs]

I should have seen this before, the issue is not with wikijs. The issue is that you are trying to import a module that cannot run on in the browser. You will have the same issue with many other server-side only modules. I may be able to make a client-side version of this library. Would that suffice?

[Pomax]

that would be perfect - curious though, what does wikijs rely on that that prevents browser bundling? (Most of the things it'd need to do like net-fetching, parsing, etc. have universal packages available)

[dijs]

Good question! That is actually exactly why I just started using isomorphic-fetch.

I thought that library would be fine in both browser and node, but apparently, it is not...

I guess when webpack tried to compile it, it sees request, fs, etc... and blows up...

So, maybe we should take it up with them?? Because I would love to get this working without having to create another module...

[Pomax]

hm, if it's https://www.npmjs.com/package/isomorphic-fetch then it looks like it's optimized for browserify. Webpack failing on system-specific APIs makes sense, but maybe if the contract for isomorphic-fetch is identical to the browser Fetch API then it might be possible to tell webpack to treat isomorphic-fetch as an external, so that it replaces require('isomorphic-fetch') with fetch when it tries to bundle.

If that works, then I guess a simple addition to the README explaining what to do when using webpack would be the entire solution. I'll see if the three example files work with that in place.

[Pomax]

Looks like even with isomorphic-fetch externalised (which itself works without a problem), something's still trying to require('request'), which causes webpack to fail.

resulting bundle:

...
    var _parser = __webpack_require__(302);
...

/***/ },
/* 302 */
/***/ function(module, exports, __webpack_require__) {

    var utils = __webpack_require__(303);
...

/***/ },
/* 303 */
/***/ function(module, exports, __webpack_require__) {

    var request = __webpack_require__(304);

Looking at where that parser is defined in the wikijs code, it seems wiki-infobox-parser is the true culprit.

[dijs]

Hmm.. do you think using either https://github.com/github/fetch or https://github.com/bitinn/node-fetch instead of https://github.com/matthew-andrews/isomorphic-fetch would fix the issue?

[Pomax]

no, those won't solve the wiki-infobox-parser issue. Externalizing the fetch requirement is actually entirely possible, but that leaves the real problem of the infobox parser using require etc.

[dijs]

OHH!!! I forgot about the parser.... Dang...

[dijs]

I can get rid of that... My bad. I only need a small part of that library.

[dijs]

Okay. Can you try test branch for me? https://github.com/dijs/wiki/pull/22

[Pomax]

yup, let me test that right now

[Pomax]

No bundling errors on the test files! Checking whether the codebase I was trying to use it in now also works...

[Pomax]

Hm, although I do see this now:

var Wiki = require('wikijs');
var wiki = new Wiki();

leads to the following error in the browser:

utils.js:39 Uncaught TypeError: Wiki is not a constructor

(the "utils.js" seems a red herring, that's just the file that performs the console log)

[Pomax]

ah, looks like the build system generates exports.default = Wiki;, but because webpack bundles the node code, things won't work with an (additional) module.exports = Wiki; (if I add that in, things work. At least, library wise. The wikipedia system of course generates a CORS error on regular use, but running it in a service worker should not be restricted by CORS)

[dijs]

Excellent, so what changes do I need to make, if any? On Wed, May 25, 2016 at 7:10 PM Mike Kamermans notifications@github.com wrote:

ah, looks like the build system generates exports.default = Wiki;, but because webpack bundles the node code, things won't work with an (additional) module.exports = Wiki; (if I add that in, things work. At least, library wise. The wikipedia system of course generates a CORS error on regular use, but running it in a service worker should not be restricted by CORS)

— You are receiving this because you commented.

Reply to this email directly or view it on GitHub https://github.com/dijs/wiki/issues/16#issuecomment-221744050

[Pomax]

probably just a small section in the README.md that explains that if you're using webpack, and you're experiencing problems, you probably want to use:

externals: {
  "isomorphinc-fetch": "fetch"
}

Although with the parser fix that might not even be necessary anymore (not tested that and currently not in a position to quickly test)

[dijs]

Thanks! On Wed, May 25, 2016 at 11:04 PM Mike Kamermans notifications@github.com wrote:

probably just a small section in the README.md that explains that if you're using webpack, and you're experiencing problems, you probably want to use:

externals: { "isometric-fetch": "fetch" }

— You are receiving this because you commented.

Reply to this email directly or view it on GitHub https://github.com/dijs/wiki/issues/16#issuecomment-221772242

[Pomax]

and you, too!