Config: syntax error from "sources" with hash-bang

[webpro]

Configuration requires to load all files that might be needed. E.g. sources: ["src/**/*.js", "lib/**/*.js"]. This might include a lot of unnecessary files (libraries may come with a large number of files that are not needed to use it), so it's not great performance wise. But the actual issue is that some matching .js files are Node executables, and thus start with a hash-bang (i.e. #!/usr/bin/env node), causing a fatal error for Buster ([Fatal] Syntax error).

• A possible solution would be to lazily serve files when requested by the AMD loader. But perhaps things work synchronously, so all files would need to be loaded upfront.
• Another is to check if the file is an executable, but hash-bangs might still be missed for incorrect non-executables.
• Or parse files and read the first line.

[augustl]

Interesting, thanks. We haven't thought of the fact that Node.JS files can also be considered executables with shebangs.

I assume your code is Node.JS only, since you have shebangs? In that case, you don't need to specify anything in "src" to run tests, since the tests can be set up to require the implementation it needs - Node.JS, unlike browsers, has a module system by default that you can use :)

This doesn't solve the problem of using things like buster-autolint and other plugins that uses the contents of "src" to determine which files that should be linted. With that in mind, I suppose the best solution is for the Node.JS runner to simply not require anything in "src" since it doesn't need to.

[webpro]

I assume your code is Node.JS only, since you have shebangs?

Actually in this case my code was for the browser. However, any (external) lib included might have a Node executable somehow (for or whatever reason, usually tests), even though the lib itself is also (mainly) targeting browsers.

[augustl]

Would a possible solution to this problem be to add a way to require specific files from a npm module in a Buster.JS config file? So that we can ask the npm module for "give me the path to the file for require('foo/bar')", instead of globbing everything in the entire module.

[webpro]

I'd say lazy loading would be optimal (both performance and configuration wise). Otherwise users might potentially end up listing a lot of files from a lot of libs.

[augustl]

Good point, I almost forgot my first proposal, to not automatically require everything in "src" on node. We could just resolve the globs, and let it be up to the plugins what to do with the paths. There's no need for Buster's node-runner itself to require "src" .

[dkl-ppi]

As i understand, the problem isn't limited to node.js, but also exists for browser environment. I don't think searching for shebangs in the source files brings a big benefit. You can also get into trouble if you try to load a normal node.js file in the browser, for example the buster.js configuration file. In my opinion buster.js isn't responsible to ensure that only the files are loaded which are compatible with the current environment. Instead, the files should be written to be compatible for node.js and browser, organized in separated folders and configured only for the compatible environment or excluded explicitly. For the current case this means you have to put the executable files in separate folders and exclude them.

[webpro]

Instead, the files should be written to be compatible for node.js and browser, organized in separated folders and configured only for the compatible environment or excluded explicitly.

You just don't have that control over external dependencies. And for sure you don't want to configure a gazillion exception in the globs. I think the only real solution is lazy loading.

[augustl]

Allright, time to fix this ;)

If you're testing node only, you have the option of simply not specifying any files for "src" in the config file. They're not really needed for node testing.

But I can't think of any reason why we would require stuff in "src" in the node runner. Will investigate.

[dominykas]

Um. Node happily eats up shebangs, so "src" or no "src" should not be an issue.

As for the browser... Don't do it. Don't load invalid JS files in the browser. You can't lazy load without forcing AMD, and one of the best things about buster is that (unlike, e.g. intern) it does not force AMD on you - do what you will.

However, if you're using AMD, then you don't need to include everything in "src" anyways - just include the seed and let the loader do its job - i.e. don't use buster itself. There's probably some magic you have to do to make the files available on buster-server (pretty sure I've seen that work?). And also turn off autorun for tests.

[augustl]

For the record, this is the error you get with buster-test -vv:

augustl@quark ~/c/b/test-test> buster-test -vv
Running tests: my tests
Loading:

Creating browser session
[Fatal] Syntax error
/src/my-file.js:1:1
#!/usr/local/bin/node
^
Pre-condition failed
Error: Pre-condition failed
    at analyzerError (/Users/augustl/code/buster/buster-test-cli/lib/run-analyzer.js:5:15)
    at Object.<anonymous> (/Users/augustl/code/buster/buster-test-cli/lib/run-analyzer.js:40:27)
    at notifyListener (/Users/augustl/code/buster/bane/lib/bane.js:54:35)
    at Object.object.emit (/Users/augustl/code/buster/bane/lib/bane.js:150:17)
    at emitIfFailed (/Users/augustl/code/buster/buster-analyzer/lib/analyzer.js:6:14)
    at countAndEmit (/Users/augustl/code/buster/buster-analyzer/lib/analyzer.js:13:5)
    at Object.bane.createEventEmitter.fatal (/Users/augustl/code/buster/buster-analyzer/lib/analyzer.js:26:9)
    at Object.processor (/Users/augustl/code/buster/buster-syntax/lib/buster-syntax.js:15:27)
    at /Users/augustl/code/buster/ramp-resources/lib/resource.js:126:26
    at Array.reduce (native)

[augustl]

Some kind of lazyness seems to be the solution for this issue. Lazyness would also be better in terms of performance. That would leave the problem of node.js files with shebangs unsolved for those that doesn't use AMD, but I think we can live with that. I don't want to add a hack for somehow ignoring files that node can handle and browsers can't when globbing up node.js files for the browser.

This lazyness will have one downside: it won't fail fast. A syntax error can only be detected when the file is actually loaded. But I think that's a sensible tradeoff.

Adding this lazyness is also a fun technical challenge, so hopefully it can ignite a spark of enthusiasm in my otherwise (unfortunate) neglected relationship to Buster.