Requiring JS and Coffee files

[juanghurtado]

Hi there! :wave:

I know which are the purposes of this plugin, but I'm curious to see how you'd solve this situation I'm involved.

Here's the deal:

I want to have a main file which includes all JS and Coffee files needed for the app. Something like this:

#= require 'file1.js'
#= require 'file2.coffee'
#= require 'file3.js'

Right now this is not possible using gulp-include and gulp-coffee because the inclusion of the *.js files will cause CoffeeScript compilation errors.

How would you solve this need? Maybe making a new Gulp plugin (gulp-include-coffee)? I don't like this approach because Gulp plugin are supposed to do just one thing, and do it well.

Maybe giving gulp-include an option to set the file extension which will be included? Something like:

gulp.src('./src/app/app.coffee')
  // First parse `app.coffee` file and include only `coffee` files
  .pipe(include({
    extension: '.coffee'
  }))
  // Compile the result with CoffeeScript
  .pipe(coffee())
  // Include the rest of the includes (in the example, plain old JS files)
  .pipe(include())
  .pipe(gulp.dest('./dist/'));

I don't know, I'm kind of lost with this… What do you think?

[wiledal]

Hey Juan!

I ran into a similar situation myself when working on a project which included vendor scripts (jquery, spine and three) before the coffee application. I solved it by having the js-includes in a separate file which I manually appended with gulp-header.

My vendor.js simply included require-comments for the js:

//= require vendor/lib/jquery.js
//= require vendor/lib/modernizr.js
//= require vendor/spine/spine.js
...

And the gulpfile:

gulp.task("scripts", function() {
    gulp.src("coffee/app.coffee")
        .pipe(include()) //  Include all coffeescript
        .pipe(coffee()) // Compile
        .pipe(header(fs.readFileSync("js/vendor.js"))) // Prepend js include-comments
        .pipe(include()) // Include all js
        .pipe(gulp.dest("public/javascripts"));
});

I must say though, I think your proposed solution makes sense for this plugin and would come in handy. Also, implementing it will not leave a big footprint.

I will have some time today to look into making some improvements.

[juanghurtado]

That is exactly the same scenario I'm in: vendor scripts with my own CoffeeScript project. I thought on including a vendor.js file, but I was sure there should be an alternative using a single file.

If you don't have the time to implement that option config I could do it myself. Give me a shout if you need me! :+1:

[wiledal]

I implemented this update together with your include_tree-extension (haven't pushed it yet), however I'm afraid this doesn't solve our troubles.

It appears CoffeeScript removes all comments when compiling which makes the second include() useless. I did figure out another way to include js-files:

`
#= require vendor/modernizr.js
#= require vendor/jquery.js

`

#= require app/AppStuff.coffee
#= require app/MoreAppStuff.coffee

Since CoffeeScript allows embedded JavaScript by using backsticks () we can just wrap the inclusion of those files and it will pass throughcoffee()` just fine.

I will still include the extensions-update since it could be useful in other ways.

[juanghurtado]

Nice catch @wiledal! The backtick-thing is a good solution for our problem. No need to modify gulp-include and uses a CoffeeScript feature. Perfect!

Thank you for your time :smile:

I'll close the issue now. Thank you!

[juanghurtado]

Hi there again @wiledal! Just a quick note:

The backtick solution doesn't always work. Imagine the following scenario:

`
//= require 'vendor/jquery.js'
`
#= require 'some-coffee-file.coffee' 

The file jquery.js (from Bower distribution) has comments with backticks:

// Discard any remaining `private` data

So the CoffeeScript compiler gets confused about what to do after the inclusions has been made.

I'm trying to imagine an alternative solution for this.

Reopening issue until we find out a way.

PS. This is not an issue with gulp-include!

[juanghurtado]

My current solution involve using a two-step include process. Given this file:

`
//= require "vendor/jquery/jquery.js"
`
#= require "modules/a.coffee"

And this gulpfile.js:

gulp.src("app.coffee")
    .pipe(include({
        extensions: "coffee"
    }))
    .pipe(coffee())
    .pipe(include({
        extensions: "js"
    }))

This is what we do:

• First we include only .coffee files, resulting in a CoffeeFile with included Coffee content, and JS comments with the JS requires for gulp-include
• Then compile the CofeeScript, resulting in a JS file with Coffee compiled and JS comments with the JS requires for gulp-include (note the backticks around the JS include comments on the initial Coffee file)
• Finally include JS files

With this solution I have no problems including JS and Coffee on the same file, even if the JS files have comments with backticks…

Not nice, but working.

Do you have any other solution?

[thasmo]

Also had problems including vanilla JS files into coffee script files and ended up using your approach, which works well but feels like what it is, a hack. But I think it's the best solution for now.

Something regarding escaping backtips was mentioned here: https://github.com/jashkenas/coffee-script/issues/3265

Maybe it's possible to automatically escape backticks, if

• the processed file is a coffeescript file and
• the file to be included is a vanilla javascript file.

Thoughts about this?

[thasmo]

Or a new option which enables/disables backtick-escaping.

[sindrenm]

Hi @juanghurtado, @thasmo,

I'm also struggling with this issue, but I can't seem to make it work with your workaround/hack. This is my setup:

gulpfile.js:

var gulp    = require("gulp");
var coffee  = require("gulp-coffee");
var include = require("gulp-include");

gulp.task("scripts", function () {
  gulp.src("./scripts/application.coffee")
    .pipe(include({ extensions: 'coffee' }))
    .pipe(coffee())
    .pipe(include({ extensions: 'js' }))
    .pipe(gulp.dest("./public/javascripts/"));
});

./scripts/application.coffee:

`
//= include ../bower_components/jquery/dist/jquery.js
`

#= include ./angular/config.coffee

Your solution makes sense to me, so I don't really get why this is happening, but I'm getting the same error that started all of this, and it seems like the CoffeeScript compiler tries to also compile the file after the third pipe:

$ gulp scripts
[16:38:28] Using gulpfile ~/.../gulpfile.js
[16:38:28] Starting 'scripts'...
[16:38:28] Finished 'scripts' after 6.23 ms

/home/.../node_modules/gulp-include/node_modules/event-stream/node_modules/map-stream/index.js:103
        throw err
              ^
/home/.../scripts/application.coffee:5170:33: error: reserved word "private"
                                                // Discard any remaining `private` data
$

So it seems as if gulp-coffee won't stop compiling. Now, I don't have much experience with Gulp (this is my first project using it), so I'm not an expert on how the streams work and if they're synchronous or asynchronous etc, but I'm thinking that the coffee() stream doesn't end when I think it does and that the pipes are not synchronous.

I also tried creating two different tasks, like this:

gulp.task("coffee", function () {
  gulp.src("./scripts/application.coffee")
    .pipe(include({ extensions: 'coffee' }))
    .pipe(coffee())
    .pipe(gulp.dest("./scripts/"));
});

gulp.task("js", function () {
  gulp.src("./scripts/application.js")
    .pipe(include({ extensions: 'js' }))
    .pipe(gulp.dest("./public/javascripts/"));
});

Running the tasks after each other manually works, like this:

$ gulp coffee
[16:52:20] Using gulpfile ~/.../gulpfile.js
[16:52:20] Starting 'coffee'...
[16:52:20] Finished 'coffee' after 5.99 ms
$ [16:52:28] Using gulpfile ~/.../gulpfile.js
[16:52:28] Starting 'js'...
[16:52:28] Finished 'js' after 5.21 ms

It creates ./public/javascripts/application.js correctly. However, running the two tasks in one command fails:

$ gulp coffee js                        
[16:54:41] Using gulpfile ~/.../gulpfile.js
[16:54:41] Starting 'coffee'...
[16:54:41] Finished 'coffee' after 6.1 ms
[16:54:41] Starting 'js'...
[16:54:41] Finished 'js' after 1.13 ms

/home/.../node_modules/gulp-include/node_modules/event-stream/node_modules/map-stream/index.js:103
        throw err
              ^
/home/.../scripts/application.coffee:5170:33: error: reserved word "private"
                                                // Discard any remaining `private` data

This leads me to believe that the tasks are not synchronously running as well. After doing more research I found that streams emit an end event when finished. Knowing this, I tried the following:

gulp.task("scripts", function () {
  var vendor = function () {
    gulp.src("./scripts/application.js")
      .pipe(include({ extensions: 'js' }))
      .pipe(gulp.dest("./public/javascripts/"));
  };

  gulp.src("./scripts/application.coffee")
    .pipe(include({ extensions: 'coffee' }))
    .pipe(coffee())
    .pipe(gulp.dest("./scripts/")).on("end", vendor);
});

Running this one task succeeds and creates the file correctly. However, this feels like a super ugly hack and it also leaves a temp file in ./scripts/application.js – which I can remove afterwards using another script, I suppose, but I'd rather have a prettier solution altogether.

So, any thought as to why it doesn't work as it should? Did any of you experience the same thing? Am I missing something completely obvious here?

Thanks!

EDIT: Oh, and this is from my package.json:

{
  "devDependencies": {
    "gulp": "^3.8.7",
    "gulp-coffee": "^2.1.1",
    "gulp-include": "^1.0.1"
  }
}

[lukehorvat]

@sindrenm I have a gulpfile very similar to the one at the top of your post, and I get the same errors about "reserved words" using gulp-include v1.1.0. On a whim, I decided to try an earlier version (v0.2.3) and it works! So there was something introduced between those two versions that is causing this...

@wiledal Any ideas?

[jasford]

I know this thread is a little old, but why not just use gulp-merge and gulp-concat to accomplish what you are looking for. This is how I do it:

var gulpMerge = require('gulp-merge');
var concat = require('gulp-concat');
var coffee = require('gulp-coffee');

gulpMerge(
  gulp.src('./assets/js/vendor/*.js'),
  gulp.src('./assets/js/app/*.coffee').pipe(coffee()),
  gulp.src('./assets/js/home.js')
)
  .pipe(concat('main.js'))
  .pipe(gulp.dest('dist/js'));

This starts up multiple gulp streams and then merges them back together when they are completed. They are then combined to a single file with gulp-concat before outputing to the dist folder.

[josh-makalo]

Hi there! Being aware that I'll likely be bashed for reviving such an old bug report, I'd like to offer my two cents to this as well.

How about offering an option to convert certain filetypes before inclusion? Obviously gulp-include wouldn't do the actual conversion itself, instead the user needs to pass the respective converter in the options array. So the user would write something like this:

var gulp = require('gulp');
var coffee = require('gulp-coffee');
var js2coffee = require('gulp-js2coffee');
var include = require('gulp-include');

gulp.src('input.coffee')
  .pipe(include({convert: {"js": js2coffee()}}))
  .pipe(coffee())
  .pipe(gulp.dest('dist'));

gulp-include would then open a new stream for each file to be converted, use the converter as a pipe step and include the stream output into the "input.coffee". In my opinion, this approach adds a powerful and flexible feature to gulp-include without much effort, doesn't require the inclusion syntax to be changed, and solves the original problem. Any thoughts on this?

Edit: If one doesn't like the idea of converting js to coffeescript, instead a converter could be used that wraps the js file in backticks and escapes any backticks within the file.

[wiledal]

I have completely abandoned Coffeescript since the beginning of this plugin, so I'm not super into finding solutions for this.

I do however do some babel-conversions and had issues with mixing es5 vendor files with my application files, which is basically the same issue. You don't want babel to recompile ThreeJS :).

My solution ended up being a separate compile and concat solution:

var gulp = require('gulp'),
    babel = require('gulp-babel'),
    merge = require('gulp-merge'),
    include = require('gulp-include'),
    concat = require('gulp-concat');

gulp.task('js', function() {
  var vendor = gulp.src('src/vendor.js')
    .pipe(include());

  var appjs = gulp.src('src/main.js')
    .pipe(include())
    .pipe(babel({
      presets: ['es2015']
    }))
      .on('error', throwError);

  return merge(vendor, appjs)
    .pipe(concat('all.js'))
    .pipe(gulp.dest('dist/js/'));
});

It works well and solves any issues regarding cross-compilation of different file types.

[KenEucker]

Revival:

@juanghurtado any chance you're still interested in this issue? I see that many people jumped in after you experiencing the same problem.

I also see that there are solutions to getting around this problem without fully relying on gulp-include to make all the things play nicely with each other.

I'm looking for anyone who is presented with this problem, or just feels like they have a solution they can provide, that may be available to help. If not with code then with a test repository that reproduces the issue for others to hack away on. At the very least, it would be good to know if there is anyone who still feels that this problem should be resolved by gulp-include.

This issue can stay open for now, but if time passes and no interest is expressed we will archive this request.

[juanghurtado]

Long time without using Gulp and/or Coffeescript. Can't help much, really.

[KenEucker]

@juanghurtado No worries, It's totally understandable given that this was first created 5 years ago. Thank you for responding.

I'm going to close this issue for now as I believe that @wiledal's latest comment shows a solution that I think would be a better route to go than to have gulp-include cover this edge case. If this becomes more than an edge case in the future we can reconsider a better solution.