Automatic Entry + Pass-through

[ThomasBrierley]

My remaining problem is pass through but I want to give the full context of what i'm trying to achieve.

I want to use native ES6-modules in the browser for development (can do this in chrome 60 now), and have a build script that bundles up the modules into an IIFE for the rest of the world for deployment which works symetrically with native modules (i.e no surprises on build) . Dropping gulp-rollup into a stream is close but does not quite go all the way, specifically:

1. Browsers do not need an explicit entry point, entry points naturally emerge as the dependency tree is assembled. I've solved this by piping the same stream into a promise that does some super simple static analysis and returns an array of entries for gulp-rollup.

2. Modules and non-modules co exist in the browser, gulp-rollup consumes the entire stream and if a file is found to not be in the tree of an entry it is blackholed.

The closest I can get with # 2 is to let the solution to # 1 pick up non-modules as entry points... while this gets the file to pass through gulp-rollup it winds up as whatever format module has been specified, in the case of IIFE this breaks any globals that may have been shared across non-module scripts before.

Any thoughts?

[lemmabit]

Interesting. It sounds like what you really want is a single output (multiple entry points will give you multiple outputs) consisting of every file in the stream rolled together so that you get each of their side effects exactly once. Is this correct?

You mention modules and non-modules co-existing, which complicates execution order and strictness. Rollup isn't designed to handle non-modules. Does this matter to you, or could your "non-modules" be marked with type=”module” and still work? If it does matter, your best bet is probably to roll up your modules and then concatenate that onto your non-modules the old-fashioned way.

[ThomasBrierley]

It sounds like what you really want is a single output (multiple entry points will give you multiple outputs) consisting of every file in the stream rolled together so that you get each of their side effects exactly once. Is this correct?

No i'm fine with multiple outputs, I have an independent injection stage, sorry i'm probably not being very clear: I have a promise that can determine the entry points in any source stream, and if there happens to be multiple entries and multiple outputs that is also fine.

My problem is that I cannot easily filter out non-modules from the stream (it's possible but it's probably far easier to just make gulp-rollup behaviour passthrough). I guess ideally i'm asking for a "passThrough" option so only those that are determined to be part of an entry tree are plucked from the stream.

e.g stream containing following is passed to gulp-rollup:

some.js // not a module
entryA.js
    \
    dep1.js
        \
        dep2.js
another.js // not a module
entryB.js
    \
    dep3.js
        \
        dep4.js

my promise can identify entryA.js and entryB.js automatically and provide to gulp-rollup. Rollup would then ideally output to:

some.js // untouched as was not part of any tree
entryA.js // bundled by rollup 
another.js // untouched as was not part of any tree
entryB.js // bundled by rollup

[lemmabit]

Okay, I see. Adding a new option to gulp-rollup for this would go against the gulp philosophy of giving each part as few responsibilities as possible. The gulp-if plugin will probably suit your needs and shouldn't be significantly more difficult to use than a passthrough option on gulp-rollup.

[ThomasBrierley]

Thanks, gulp-if does get it working how I wanted, although I end up having to parse everything twice which is a bit dirty.

Adding a new option to gulp-rollup for this would go against the gulp philosophy of giving each part as few responsibilities as possible

I totally get that, and you are right, to expect gulp-rollup to be a general filter as well is unreasonable of me, e.g no one expects uglify to passthrough non-js files...

Thinking about it more clearly now, It's only from the point of view of automatic entry detection with a mixed JS module source that it makes sense to passthrough pieces as part of that process. Because that info is a free byproduct that you can use to push files back into the stream with.

I've been trying to do this separately, but it will inevitably end up with a less elegant pipeline because it has to wrap like gulp-if or split and merge streams at the same time as feeding promises into the middle. It's looking more and more like something that needs to be solved more holistically as part of the same module.

In which case I guess ultimately i'm asking: would you be interested in including automatic entry detection in gulp-rollup? because passthrough would only work as part of that option, rather than be a general filter.

[EDIT]

I'm currently trying a different more simple approach: just wrap your plugin for a neater package and to share duplicate work between the filtering stage and the analysis stage internally. Maybe this will still allow for separation without making it a mess to use.

[ThomasBrierley]

Nevermind, I think what I am trying to achieve is a specific use case for rollup, but your plugin is a more general wrapper for rollup in gulp and I should not push to integrate it. Thanks for your help and sorry for the monologue :smile:

[ThomasBrierley]

In-case it is of any interest...

I ended up providing this functionality in a separate gulp plugin (much more basic than gulp-rollup) i've been using it for about 5 months without issue:

https://www.npmjs.com/package/gulp-auto-rollup

It doesn't utilise a full parser as import export keywords are only used at the root level so it's possible to get away with plain regex.

A vital counter part for utilising ES6 modules in develop mode is a script reference injector, I am using gulp-inject and needed to extend the js transform with:

inject.transform.html.js = (path, file) => {
    file = file.contents.toString();
    let test = /(?:^|\n)\s*(import|export)\s+/;
    let type = test.test(file) ? 'module' : '';
    return `<script type="${type}" src="${path}"></script>`;
};

This adds the appropriate type="module" attribute when the bundling via gulp-auto-rollup is turned off for develop mode (i.e serving), this makes it seamless to bundle in the build but not the serve task. I found this important because bundling slows down the build so I didn't want it slowing down serve (all modern browser support ES6 modules now)

[lemmabit]

@ThomasBrierley I've been wondering all this time, is it really necessary not to treat your non-import/export-laden code as modules? In almost all modern projects, it ought to be perfectly okay to just treat every file as a module. Seems a lot easier.

[ThomasBrierley]

It would be nice but I don't think it works.

When loading a normal script as a module there are two major differences that may cause it to fail: async loading and global scope isolation.

Take jQuery as an example: It must load before other scripts that use it's global $ reference. The first problem is that this library may declare this global implicitly const $ = blah or explicitly attach it to the window object window.$ = blah both will work in a normal script but only the later will expose the global in a module. Secondly even if the global is exposed via window, it is not part of the dependency tree because import/export is not being used... as a result there is no guarantee that it will load before other scripts referencing it's global.

This isn't pure speculation, i've tried going down this route and it's frustrating until you realise it can't possibly work (with ES6-modules alone at least).

.. Those problems are more resolved by that gulp-inject transform than my plugin (the only reason I need to differentiate at all is because I don't want to use rollup when developing).

The thing I felt was the most irritating that made me do the gulp-auto-rollup is that rollup does not implicitly determine the root node of the tree or "entry point" as they call it, even though the browser does this. This is probably more to do with my preferences, I like the build to not have explicit project specific details baked into it where it's unecessary.

[lemmabit]

@ThomasBrierley I don't know, I've never had any problems with this sort of thing. I'd just refactor my code so it's all exports instead of globals. If you depend on other people's code and can't refactor it, there are Rollup plugins that can help, like rollup-plugin-legacy.

I'll note that modules don't load async by default; they load defer, and defer scripts execute in the order in which they appear in the document.

The thing I felt was the most irritating that made me do the gulp-auto-rollup is that rollup does not implicitly determine the root node of the tree or "entry point" as they call it, even though the browser does this.

I don't think the browser does that. I think it just executes every module script so you get all of their side effects exactly once, like I suggested in my first comment. It would be pretty easy to do this with a Rollup plugin.

As a whole, I'm not really sold on all this fragile, auto-magical stuff you seem to be dealing in. But hey, I can't tell you what to do.

[ThomasBrierley]

You are of course correct, measures can be taken to "wrap" libraries out of my control. My choice is entirely subjective, for my use case I wanted to allow my colleagues to transition easily (both in terms of existing projects and learning how to use modules along side existing methods without the overhead of manually wrapping non-modules).

In the build system this is for, you can literally just plonk modules and normal scripts anywhere you want and it works seamlessly without any configuration or manual references, we've been doing this for months now and it allows everyone to just focus on coding.

In the near future I've no doubt this will become moot anyway as modules become the standard way to do things in both development and production.

I don't think the browser does that. I think it just executes every module script so you get all of their side effects exactly once, like I suggested in my first comment. It would be pretty easy to do this with a Rollup plugin.

It must first obtain a valid execution order with a DAG sort, and then there is no where that you need to declare "this is the root of my dependency tree" with native modules because it's implicit to the dependency resolution. I can objectively say: there is nothing fragile about finding the roots of an acyclic graph, just because it's implicit doesn't mean it's necessarily non-deterministic.

Anyway, we don't need to agree, I just wanted to let you know how it panned out in case it is, or becomes useful to you.