tschaub / gulp-newer

Pass through newer source files only
https://npmjs.org/package/gulp-newer
226 stars 24 forks source link

many:1 always returning that files are newer #75

Closed JacobDB closed 6 years ago

JacobDB commented 6 years ago

I'm trying to use many:1 mapping to only run my task if files have been modified. This works great for my styles task, but I'm unable to get it to work for my scripts task.

I have a folder like so:

This compiles to:

I have my dest in gulp-newer set like so:

newer("dev/assets/scripts/modern.js")

And yet, it's always passing through everything. I tried looking through the source code for gulp-newer to see what was going on but couldn't figure it out. I'd appreciate some guidance.

JacobDB commented 6 years ago

Quick update on this, literally the same exact setup works fine for my styles task but not my scripts task. I've pasted the two tasks below.

You can see in that I'm retrieving the path to use as the destination in exactly the same way for both tasks, and then plugging the path in to gulp-newer in exactly the same way. It really doesn't may any sense to me.

styles task:

module.exports = {
    styles(gulp, plugins, ran_tasks, on_error) {
        // task-specific plugins
        const POSTCSS   = require("gulp-postcss");
        const SASS      = require("gulp-sass");
        const STYLELINT = require("gulp-stylelint");

        // styles task, compiles & prefixes SCSS
        return new Promise ((resolve) => {
            // set CSS directory
            const CSS_DIRECTORY = plugins.argv.dist ? global.settings.paths.dist + "/assets/styles" : global.settings.paths.dev + "/assets/styles";

            const ALL_FILE_NAMES   = plugins.fs.existsSync(CSS_DIRECTORY) ? plugins.fs.readdirSync(CSS_DIRECTORY) : false;
            const HASHED_FILE_NAME = ALL_FILE_NAMES ? ALL_FILE_NAMES.find((name) => {
                return name.match(new RegExp(CSS_DIRECTORY.split(".")[0] + ".[a-z0-9]{8}.css"));
            }) : "modern.css";

            // process styles
            gulp.src(global.settings.paths.src + "/assets/styles/**/*.scss")
                // prevent breaking on error
                .pipe(plugins.plumber({
                    errorHandler: on_error
                }))
                // check if source is newer than destination
                .pipe(plugins.newer({
                    dest: CSS_DIRECTORY + "/" + HASHED_FILE_NAME,
                }))
                // lint
                .pipe(STYLELINT({
                    debug: true,
                    failAfterError: true,
                    reporters: [
                        {
                            console: true,
                            formatter: "string",
                        },
                    ],
                }))
                // initialize sourcemap
                .pipe(plugins.sourcemaps.init())
                // compile SCSS (compress if --dist is passed)
                .pipe(plugins.gulpif(plugins.argv.dist, SASS({
                    includePaths: "./node_modules",
                    outputStyle:  "compressed",
                }), SASS({
                    includePaths: "./node_modules",
                })))
                // process post CSS stuff
                .pipe(POSTCSS([
                    require("pixrem"),
                    require("postcss-clearfix"),
                    require("postcss-easing-gradients"),
                    require("postcss-inline-svg"),
                    require("postcss-flexibility"),
                    require("postcss-responsive-type"),
                ]))
                // generate a hash and add it to the file name
                .pipe(plugins.hash({
                    template: "<%= name %>.<%= hash %><%= ext %>",
                }))
                // write sourcemap (if --dist isn't passed)
                .pipe(plugins.gulpif(!plugins.argv.dist, plugins.sourcemaps.write()))
                // output styles to compiled directory
                .pipe(gulp.dest(CSS_DIRECTORY))
                // notify that task is complete, if not part of default or watch
                .pipe(plugins.gulpif(gulp.seq.indexOf("styles") > gulp.seq.indexOf("default"), plugins.notify({
                    title:   "Success!",
                    message: "Styles task complete!",
                    onLast:  true,
                })))
                // push task to ran_tasks array
                .on("data", () => {
                    if (ran_tasks.indexOf("styles") < 0) {
                        ran_tasks.push("styles");
                    }
                })
                // generate a hash manfiest
                .pipe(plugins.hash.manifest("./.hashmanifest-styles", {
                    deleteOld: true,
                    sourceDir: CSS_DIRECTORY,
                }))
                // output hash manifest in root
                .pipe(gulp.dest("."))
                // resolve the promise
                .on("end", () => {
                    resolve();
                });
        });
    }
};

scripts task:

module.exports = {
    scripts(gulp, plugins, ran_tasks, on_error) {
        // task-specific plugins
        const ESLINT  = require("gulp-eslint");
        const GLOB    = require("glob");
        const WEBPACK = require("webpack-stream");

        // scripts task, lints, concatenates, & compresses JS
        return new Promise ((resolve) => {
            // set JS directory
            const JS_DIRECTORY = plugins.argv.dist ? global.settings.paths.dist + "/assets/scripts" : global.settings.paths.dev + "/assets/scripts";

            // set the source directory
            const SOURCE_DIRECTORY = global.settings.paths.src + "/assets/scripts";

            const WEBPACK_CONFIG = {
                mode:   "development",
                entry:   {
                    "critical":       GLOB.sync(SOURCE_DIRECTORY + "/critical/**/*.js"),
                    "legacy":         GLOB.sync(SOURCE_DIRECTORY + "/legacy/**/*.js"),
                    "modern":         GLOB.sync(SOURCE_DIRECTORY + "/modern/**/*.js"),
                    "service-worker": GLOB.sync(SOURCE_DIRECTORY + "/service-worker/**/*.js"),
                },
                output: {
                    path:     plugins.path.resolve(__dirname, JS_DIRECTORY),
                    filename: "[name].js",
                },
            };

            // update webpack config for the current target destination and file name
            WEBPACK_CONFIG.mode = plugins.argv.dist ? "production" : WEBPACK_CONFIG.mode;

            const ALL_FILE_NAMES   = plugins.fs.existsSync(JS_DIRECTORY) ? plugins.fs.readdirSync(JS_DIRECTORY) : false;
            const HASHED_FILE_NAME = ALL_FILE_NAMES ? ALL_FILE_NAMES.find((name) => {
                return name.match(new RegExp(JS_DIRECTORY.split(".")[0] + ".[a-z0-9]{8}.js"));
            }) : "modern.js";

            gulp.src(global.settings.paths.src + "/assets/scripts/**/*")
                // prevent breaking on error
                .pipe(plugins.plumber({errorHandler: on_error}))
                // check if source is newer than destination
                .pipe(plugins.newer({
                    dest: JS_DIRECTORY + "/" + HASHED_FILE_NAME,
                }))
                // lint all scripts
                .pipe(ESLINT())
                // print lint errors
                .pipe(ESLINT.format())
                // run webpack
                .pipe(WEBPACK(WEBPACK_CONFIG))
                // generate a hash and add it to the file name
                .pipe(plugins.hash({template: "<%= name %>.<%= hash %><%= ext %>"}))
                // output scripts to compiled directory
                .pipe(gulp.dest(JS_DIRECTORY))
                // generate a hash manfiest
                .pipe(plugins.hash.manifest(".hashmanifest-scripts", {
                    deleteOld: true,
                    sourceDir: JS_DIRECTORY
                }))
                // output hash manifest in root
                .pipe(gulp.dest("."))
                // notify that task is complete, if not part of default or watch
                .pipe(plugins.gulpif(gulp.seq.indexOf("scripts") > gulp.seq.indexOf("default"), plugins.notify({
                    title:   "Success!",
                    message: "Scripts task complete!",
                    onLast:  true,
                })))
                // push task to ran_tasks array
                .on("data", () => {
                    if (ran_tasks.indexOf("scripts") < 0) {
                        ran_tasks.push("scripts");
                    }
                })
                // resolve the promise on end
                .on("end", () => {
                    resolve();
                });

        });
    }
};
JacobDB commented 6 years ago

I've now tested on three machines; two in WSL, one actual Ubuntu 18.04 machine, and all of them are triggering the task every time it's ran. Additionally, the Ubuntu machine is always tirggering the html task as well, and I saw something similar earlier with my media task. In the case of my media task, deleting my local repository and re-cloning fixed it, but that doesn't work in the case of scripts.

All this leads me to believe that there's some sort of underlying issue with this module that sometimes causes it to incorrectly identify some files as newer even when they're not.

JacobDB commented 6 years ago

If I add console.log(newer) at index.js:198, files are definitely correctly being identified as newer or older. If the files are older, I get a bunch of false in my console, and if any have been modified, I get a few true.

If this is returning false correctly, how can it still be passing the files to be processed?

JacobDB commented 6 years ago

Oh! I think I just realized what's happening. Webpack isn't actually reading the input files in the task, it's reading what I'm feeding it with the config! So that means that regardless of what gulp-newer is doing, it'll always run. That puts me on the right path, will track it down a solution in the morning.

JacobDB commented 6 years ago

Definitely not an issue with gulp-newer, closing.