NCEAS / metacatui

MetacatUI: A client-side web interface for DataONE data repositories
https://nceas.github.io/metacatui
Apache License 2.0
42 stars 26 forks source link

Use a build tool to improve performance #2194

Open robyngit opened 10 months ago

robyngit commented 10 months ago

MetacatUI currently relies on RequireJS for asset loading. We aim to improve performance by integrating a modern bundler such as Webpack or Parcel. The goal is to improve MetacatUI's performance and load time through minification, uglification, and bundling of JavaScript files and other assets (e.g. CSS files), as well as code-splitting and tree-shaking techniques.

Although we would eventually like to replace RequireJS with ES6 modules, as a first step, we want RequireJS to co-exist with the new bundler. This way, we preserve full backward compatibility, allowing repositories to choose whether or not to adopt the new build system. Likely, the bundled code will be organized in a new /dist/ directory, distinct from the current /src/ directory.

Considerations:

Metrics for Evaluating App Performance:

To measure the success of performance improvements, we can focus on:

  1. Load Time: Measure how quickly the app is ready for use.
  2. Interactivity: Assess the time it takes for all app features to be fully functional.
  3. First Content Display: Track the time until the first content appears on the screen.

Criteria for Choosing a Bundler

Must-Have:

  1. Works with RequireJS: The bundler should operate alongside RequireJS to maintain backward compatibility.
  2. Optimizes Performance: Should excel in reducing code size and improving loading through techniques like minification and code-splitting.
  3. Supports Dynamic Loading: Must be able to handle dynamic script loading similar to our existing setup.
  4. HTML Template Handling: Should allow for the bundling of HTML templates in a manner compatible with our JavaScript.
  5. CSS Optimization: At a minimum, the bundler should offer CSS minification.

Nice-to-Have:

  1. Quick Builds: Faster compilation times are a bonus.
  2. Polyfill Options: The capability to manage or even replace our current polyfill file.
  3. Cache Control: Built-in cache management features would be beneficial.
  4. Environment Variables: If it can manage API keys securely, that's a plus.
  5. Community and Longevity: Signs of strong community support and future maintainability are advantageous.

Bundler Options

Two bundlers seem to be the most promising candidates:

  1. Webpack: Given that we need backwards compatibility with RequireJS, Webpack seems to be the most suitable option. Webpack is mature, highly configurable, and has a rich plugin ecosystem. The downside is that it can be challenging to configure and has a steep learning curve.
  2. Parcel: Parcel could also be a strong candidate if we're willing to compromise on some level of control for ease of use and speed of setup, since it aims to be zero-config. We need to evaluate if it is suitable for AMD setups with RequireJS.

Bundlers ruled out:

Next Steps

  1. Investigate Other Bundlers: Explore other bundlers that could be suitable for our use case.
  2. Shortlist Bundlers: Compare bundlers using our criteria to create a shortlist.
  3. Test Compatibility: Run initial tests with shortlisted bundlers for compatibility with our setup.
  4. Evaluate Performance: Use the metrics above to test the performance of compatible bundlers.
  5. Document Findings: Summarize performance data, challenges, and a potential implementation plan.
  6. Pick a Bundler: Choose the most suitable option.
  7. Implement: Set up the new build process.
  8. Guide: Write a how-to guide for adopting the new build process in repositories.
robyngit commented 10 months ago

Note that RequireJS also has an optimization tool that we have considered using in the past.

robyngit commented 10 months ago

Initial experiments with webpack

My experimental `webpack.config.js` ```javascript const path = require("path"); const HtmlWebpackPlugin = require("html-webpack-plugin"); const CopyPlugin = require("copy-webpack-plugin"); const baseUrlMetacatUI = path.resolve(__dirname, "src", "js"); const baseUrl = path.resolve(__dirname, "src"); const recaptchaURL = "https://www.google.com/recaptcha/api/js/recaptcha_ajax"; module.exports = { entry: "./src/js/app.js", output: { path: path.resolve(__dirname, "dist"), filename: "js/app.js", }, resolveLoader: { alias: { text: "text-loader" } }, resolve: { preferRelative: true, extensions: [".js"], alias: { collections: path.resolve(baseUrlMetacatUI, "collections"), common: path.resolve(baseUrlMetacatUI, "common"), models: path.resolve(baseUrlMetacatUI, "models"), routers: path.resolve(baseUrlMetacatUI, "routers"), templates: path.resolve(baseUrlMetacatUI, "templates"), views: path.resolve(baseUrlMetacatUI, "views"), themes: path.resolve(baseUrlMetacatUI, "themes"), img: path.resolve(baseUrl, "img"), jquery: path.resolve(baseUrl, "components/jquery-1.9.1.min"), jqueryui: path.resolve(baseUrl, "components/jquery-ui.min"), jqueryform: path.resolve(baseUrl, "components/jquery.form"), underscore: path.resolve(baseUrl, "components/underscore-min"), backbone: path.resolve(baseUrl, "components/backbone-min"), localforage: path.resolve(baseUrl, "components/localforage.min"), bootstrap: path.resolve(baseUrl, "components/bootstrap.min"), text: path.resolve(baseUrl, "components/require-text"), // <-- ? jws: path.resolve(baseUrl, "components/jws-3.2.min"), jsrasign: path.resolve(baseUrl, "components/jsrsasign-4.9.0.min"), async: path.resolve(baseUrl, "components/async"), recaptcha: [recaptchaURL, "scripts/placeholder"], nGeohash: path.resolve(baseUrl, "components/geohash/main"), fancybox: path.resolve( baseUrl, "components/fancybox/jquery.fancybox.pack" ), //v. 2.1.5 annotator: path.resolve( baseUrl, "components/annotator/v1.2.10/annotator-full" ), bioportal: path.resolve( baseUrl, "components/bioportal/jquery.ncbo.tree-2.0.2" ), clipboard: path.resolve(baseUrl, "components/clipboard.min"), uuid: path.resolve(baseUrl, "components/uuid"), md5: path.resolve(baseUrl, "components/md5"), rdflib: path.resolve(baseUrl, "components/rdflib.min"), x2js: path.resolve(baseUrl, "components/xml2json"), he: path.resolve(baseUrl, "components/he"), citation: path.resolve(baseUrl, "components/citation.min"), promise: path.resolve(baseUrl, "components/es6-promise.min"), metacatuiConnectors: path.resolve( baseUrlMetacatUI, "/js/connectors/Filters-Search" ), // showdown + extensions (used in the MarkdownView to convert markdown to html) showdown: path.resolve(baseUrl, "components/showdown/showdown.min"), showdownHighlight: path.resolve( baseUrl, "components/showdown/extensions/showdown-highlight/showdown-highlight" ), highlight: path.resolve( baseUrl, "components/showdown/extensions/showdown-highlight/highlight.pack" ), showdownFootnotes: path.resolve( baseUrl, "components/showdown/extensions/showdown-footnotes" ), showdownBootstrap: path.resolve( baseUrl, "components/showdown/extensions/showdown-bootstrap" ), showdownDocbook: path.resolve( baseUrl, "components/showdown/extensions/showdown-docbook" ), showdownKatex: path.resolve( baseUrl, "components/showdown/extensions/showdown-katex/showdown-katex.min" ), showdownCitation: path.resolve( baseUrl, "components/showdown/extensions/showdown-citation/showdown-citation" ), showdownImages: path.resolve( baseUrl, "components/showdown/extensions/showdown-images" ), showdownXssFilter: path.resolve( baseUrl, "components/showdown/extensions/showdown-xss-filter/showdown-xss-filter" ), xss: path.resolve( baseUrl, "components/showdown/extensions/showdown-xss-filter/xss.min" ), showdownHtags: path.resolve( baseUrl, "components/showdown/extensions/showdown-htags" ), // woofmark - markdown editor woofmark: path.resolve(baseUrl, "components/woofmark.min"), // drop zone creates drag and drop areas Dropzone: path.resolve(baseUrl, "components/dropzone-amd-module"), // Packages that convert between json data to markdown table markdownTableFromJson: path.resolve( baseUrl, "components/markdown-table-from-json.min" ), markdownTableToJson: path.resolve( baseUrl, "components/markdown-table-to-json" ), // Polyfill required for using dropzone with older browsers corejs: path.resolve(baseUrl, "components/core-js"), // Searchable multi-select dropdown component semanticUItransition: path.resolve( baseUrl, "components/semanticUI/transition.min" ), semanticUIdropdown: path.resolve( baseUrl, "components/semanticUI/dropdown.min" ), // To make elements drag and drop, sortable sortable: path.resolve(baseUrl, "components/sortable.min"), //Cesium cesium: path.resolve(baseUrl, "components/cesium/Cesium"), //Have a null fallback for our d3 components for browsers that don't support SVG d3: path.resolve(baseUrl, "components/d3.v3.min"), LineChart: path.resolve(baseUrlMetacatUI, "views/LineChartView"), BarChart: path.resolve(baseUrlMetacatUI, "views/BarChartView"), CircleBadge: path.resolve(baseUrlMetacatUI, "views/CircleBadgeView"), DonutChart: path.resolve(baseUrlMetacatUI, "views/DonutChartView"), MetricsChart: path.resolve(baseUrlMetacatUI, "views/MetricsChartView"), }, }, module: { rules: [ { test: /\.js$/, exclude: /node_modules/, }, // text-loader { test: /\.txt$/, use: "text-loader", }, { test: /\.svg$/i, use: "raw-loader", }, ], }, plugins: [ new HtmlWebpackPlugin({ template: "./src/index.html", }), new CopyPlugin({ patterns: [ { from: "./src/components/require.min.js", to: "./components/require.min.js", }, { from: "./src/js/themes", to: "./js/themes", }, { from: "./src/loader.js", to: "./loader.js", }, { from: "./src/config/config.js", to: "./config/config.js", }, { from: "./src/js/polyfill.js", to: "./js/polyfill.js", } ], }), ], externals: ["views/RegistryView", "gmaps", "cesium", "require"], }; ```
robyngit commented 10 months ago

TL;DR: Good news: Minifying with Gulp is very easy. Bad news: Minifying the JS files has little impact on performance.

Steps to set up Gulp

Setting up Gulp to minify the JS was very straight-forward compared to attempting to bundle and minify with Webpack.

  1. Install Gulp Globally (optional, but makes it easier to run gulp from the command line):

    npm install -g gulp-cli
  2. Install Gulp Locally:

    npm install --save-dev gulp
  3. Install Required Gulp Plugins: Install the gulp-terser and gulp-tap plugins using npm.

    npm install --save-dev gulp-terser gulp-tap
  4. Create The Gulpfile (gulpfile.js):

    gulpfile.js
const gulp = require('gulp');
const terser = require('gulp-terser');
const tap = require('gulp-tap');

gulp.task('copy-all', function() {
    // Copy all files from src to dist
    return gulp.src('src/**/*')
        .pipe(gulp.dest('dist'));
});

gulp.task('minify-js', function() {
  return gulp.src('dist/**/*.js')
      .pipe(tap(function(file) {
          return gulp.src(file.path, { base: 'dist' })
              .pipe(terser())
              .on('error', function(err) {
                  console.warn(`Error in file ${file.path}: ${err.toString()}`);
                  this.emit('end');
              })
              .pipe(gulp.dest('dist')); // Save it back to the dist directory
      }));
});

gulp.task('build', gulp.series('copy-all', 'minify-js'));

  1. Run the Build Task: Execute the build task to copy all files from src/ to dist/ and then minify all JavaScript files.

    gulp build
  2. Switch the dev server to run from dist rather than src: In server.js switch const src_dir = "src"; to const src_dir = "dist";

Performance differences

I ran Chrome's lighthouse test twice: once with files served from src (unminified JS), and once files saved from dist (minified JS). The difference in performance was disappointing:

Un-minified files

Overall performance with unminified files is 12

Minified files

Overall with minified files is 13

Conclusions

I think the recommendations detailed by Lauren are the tasks we should focus on in order to improve performance.

ianguerin commented 5 months ago

In considering Issue#224:

I've spent some time trying to see what it would take to migrate to a tool like Webpack. I think it would required migrating from require.js style modules to ES6 modules first. In the meantime, the r.js tool for optimizing and bundling does seem to be a significantly easier effort. I have a branch in my fork of this repo where I've used r.js to create one single bundle file that could be loaded in production instead (the commit). On my local machine this takes me from loading 186 JavaScript files (186 http requests) to loading only 17. I haven't been able to get minification to work (the single JS file is almost 8MB!!) but I think with some more effort it should be possible to figure out the remaining issues that are blocking that.

robyngit commented 5 months ago

Nice! Thank you for looking more into this @ianguerin. Did you measure or notice any differences in load time? I'd guess even with minification, those 17 files might be too large

ianguerin commented 5 months ago

I did not see a significant overall "score" using the Chrome DevTools Lighthouse extension, though it did get better. I did see an increase in time to first contentful load (bad) I've attached my two lighthouse runs below, before my changes and after my changes.

Before.pdf After.pdf

I wouldn't recommend going through this risky of a change for such a minor improvement, but I think playing around with some of the configuration rules, maybe [modules configuration for r.js] (https://github.com/requirejs/r.js/blob/acec5366eb9094e670b6d1a87457634e74d6384e/build/example.build.js#L355) could be beneficial, and this commit has at least a PoC to getting that to work.