search

xp-forge / frontend

Web frontends
1 stars 1 forks source link

Bundle dependencies #11

Closed thekid closed 3 years ago

thekid commented 3 years ago

Our goal

-    <link href="https://cdn.jsdelivr.net/npm/fomantic-ui@2.8.7/dist/semantic.min.css" rel="stylesheet">
-    <link href="https://cdn.jsdelivr.net/npm/simplemde@1.11.2/dist/simplemde.min.css" rel="stylesheet">
+    <link href="/static/bundle.de63f89.css" rel="stylesheet">

-    <script src="https://cdn.jsdelivr.net/npm/jquery@3.6.0/dist/jquery.min.js"></script>
-    <script src="https://cdn.jsdelivr.net/npm/fomantic-ui@2.8.7/dist/semantic.min.js"></script>
-    <script src="https://cdn.jsdelivr.net/npm/handlebars@v4.7.6/dist/handlebars.js"></script>
-    <script src="https://cdn.jsdelivr.net/npm/simplemde@1.11.2/dist/simplemde.min.js"></script>
+    <script src="/static/bundle.de63f89.js"></script>

This way, we reduce the number of HTTP requests made to our site - including immutable caching, we can ensure best performance.

Standard bundling procedure

We would need to:

For applications with heavier JavaScript and/or TypeScript usage, you'd probably already have this toolchain installed and ready.

Question

The question is if we want to provide a more light-weight alternative - as bundling JS and CSS is basically concatenating the files together in a huge bundle.[XXXX].(css|js) file.

thekid commented 3 years ago

Some ideas that get us half-way there:

However, all of these still require npm. There is one exception, https://github.com/fxpio/composer-asset-plugin (The Composer Asset Plugin allows you to manage project assets (css, js, etc.) in your composer.json without installing NPM or Bower.)

thekid commented 3 years ago

The following could work inside composer.json:

"require-assets": {
  "dropzone@^5.8": ["dist/dropzone.min.js"],
  "jquery@^3.6": ["dist/jquery.min.js"],
  "fomantic-ui@^2.8": ["dist/semantic.min.js", "dist/semantic.min.css"],
  "handlebars@^4.7": ["dist/handlebars.js"],
  "simplemde@^1.11": ["dist/simplemde.min.js", "dist/simplemde.min.css"],
  "transliteration@^2.1": ["dist/browser/bundle.umd.min.js"]
}

Can we create a composer plugin to fetch and bundle these when running composer up? It would do the following:

This is slimmer than downloading the complete package - we only need their assets from dist!

thekid commented 3 years ago

Standalone POC script using require-assets: https://gist.github.com/thekid/91401cf5a7130d5709da95caf25b4347 (w/o fingerprinting):

$ time xp bundle.script.php src/main/webapp/static/
dropzone@^5.8 => 5.8.1
> [js]: https://cdn.jsdelivr.net/npm/dropzone@5.8.1/dist/dropzone.min.js 115245
jquery@^3.6 => 3.6.0
> [js]: https://cdn.jsdelivr.net/npm/jquery@3.6.0/dist/jquery.min.js 89501
# ...

C:\Tools\...\webapp\static\bundle.js: 1293144 bytes
C:\Tools\...\webapp\static\bundle.css: 1373312 bytes

real    0m14.192s
user    0m0.000s
sys     0m0.000s

On subsequent runs, with requests now cached, the performance increases:

image

thekid commented 3 years ago

Comparison, here's my mileage with the following package.json:

{
  "dependencies": {
    "dropzone": "^5.8",
    "jquery": "^3.6",
    "fomantic-ui": "^2.8",
    "handlebars": "^4.7",
    "simplemde": "^1.11",
    "transliteration": "^2.1"
  }
}

image

After I couldn't get Fomantic UI to install using Windows, I opened a WSL shell to the same directory:

image

More than 2 minutes, 696 packages from 416 contributors, 93 MB disk space used in node_modules.

Finally, after adding the following to webpack.config.js:

const path = require('path');

module.exports = {
  mode: 'production',
  target: 'web',
  entry: './index.js',
  output: {
    filename: 'bundle.js',
    path: path.resolve(__dirname, 'src/main/webapp/static'),
  },
  externals: {
    fs: 'commonjs fs'
  }
};

...and using this inside index.js: 

require('dropzone');
require('handlebars');
require('jquery');
require('simplemde');
require('transliteration');
require('fomantic-ui');

...I get a bundle.js with almost the same filesize as the one I concatenated with PHP (it takes webpack 13 seconds to do this). However, loading it with the browser, I get the following:

image

After more of an hour of googling error messages, I give up 😖

thekid commented 3 years ago

After more of an hour of googling error messages, I give up 😖

Here's the answer that helped: https://stackoverflow.com/a/51595936

global.$ = global.jQuery = require('jquery');   // <-- This line!

require('dropzone');
require('handlebars');
require('simplemde');
require('transliteration');
require('fomantic-ui');
thekid commented 3 years ago

As to web fonts, I understand it's better to use them from Google instead of bundling them: https://developers.google.com/fonts/faq#will_web_fonts_slow_down_my_page

thekid commented 3 years ago

With bundle.css and bundle.js in place, we can reference them via <link href="/static/bundle.css" rel="stylesheet"> and <script src="/static/bundle.js"></script>. If we add fingerprinting (bundle.XXXX.(js|css)), we would also need to exchange references inside our HTML templates every time the fingerprint changes, at the risk of forgetting it. If we can pass the fingerprint somehow via variables, we could write something along the lines of the following inside Handlebars:

<link href="/static/bundle.{{assets.fingerprint}}.css" rel="stylesheet">

...or - better maybe as a) it keeps relative references inside the CSS files intact and b) allows the easy use of a CDN - using an asset path:

<link href="{{assets.path}}/bundle.css" rel="stylesheet">

...or - with even more control:

<link href="{{asset 'bundle.css'}}" rel="stylesheet">

How do other frameworks handle this?

Ruby on rails

Fingerprinting is enabled by default for both the development and production environments. Because templates use the syntax shown below, there is complete control over what it actually resolves to:

<%= stylesheet_link_tag "application", media: "all" %>
<%= javascript_include_tag "application" %>
<%= image_tag "rails.png" %>

See https://guides.rubyonrails.org/asset_pipeline.html#what-is-fingerprinting-and-why-should-i-care-questionmark

Laravel

After generating the versioned file, you won't know the exact filename. So, you should use Laravel's global mix function within your views to load the appropriately hashed asset.

<script src="{{ mix('/js/app.js') }}"></script>

See https://laravel.com/docs/8.x/mix#versioning-and-cache-busting

Buffalo

Stores a map of all resources in a file called manifest.json by using the Webpack manifest plugin, which contains something like:

{
  "something.txt":         "/assets/something.txt",
  "images/something.png":  "/assets/images/something.png",
  "/images/something.png": "/assets/images/something.png",
  "application.css":       "/assets/application.aabbc123.css"
}

...and then provides helpers, e.g.:

<%= stylesheetTag("application.css") %>
<%= javascriptTag("application.js") %>

See https://github.com/gobuffalo/buffalo/pull/583/files, https://github.com/gobuffalo/buffalo/blob/master/genny/assets/webpack/templates/webpack.config.js.tmpl and https://gobuffalo.io/en/docs/assets/

Ideas

Generate a PHP class called Assets

.... and declare members including the fingerprint calculated:

class Assets {
  const FINGERPRINT = '1c3dee6';
  private $base;

  public function __construct(string $base= '/') { $this->base= rtrim($base, '/').'/'; }

  public function path(): string { return $this->base.self::FINGERPRINT; }
}

Open questions:

Add a configuration file

... to src/main/etc

[assets]
fingerprint=1c3dee6

The app would then pick it up using $this->environmet->config('assets') and pass the configuration value to the frontend.

Open questions:

thekid commented 3 years ago

To implement this:

<link href="{{asset 'bundle.css'}}" rel="stylesheet">

...we would add the class below to xp-forge/handlebars-templates:

<?php namespace web\frontend\helpers;

class Assets extends Extension {
  private $base;

  public function __construct(string $base= '/') { $this->base= rtrim($base, '/').'/'; }

  public function helpers(): iterable {
    yield 'asset' => fn($in, $context, $options) => $this->base.$options[0];
  }
}

...which would then be used as follows:

new Frontend(
  new HandlersIn('com.example.web'),
  new Handlebars(
    $this->environment->path('src/main/handlebars'),
    new Assets('/assets/'.$fingerprint),  // or even: new Assets('https://cdn.example.com/assets/web/'.$fingerprint) ☁
  ),
)

...but leaves the question open where $fingerprint comes from.

Variations

Assets::manifest('manifest.json');   // As provided by https://github.com/shellscape/webpack-manifest-plugin
Assets::in('/assets/'.$fingerprint); // Use a given path
Assets::at('https://cdn.example.com/assets/web/'.$fingerprint); // Verifies parameter is a URL
thekid commented 3 years ago

An idea is to make the generated AssetsFrom class extend web.frontend.FilesFrom, and handle caching as described in https://github.com/xp-forge/web/issues/71#issuecomment-803468156

Immutable assets

Use immutable and a max age of 1 year:

GET /assets/bundle.dc0c4970.css HTTP/1.1

HTTP/1.1 200 OK
Cache-Control: max-age=31536000, immutable

Other assets

Cache for 28 days (probably needs to be configurable!)

GET /assets/image/default.png HTTP/1.1

HTTP/1.1 200 OK
Cache-Control: max-age=2419200, must-revalidate, stale-while-revalidate=86400

To determine whether a filename contains a fingerprint, we could use a regex on the filename as seen here.

Here's an example of what a change would look like:

-$files= new FilesFrom($this->environment->path('src/main/webapp'));
+$files= new AssetsFrom($this->environment->path('src/main/webapp'), [
+  '/[a-z-]+\.[0-9a-f]+\.[a-z]+$/' => CacheControl::immutable(),
+  null                            => CacheControl::mustRevalidate(maxAge: 2419200)),
+ ]);

The hashmap looks like too much boilerplate though, this should be easier!

thekid commented 3 years ago

Can we create a composer plugin to fetch and bundle these when running composer up?

It's probably easier to document adding something along the lines of the following to composer.json:

"scripts": {
  "post-update-cmd": "xp bundle.script.php src/main/webapp/static",
}
thekid commented 3 years ago

The question is if we want to provide a more light-weight alternative

I think it's worth it, the solution above is a lot quicker than the npm install && webpack alternative, ranging from cached to fresh install 6-13 seconds compared to 18-130 seconds (for just bundling JS, YMMV), not including the errors and disk space usage.

Yes, we cannot provide the same CSS and JS minification or any of the various plugins Webpack provides, but we're not preventing users from using that, either. So we're providing a basic bundling infrastructure for the basic needs - if users want to further, they can opt for webpack.

thekid commented 3 years ago

We should also make it possible to create multiple bundles. This way, if not all pages need all dependencies, we can reduce load time.

Of course, bundling often means an increase in file size. Since website loading time is a prime concern for users, it’s important to keep JavaScript bundles as small as possible. One method for reducing bundle size is “code splitting,” or creating multiple bundles from one codebase. All this requires is to set up entry points in the bundler configuration. Each entry is treated by the bundler as the starting point to recursively gather all required files being imported.

(Quoted from https://www.simplethread.com/javascript-modules-and-code-bundling-explained/)

This could be done by requiring subkeys in require-assets:

"require-assets": {
  "vendor": {
    "jquery@^3.6": ["dist/jquery.min.js"],
    "fomantic-ui@^2.8": ["dist/semantic.min.js", "dist/semantic.min.css"],
    "handlebars@^4.7": ["dist/handlebars.min.js"],
    "transliteration@^2.1": ["dist/browser/bundle.umd.min.js"]
  },
  "editor": {
    "dropzone@^5.8": ["dist/dropzone.min.js"],
    "simplemde@^1.11": ["dist/simplemde.min.js", "dist/simplemde.min.css"],
  }
}

For the simple case: this would be written as follows:

"require-assets": {
  "bundle": {
    "jquery@^3.6": ["dist/jquery.min.js"],
    "fomantic-ui@^2.8": ["dist/semantic.min.js", "dist/semantic.min.css"],
    "handlebars@^4.7": ["dist/handlebars.min.js"],
    "transliteration@^2.1": ["dist/browser/bundle.umd.min.js"],
    "dropzone@^5.8": ["dist/dropzone.min.js"],
    "simplemde@^1.11": ["dist/simplemde.min.js", "dist/simplemde.min.css"],
  }
}

The upside is, we wouldn't be hardcoding the name bundle anywhere in our bundling tool 😉

Update: The bundler tool has been updated to be able to parse this two-level format. Here's how that plays out inside a composer run:

image

thekid commented 3 years ago

Added another optimization to not revalidate each and every dependency of an asset if the asset itself was successfully revalidated using If-None-Match / If-Modified-Since. This speeds up execution in the above example from 0m6.440s -> 0m3.533s

thekid commented 3 years ago

Yes, we cannot provide the same CSS and JS minification or any of the various plugins Webpack provides

At least for minification, we could try using https://github.com/matthiasmullie/minify

thekid commented 3 years ago

Computing the hashes for the bundles is easy (see below, uses the same placeholder format as webpack), however, we would need to move around all dependencies fetched in order to also fingerprint them:

diff --git a/src/main/php/xp/frontend/BundleRunner.class.php b/src/main/php/xp/frontend/BundleRunner.class.php
index 8ca6e79..557c12d 100755
--- a/src/main/php/xp/frontend/BundleRunner.class.php
+++ b/src/main/php/xp/frontend/BundleRunner.class.php
@@ -97,9 +97,9 @@ class BundleRunner {

     $cdn= new CDN($fetch);
     $resolve= new Resolver($fetch);
-    $bundles= 0;
     $pwd= new Folder('.');

+    $bundles= 0;
     try {
       $timer= (new Timer())->start();
       foreach ($require as $name => $spec) {
@@ -117,13 +117,19 @@ class BundleRunner {

         // Generate bundles
         foreach ($result->bundles() as $type => $source) {
-          $bundle= with ($source, new File($target, $name.'.'.$type), function($in, $file) {
+          $expanded= strtr($name, ['[hash]' => $source->hash]).'.'.$type;
+          $bundle= with ($source, new File($target, $expanded), function($in, $file) {
             $in->transfer($file->out());
             return $file;
           });
           $bundles++;

-          Console::writeLinef('%s: %.2f kB', str_replace($pwd->getURI(), '', $bundle->getURI()), $bundle->size() / 1024);
+          Console::writeLinef(
+            '%s: %.2f kB (%s)',
+            str_replace($pwd->getURI(), '', $bundle->getURI()),
+            $bundle->size() / 1024,
+            $source->hash
+          );
         }
         Console::writeLine();
       }
diff --git a/src/main/php/xp/frontend/Result.class.php b/src/main/php/xp/frontend/Result.class.php
index 9d2eab1..7617f7a 100755
--- a/src/main/php/xp/frontend/Result.class.php
+++ b/src/main/php/xp/frontend/Result.class.php
@@ -5,6 +5,7 @@ use util\cmd\Console;
 class Result {
   private $cdn, $handlers;
   private $sources= [];
+  private $hashes= [];

   public function __construct(CDN $cdn, array $handlers) {
     $this->cdn= $cdn;
@@ -36,10 +37,14 @@ class Result {

   public function prefix($type, $bytes) {
     $this->sources[$type][0][]= $bytes;
+    $this->hashes[$type]??= hash_init('sha1');
+    hash_update($this->hashes[$type], $bytes);
   }

   public function concat($type, $bytes) {
     $this->sources[$type][1][]= $bytes;
+    $this->hashes[$type]??= hash_init('sha1');
+    hash_update($this->hashes[$type], $bytes);
   }

   /**
@@ -49,7 +54,7 @@ class Result {
    */
   public function bundles() {
     foreach ($this->sources as $type => $list) {
-      yield $type => new Source($list);
+      yield $type => new Source($list, substr(hash_final($this->hashes[$type]), 0, 9));
     }
   }
 }
\ No newline at end of file
diff --git a/src/main/php/xp/frontend/Source.class.php b/src/main/php/xp/frontend/Source.class.php
index a720df7..d83d4d0 100755
--- a/src/main/php/xp/frontend/Source.class.php
+++ b/src/main/php/xp/frontend/Source.class.php
@@ -5,10 +5,12 @@ use lang\Closeable;

 class Source implements Closeable {
   private $list;
+  public $hash;

   /** Creates a new source */
-  public function __construct(array $list) {
+  public function __construct(array $list, string $hash) {
     $this->list= $list;
+    $this->hash= $hash;
   }

   /** Transfers this source to an output stream */

...so maybe we'll keep this for a second pull request

thekid commented 3 years ago

There is one problem with how we're using composer.json with our proprietary format - automated tools like Dependabot etc. will not discover outdated libraries. We can also add proprietary keys to package.json, e.g. in the following below dependencies:

{
  "dependencies": {
    "dropzone": "^5.8",
    "fomantic-ui": "^2.8",
    "handlebars": "^4.7",
    "jquery": "^3.6",
    "simplemde": "^1.11",
    "transliteration": "^2.1"
  },
  "bundles": {
    "vendor": {
      "jquery": "dist/jquery.min.js",
      "fomantic-ui": "dist/semantic.min.js|dist/semantic.min.css",
      "handlebars": "dist/handlebars.min.js",
      "transliteration": "dist/browser/bundle.umd.min.js"
    },
    "editor": {
      "dropzone": "dist/dropzone.min.js",
      "simplemde": "dist/simplemde.min.js|dist/simplemde.min.css"
    }
  }
}

Note: They syntax with the pipe character (|) was necessary because NPM re-formats the file, expanding arrays over multiple lines, making it hard to read!

thekid commented 3 years ago

First step - creating bundles via xp bundle and gzipping them - released in https://github.com/xp-forge/frontend/releases/tag/v2.2.0

thekid commented 3 years ago

Why I use Rollup, and not Webpack

https://link.medium.com/sEhXNG8Bcfb