Assets concatenation

[roperzh]

Hi guys,

Since too Many HTTP Requests is one of the biggest contributors to performance problems in most web pages, I was thinking about the best route to concatenate JS / CSS files in hugo.

Do you have a suggested workflow?.

[natefinch]

Test, then optimize. Build your site in the simplest way possible, then use tooling to figure out what, if anything, makes it slow. Honestly, I doubt it'll be http calls to download assets. On Aug 22, 2014 6:13 PM, "Roberto Dip" notifications@github.com wrote:

Hi guys,

Since too Many HTTP Requests is one of the biggest contributors to performance problems in most web pages, I was thinking about the best route to concatenate JS / CSS files in hugo.

Do you have a suggested workflow?.

— Reply to this email directly or view it on GitHub https://github.com/spf13/hugo/issues/432.

[roperzh]

I agree with you, but fewer http requests improves the load time of the page.

There are several studies that validate this statement, you can check some of these:

• google
• yahoo
• Steve Souders Book

Also middleman (uses sprockets), and most of the static site generators powered by node allows you to concatenate and minify your assets. ( not sure about jekyll ).

This is not a very specific request, it's a common pattern building web pages. Nevertheless maybe this is beyond the responsibilities of hugo.

[natefinch]

Yeah, but would it improve it in a noticeable way? Here's some stats from my totally unoptimized site: http://tools.pingdom.com/fpt/#!/biO21I/http://npf.io The send times are negligible. I really wouldn't worry about it. On Aug 22, 2014 7:41 PM, "Roberto Dip" notifications@github.com wrote:

I agree with you, but fewer http requests improves the load time of the page.

There are several studies that validate this statement, you can check some of these:

• google https://developers.google.com/speed/docs/best-practices/request
• yahoo https://developer.yahoo.com/performance/rules.html
• Steve Souders Book http://stevesouders.com/hpws/rules.php

Also middleman (uses sprockets), and most of the static site generators powered by node allows you to concatenate and minify your assets. ( not sure about jekyll ).

This is not a very specific request, it's a common pattern building web pages. Nevertheless maybe this is beyond the responsibilities of hugo.

— Reply to this email directly or view it on GitHub https://github.com/spf13/hugo/issues/432#issuecomment-53134843.

[spf13]

I think it is outside the responsibility of Hugo today. Perhaps after 1.0.

I think discussing different strategies is healthy.

I spent a ton of time optimizing spf13.com. I event went so far as to write 100% of the CSS by hand. No frameworks. Mostly I did that just to see what it would take. Overall I'm very happy with the results, but I agree with Nate, I don't think it really mattered all that much.

Still if the author experience was very low friction, then there's no reason not to do it.

I would certainly be open to discussing different ways we may want to address this in hugo, but we may never implement any of them.

[Netherdrake]

@natefinch @spf13 Web dev here. Compiling assets (concatenate + minify) is by far the most important thing to do in production as far as improving the load times goes. The benefits of doing this are exponential in relationship with the amount of assets. It is an absolute must do (in conjunciton with caching).

There are however existing tools for this, for instance gulp.js. No need to reinvent the wheel within Hugo.

@roperzh What you may want to do is use gulp/grunt to compile assets, save them with hash filenames for easy caching (eg. d3222ebb686ad7bfbec098697608212e.css) and in nginx (or any other server) configure cache for .css, .js and image files indefinitely.

[roperzh]

I agree with you guys. My intention with this issue was to know if there is a suggested workflow for this task.

I mean if you use gulp.js you are including an extra dependency (node + gulp.js) into you project.

An even simple way could be a makefile to concatenate the files with something like this:

cat static/js/{file1,file2}.js  > build/app.js

But then you need to replace all the script references from all the *.html files in the build. There is a way to have a conditional for builds?, so for example I can end with something like this:

{{ if .isBuild }} 
  <script src="app.js"></script>
{{ else }}
  <script src="file1.js"></script>
  <script src="file2.js"></script>
  ...
{{ end }}

[luislavena]

@Netherdrake @natefinch @spf13 I don't think there was an idea of reinvent the wheel on this :smile:

Since Hugo lacks some pre and post hooks for processing, I believe @roperzh was trying to gather some feedback and recommendations about it.

Perhaps documenting best approaches to this (in combination with Hugo) will be helpful for those who sees the power of Hugo but still have the ability to include those elements.

Thank you.

[Netherdrake]

@roperzh Here is how I'd do it.

Have 2 themes, the one you work on/maintain and the one that is the output of a gulp build.

themes/
    mytheme/
    mytheme-build/

Than, run hugo with hugo --theme=mytheme-build for production. Problem solved.

Edit: gulp is just one of the many tools to solve this problem. Pretty much every language with web dev community has at least one. In fact, a quick google search later, here is one written in Go.

[roperzh]

@Netherdrake Sounds like a good solution!, thank you!

[aanomm]

On desktop, with good connection speeds, concatenation might seem a trivial topic. But on mobile, and/or with poor connection speeds, concatenation is the critical topic. Not all nations are equal, especially when it comes to the state of mobile networks: take Australia for example, who lag so embarrassingly behind the wider Western world: HTTP requests can kill connectivity, and given the growth and continued-growth of mobile, it really does matter. I'm not saying that concatenation should be a part of Hugo's scope, but I would certainly welcome it.

While not comparable to Hugo, I know that mixture.io and some other's like it indeed concatenate by default.

[natefinch]

So, it seems like this is just something each individual author should do. The css and js files are generally just static, so Hugo doesn't deal with them at all. If you want to concatenate and minify, you can. On Aug 27, 2014 7:51 PM, "loadsfaster" notifications@github.com wrote:

On desktop, with good connection speeds, concatenation might seem a trivial topic. But on mobile, and/or with poor connection speeds, concatenation is the critical topic. Not all nations are equal, especially when it comes to the state of mobile networks: take Australia for example, who lag so embarrassingly behind the wider Western world. I can speak from experience that HTTP request can kill connectivity. I'm not saying that concatenation should be a part of Hugo's scope, but would absolutely welcome it.

While not comparable to Hugo, I know that mixture.io and some other's like it indeed concatenate by default.

— Reply to this email directly or view it on GitHub https://github.com/spf13/hugo/issues/432#issuecomment-53656854.

[spf13]

For CSS concatenation is very straightforward. JS is much less so.

I have some ideas on a good and simple way to do this sort of thing from within Hugo. I don't ever want Hugo to be complex when existing tools work well to achieve the same functionality.

I'd be happy to provide a convenient way to use existing tools though.

We do this with pygments and i think it turned out well.

Stay tuned. I'll be looking into it after the v0.12 release ships.

[ryan-kimber]

It seems to me that this is outside the scope of Hugo as well. If you're technical enough to be worried about compression and concatenation, then look into a build tool like Gulp or Grunt which are purpose built for preparing a site for deployment.

I'm working on a blog post that I'll put up shortly that discusses pairing Hugo with Gulp.

For our site, http://launchcode5.com, our project is set up with Hugo to generate the content, Gulp to actually do the watching and serving of built files locally and Gulp to do concatenation, CDN'izing dependencies, running SASS and single-command deployment.

(congratulations, by the way, on building such a great tool. I've forever been frustrated with with WordPress and it's siblings. They're great for their audience, but that audience is not me or most of the developers I know.)

[spf13]

@ryan-kimber It would be great to have a write up on your workflow. I'm sure it would help out a lot of people (myself included). I'd love to add it as a tutorial to the documentation.

[spf13]

I actually think that it would be great if Hugo did some basic housekeeping. I don't want to ever come close to replacing Gulp or the like, but if Hugo resized images or compressed them... or concatenated css files. These simple things would be really helpful for most people and don't require the extra work of setting up the external deps.

[rickb777]

I think asset concatenation is a useful technique. However, the Sass and Less compilers already to this rather well, so anything that duplicates them is a bit of a waste of effort.

Conversely, another almost-for-free optimisation is to pre-compress text assets using Gzip. This is of no use to Apache sites, but Nginx handles the browser content negotiation and serves gzipped content where possible, plain content otherwise. (Ditch Apache, folks!)

This is a double win - compression happens only once, when content is generated (which is Hugo's domain); the smaller files are quick to read off disk, require less memory in the server and take up fewer network bytes. Obviously, the uncompressed files must be present too.

So I would be pleased to see text asset gzipping as an option built into Hugo.

[nathany]

Asset concatenation is an anti-pattern when served from an HTTP/2 capable web server or CDN. Multiple files can be multiplexed on a single connection and sent just as quickly. All major browsers support HTTP/2 these days.

What asset concatenation does is defeat caching. A small change requires downloading the full concatenated all.js or all.css instead of a smaller file.

[ryan-kimber]

While HTTP2 servers help by not having to create an entire connection (particularly over HTTPS) to retrieve resources after the first one, requesting multiple files still adds significantly, to page load times due to round-trip latency, particularly for mobile devices (developers frequently underestimate the latency added by mobile, especially so for rural areas and countries outside of the G20).

Additionally, asset concatenation is only one part of post-processing that should be done on resources to be served to the browser: there's also variable substitution, minification. SASS processing... and so forth (depending on what tools you use).

A while back, I posted a gulp & hugo seed project https://github.com/ryan-kimber/gulp-hugo-seed that I use to get the benefits Hugo offers in HTML generation plus the pre-deployment processing I need. I should have posted that hear earlier.

On Mon, Apr 18, 2016 at 11:22 PM, Nathan Youngman notifications@github.com wrote:

Asset concatenation is an anti-pattern when served from an HTTP/2 capable web server or CDN. Multiple files can be multiplexed on a single connection and sent just as quickly. All major browsers support HTTP/2 these days.

What asset concatenation does is defeat caching. A small change requires downloading the full concatenated all.js or all.css instead of a smaller file.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/spf13/hugo/issues/432#issuecomment-211708907

[nathany]

My understanding is that the latency is in establishing the connection, not sending frames that request data. HTTP/1.1 is slow because it needs to establish dozens of connections just to fetch a website.

But I haven't tested HTTP/2 vs. HTTP/1.1 on a mobile phone in a rural area myself to know the latencies for certain.

there's also variable substitution, minification, SASS processing

true, but those are separate issues, eg. #16

Preprocessors also provide asset concatenation if for one reason or another somebody wants it. I don't think we need a separate issue for this.

[ryan-kimber]

I agree that external tools, like gulp or hugo make this change unnecessary. However, it shouldn't be justified by saying concatenation isn't something developers should do; that's misleading.

There are still pretty significant benefits to concatenating under HTTP2, including fewer requests, fewer headers, and better compression when like-typed files are concatenated together. Additionally, HTTP2 isn't supported in the 'Android Browser' still used by many Android phones, IE8 & 9, 10 or 11 (over HTTP), or any version of Safari on HTTP (though HTTPS is supported).

http://stackoverflow.com/questions/28630108/does-minifying-and-concatenating-js-css-files-and-using-sprites-for-images-stil

On Tue, Apr 19, 2016 at 10:25 PM, Nathan Youngman notifications@github.com wrote:

My understanding is that the latency is in establishing the connection, not sending frames that request data. HTTP/1.1 is slow because it needs to establish dozens of connections just to fetch a website.

But I haven't tested HTTP/2 vs. HTTP/1.1 on a mobile phone in a rural area myself to know the latencies for certain.

there's also variable substitution, minification, SASS processing true, but those are separate issues, eg. #16 https://github.com/spf13/hugo/issues/16

Preprocessors also provide asset concatenation if for one reason or another somebody wants it. I don't think we need a separate issue for this.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/spf13/hugo/issues/432#issuecomment-212216328

[nathany]

In my belief, concatenation is something developers should not do, unless they are still serving via HTTP/1.1. That's why I said it. To inform people who are not familiar with HPACK compression of headers, multiplexing of multiple requests over a single connection, and the other benefits of HTTP/2.

This issue was opened in 2014. Asset concatenation doesn't provide the benefits it once did.

If this issue hasn't been resolved yet, and if it's no longer a best practice, then I suggest that Hugo should have no official support for it.

That doesn't mean you can't do it if you have a reason to (Android Browser, what-not), it just means we could close one of 249 open issues.

[mrichman]

FWIW, I have a deploy.sh script which invokes https://www.npmjs.com/package/minifier just before pushing my static content up to Github Pages.

[maxmilton]

If anyone's still interested in this I have a full hugo site DEV + deployment workflow over here: https://github.com/MaxMilton/MaxMilton.com

Instructions for use are in the README.md. Uses gulp as a task runner, browser-sync for a live dev server, and a simple bash script for deployments. One could even go simpler and not use gulp—for me it's just convenient.

One of the best things about hugo is its relative simplicity. It's great at taking some input pages + layout/data templates and outputting HTML. Beyond that, using dedicated tools is probably a better idea than bloating hugo with features only a fraction of developers may use. The speed and simplicity is what attracted me to hugo in the first place!

[bep]

This issue has been automatically marked as stale because it has not been commented on for at least four months.

The resources of the Hugo team are limited, and so we are asking for your help.

If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, and you feel that it is still valuable, please open a proposal at https://discuss.gohugo.io/.

This issue will automatically be closed in four months if no further activity occurs. Thank you for all your contributions.

[bep]

Note/Update: This issue is marked as stale, and I may have said something earlier about "opening a thread on the discussion forum". Please don't.

If this is a bug and you can still reproduce this error on the latest release or the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.