Add a base set of resource filters/processors

[bep]

This is related to #4381 -- but is something I have been thinking about lately. We will eventually get some plugins/hooks that people can exploit, but the current "lazy processing resource" model is an easy model to understand, and it scales well.

I have been thinking about this in terms of "what would be the minimal set that would protect me against having to do to any npm install on my Hugo projects, i.e. fully native. The more I'm thinking about it, the more stupid it gets that I still do this for fairly plain tasks with slow JS software with a rather temperamental versioning system. So I intend to get this done fairly quick.

I will mentally limit myself to 5 tasks here. The NPM registry is enormous, so there will be use cases that does not fit with Hugo alone.

My 5 is:

• SASS/SCSS to CSS (this is the natural only choice, for tech reasons. And Bootstrap 4 is SCSS)
• Asset files concatenation (simple bundling of CSS, JS) (we would probably need some kind of assets directory for this)
• Minify (MIME type aware minification of JS, CSS)
• Digest (integrity hash, see https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity)
• Fingerprint (hash into the permalink, .e.g main-88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589.css).

Without going into details in the above, could this match your workflow?

Update: The syntax of the above will be something like this:

$css :=  resource "main.scss" | resource.Sass | resource.Digest | resource.Fingerprint

Pick what you want. The above can also be used with the "bundled resources" in Page.

And of course Hugo-fast. Also see #4381 #

[kaushalmodi]

I don't care much about SASS (as I don't know it), but the other 4 are great.

[bep]

@kaushalmodi I have updated the "vote text" above. It is allowed to think about what may be useful to others.

[trys]

They all sound perfect to me! It would be great if the hash could be optional. Also, would there need to be a manifest file (maybe a data template) that linked main-88d4266fd.css to main.css to keep the template files clean?

[bep]

@trys I think you will be happy after reading #4381

[kaushalmodi]

@bep

It is allowed to think about what may be useful to others.

I agree, that's why I have already voted on the 5 point option (that first vote was mine) :D

[trys]

@bep that looks like all sorts of awesome!

[RealOrangeOne]

Asset files concatenation (simple bundling of CSS, JS)

Simple concatenation works fine for CSS, but for JS, it's a little more complicated, if you want to do it properly.

An ideal great feature would be to rather than concatenate, bundle the files. Much like http://browserify.org/ does, you give it a single entrypoint, and it spits out a single file, and uses the calls to require() to pull in external files as required. I can't see a very large market for concatenating JS files, but I can see a huge market for bundling!

(Not exactly sure if this is what you meant in the bullet point, just thought i'd share in case)

[bep]

(Not exactly sure if this is what you meant in the bullet point, just thought i'd share in case)

I meant simply concatenating the files. I.e. what "cat file1 >> file2" does. Unless there is some great "Go bundling library" I can hook into, I'm not spending 100 hours digging into this problem (unless someone wants to pay me, that is).

[bep]

And I think you are overthinking the problem. The "cat file1 >> file2" is more or less what the Ruby "bundler" in the competition does. I have implemented that in Go somewhere. If that is not good enough, you will have to look elsewhere.

[jloh]

Is the digest going to be for local files only or remote as well?

[bep]

Is the digest going to be for local files only or remote as well?

Local. At least in its first iteration: We have a Resource interface -- and they are currently local only, but that can change.

[crgeary]

I don't suppose there is any chance for autoprefixer support with the SASS/CSS features? I could then remove almost all of the Node/Gulp I currently have in the indigotree/atlas boilerplate.

Update: If it helps the decision, 3 of the 5 starter kits on the Hugo Docs use an Autoprefixer for their CSS.

[bep]

I don't suppose there is any chance for autoprefixer support with the SASS/CSS features?

If libsass supports it, sure.

[thewebmastercom]

Looking at the Source for Foundation, I don't think it will work (too complicated build process), so doubt it would be able to replace my current workflow there (at least not without a lot of work that would make Foundation difficult to update)..

However, looking at the Bootstrap 4 source, the SASS \ JS looks very simple to compile \ combine. Would certainly make me consider using Bootstrap as my framework of choice in the future if Hugo did all the work.

[bep]

A general comment is that the above should solve real use cases. Certainly not all, but I imagine having a prototype Boostrap 4 site up and running as a proof of concept before pushing this to the world.

[RickCogley]

Plus one for autoprefixer. I am currently using postcss/cssnext which is pretty flexible, but, npm can be fiddly (so many libraries... SO many, I feel like I'm using perl). I've heard that some teams insist on linting css or js in a certain way, so although I was going to mention that, you can do it in-editor. Doing it in pre-processing would make it possible to achieve some quality control for a team, though.

My wish in this area is related to making a separate css bundle for each language in a multi-language site. Different languages often need different webfonts, so I'd like to be able to get this feature to bundle say "main.css" with "en.css" for the English side, or, "main.css" with "ja.css" for the Japanese side, so that I can use different webfonts stacks per language by calling say bundle.en.min.css or bundle.ja.min.css in my <head>.

On a current project, I'm using:

:lang(ja) {
  font-family: 'jp font 1', 'jp font 2', serif;
}
:lang(en) {
  font-family: 'en font 1', 'en font 2', serif;
}

... in a single css file, but I found I cannot override these classes (say, for styling an English string inside a Japanese text block) using normal classes like .en-serif, having to use a lang="en" instead. So I'd rather avoid lang and just use generic classes like body-serif or display-sans, changing the meaning of those with a per-language css that gets bundled with the main.

[davidsneighbour]

@RickCogley you can do that by having separate root files for your SASS compilation where you have the same base styles and then per language separate detail files included. In your theme then you add something like theme-{{ $.languagecode }}.css in the header. What I mean is libsass should cover this.

[rdwatters]

My personal workflow? Yes, this sounds great. The consideration of Bootstrap also makes sense in terms of being the most popular FE framework.

However, I’d defer to other frontend devs, many of whom seem to use module loaders/bundlers (viz. Webpack) for just about everything, even outside the context of creating apps.

Nice-to-Have: some sort of uglify on top of minify for JS—just in consideration of file size—although I have no idea if there are Go implementations, so take that request as you will :smile:

[budparr]

While I personally wouldn't need this (huge fan of Webpack's super powers), I like the idea generally because of the simplicity of, as you say, no NPM modules. Definitely a good idea for small sites.

In addition to what the others have said (auto-prefixer is important), I find tools like PurgeCSS or PurifyCSS—which remove unused CSS—pretty valuable. I did a quick check of Awesome Go and didn't see any packages for that sort, but maybe there's something out there.

[Lego2012]

SCSS. Bootstrap 4 is SCSS (not Sass)

[vsopvsop]

Difference between .sass and .scss https://responsivedesign.is/articles/difference-between-sass-and-scss/

[bep]

SCSS. Bootstrap 4 is SCSS (not Sass)

@Lego2012 we can work on the naming here, but every report I have read says that Bootstrap 4 compiles with libsass, which is the important thing here.

https://twitter.com/SassCSS/status/634106348951748608

@RickCogley there are some hidden multilingual powers in here that I think can make some very elegant setups.

Just thinking out lout with some options:

1) Multiple static dirs:

/jp/main.scss /en/main.scss

Doing this in a template will just work:

$css :=  resource "main.scss" | resource.Sass

2) Put the scss into the "home bundle":

/content
_index.md
main.scss
main.jp.scss

And then:

$css := .Site.Home.Resources.GetMatch "main*" | resource.Sass

Definitely a good idea for small sites.

@budparr I suspect you slightly underestimate this. I agree that this setup falls short in more advanced JS setups, but for the typical Hugo sites (lots of content, documentation sites, blogs), this should scale very well.

The "one binary" thing is what attracts many with Hugo, and if I were the "director of technical documentation" and was about to create this big new documentation site for my product, I would be willing to give in to quite a few sacrifices to get my site into the 5 bullet points above.

[Lego2012]

we can work on the naming here

@bep I made the remark from a front end developers view. And from there it's not a question of naming. Working with SCSS compared to Sass is a quite big difference.

From a technical perspective (libsass) you're absolutely right of course.

[bep]

and if I were the "director of technical documentation" and was about to create this big new documentation site for my product,

And I was the security director of that same company, I would also love not letting NPM lose inside my network.

[gour]

If it's not too hard, I'd like to see support for Less (e.g. UIkit framework) as well.

[bep]

If it's not too hard, I'd like to see support for Less

There is no native (Go or C) Less compiler available, I'm afraid.

[bep]

Also see https://github.com/gohugoio/hugo/issues/4449

[budparr]

@bep It undoubtedly scales. I was referring to the flexibility you can get with other tools. Happy to go into that more, but I doubt it would add to the conversation here. Overall I think this is a great feature for many people, as I mentioned.

[regisphilibert]

I suppose this will work with LiveReload as well? Usually, we watch for every sass file, so that when one of the files imported into "main.scss" is saved, we reprocess "main.scss". I think this is the way to go. Same goes with JS concatenation.

I personally can't find any use case where this behaviour could be challenged, but if someone does, he/she should speak up ;)

[bep]

I suppose this will work with LiveReload as well?

Yes, that is a must. There will be some challenges implementing this in a good way, but it should be possible.

[jbrodriguez]

Minification of html would be awesome too. It helps with overall size.

libsass doesn't 'autoprefix', but it's certainly needed, for browser compatibility.

[DirtyF]

As I rely on Netlify for assets optimization, Sass compilation would already allow to avoid jugglin' with external tools.

What would be nice to benefit from functions that could extract critical CSS for instance. Right now you need to use npm tools for that.

[bep]

libsass doesn't 'autoprefix',

Maybe not in the strict definition of "auto", but there are plenty of good SASS vendor prefix examples via mixins out there. There are benefits in keeping some manual control.

[rdwatters]

...but there are plenty of good SASS vendor prefix examples via mixins out there. @jbrodriguez @thewebmastercom

https://www.bourbon.io/docs/latest/#prefixer

I'm excited for this feature. Thanks much @bep

[client9]

re: @budparr

I find tools like PurgeCSS or PurifyCSS—which remove unused CSS—pretty valuable.

re: @DirtyF

What would be nice to benefit from functions that could extract critical CSS for instance.

Turns out I was working on exactly this problem. See https://github.com/client9/csstool It's all golang. Scans my blog and cuts out un-needed CSS in under 200ms and cuts out 90% of bootstrap.

go get -u github.com/client9/csstool/css
curl -s https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css | \
    css cut --html 'public/**/*.html' > static/bootstrap-csscut.min.css

Not sure if this fits in with this ticket @bep , and so my apologizes if this is ticket pollution/spam. Happy to adjust this to whatever workflow if needed. I'll be working on packaging and a lot of testing in the meantime.

Feedback most welcome at https://github.com/client9/csstool/issues

And thanks @bep for working on this feature!

update: 2018-03-04 -- switched to spf13/cobra for CLI interaction. Usage is css cut --html not csscut -html now.

[regisphilibert]

I don't think this is urgent, but later on Babel may sound like a good addition. Being able to use modern javascript syntaxes has a great (at time critical) appeal and may be one of the most important reasons your project may need a build tool.

Beside being able to Babel your JS files and @bep's 5 , I really can't think of any other critical need for a build tool.

[RickCogley]

Been thinking about this, and wondering what about source maps for better debugging of minified js and css assets?

[hanzei]

isn't this issue finished with hugo 0.43?

[mpourismaiel]

Adding support for babel in the pipelines would be fantastic. Babel is configurable through .babelrc. Although I don't think it's possible to run babel without installing it's JS dependencies.

[hanzei]

@mpourismaiel Would you mind opening a new issue for this feature request?

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.