search

source-foundry / Hack

A typeface designed for source code
http://sourcefoundry.org/hack/
Other
16.5k stars 614 forks source link

Proposal: Web font builds #286

Closed chrissimpkins closed 7 years ago

chrissimpkins commented 7 years ago

woff is supported in all modern browsers. TTF, EOT, and SVG web fonts are no longer recommended.

Suggest that we build woff (+/- woff2) files only for web font releases.

chrissimpkins commented 7 years ago

web font build scripting is underway in the dev-build-scripts branch.

chrissimpkins commented 7 years ago

woff browser support from https://caniuse.com/#feat=woff

woff

chrissimpkins commented 7 years ago

woff2 browser support from https://caniuse.com/#feat=woff2

woff2

chrissimpkins commented 7 years ago

Scripting for ttf to woff build is now available in build-web.sh build-woff.sh script dev-build-scripts branch.

To install dependencies before the build, use:

$ ./build-woff.sh --install-dependencies

All subsequent builds:

$ ./build-woff.sh

Builds currently land on the path build/web/*.woff

chrissimpkins commented 7 years ago

Added new build-woff2.sh script in dev-build-scripts branch. Builds woff2 web fonts from ttf build files. I confirmed that there is a significant compression advantage c/w the woff files. (~150kb to ~100kb).

We will plan to release both woff and woff2 web font builds.

woff2 build adds a new build dependency = https://github.com/source-foundry/woff2

This is a fork of the Google woff2 repo that is tracking upstream changes and fixes a Makefile bug (https://github.com/google/woff2/issues/73) with compiles on OSX with fix from https://github.com/JLHwung/woff2/tree/fix-macos-compilation-error. @JLHwung Makefile changes are merged to master branch of the Source Foundry fork.

Builds work identical to build-woff.sh script (see https://github.com/source-foundry/Hack/issues/286#issuecomment-325031010):

To install dependencies before the build, use:

$ ./build-woff2.sh --install-dependencies

All subsequent builds:

$ ./build-woff2.sh

Builds are on path build/web/*.woff2.

chrissimpkins commented 7 years ago

https://github.com/source-foundry/Hack/commit/4748b6efac7e4aeaf29f2f9f613965f461132939 adds ASCII + Latin-1 + Latin Extended A glyph name subset data to the lib.plist file.

This will allow us to create a web font subset build that will be smaller in filesize.

chrissimpkins commented 7 years ago

PR #291 under review for contents of the ASCII + Latin-1 + Latin Extended A glyph names.

chrissimpkins commented 7 years ago

new build-subsets.sh script added and updated to version passing tests as of https://github.com/source-foundry/Hack/commit/14f8d41f3275c7410c992e005820202376a87225

This completes the build process from UFO to ttf to woff/woff2 to woff/woff2 subsets.

chrissimpkins commented 7 years ago

Final web font subsets require examination to confirm the glyphs in the sets.

chrissimpkins commented 7 years ago

@burodepeper If I update the subset lib.plist files in #291 and push a set of woff and woff2 web fonts are you able to proof the available web font glyphs against your recommendations for the contents? I am going to try to wrap up licensing work and begin on hinting updates.

chrissimpkins commented 7 years ago

^^ subsets only clearly. Just need to know that what we expect to be in them is in them.

burodepeper commented 7 years ago

@chrissimpkins From your code corpora project, do you have a summary of what characters are used?

burodepeper commented 7 years ago

@chrissimpkins We could also scrape a number of large (news) sites and make an inventory of what characters appear in their content. Combined with your code corpora, that should give at least a somewhat adequate representation of code and content. Agreed?

chrissimpkins commented 7 years ago

@burodepeper Not following. You mean glyphs used in source code?

burodepeper commented 7 years ago

You mean glyphs used in source code?

Yep, everything besides ASCII basically.

chrissimpkins commented 7 years ago

@burodepeper I didn't mean anything that detailed at this stage. I just want to make sure that we have contents of the ASCII + Latin-1 + Latin Ext A sets in each of the webfonts. Character set manipulation vs. source code is something we could do, but would kick that down the line for a bit. There won't be a clean analysis in Code Corpora because we will be capturing comments there. We would probably need to remove all comment blocks to get an appropriate source only representation.

chrissimpkins commented 7 years ago

and Code Corpora will likely be a very biased (English leaning) sample of source...

My goal there was more to examine the frequencies of ASCII glyph use across common programming languages rather than to provide a true representation of the breadth of non-ASCII glyphs that are used in source code.

burodepeper commented 7 years ago

So, you mean you want me to check if the webfonts have all the characters that are supposed to be in them?

chrissimpkins commented 7 years ago

Welcome to generate any list of glyph contents for the web font subsets that you feel is appropriate. They are reasonably small subsets at this stage and we have room to add. I can support this by modifying the subset lib.plist files so that we can generate a set of web fonts that contains (I hope, this is the request) the glyphs that you recommend. This seems to involve significant research work and falls outside of our design guidelines with much work yet remaining within the high priority areas of the design guidelines. We can (and should!) revisit the design guidelines if you find other glyphs that seem to be important. Perhaps we broaden to include support for strings/comments in a subset of languages? Slippery slope though.

Let me know how you want to approach it. I very much rely on you in the web arena.

burodepeper commented 7 years ago

I'm still not sure what you want me to proof. I think the subset we've landed on (for now) is good. And there's always the full set as backup. We can revisit the optimal subset later.

chrissimpkins commented 7 years ago

Can you please proof the set of web fonts that I push this evening against the sets that you recommended for web fonts to confirm that they:

chrissimpkins commented 7 years ago

No request to further optimize the subsets :)

That was your own suggestion in https://github.com/source-foundry/Hack/issues/286#issuecomment-325992076

chrissimpkins commented 7 years ago

Added currency and general punctuation glyphs to the ASCII + Latin-1 + Latin Extended A sets. This is where we are with file sizes in current full set and subset builds:

web 2017-08-30 21-10-19

chrissimpkins commented 7 years ago

I reviewed the Hack-Regular.ttf glyphs as I created the list of glyphs for the subsets.

currency set does not include uni20B6 (some old French currency glyph from middle ages?)

There are a number of missing Unicode code points across the general punctuation set.

The defined/expected glyphs are in the plist files in https://github.com/source-foundry/Hack/tree/dev-build-scripts-subset/source/subset-lib (currently we are in dev-build-scripts-subset branch and this will be merged to dev-build-scripts when the web font subset work is complete). They are located under the <key>com.schriftgestaltung.Keep Glyphs</key> key.

chrissimpkins commented 7 years ago

New web font (full and subset builds) are available in https://github.com/source-foundry/Hack/tree/dev-build-scripts-subset/build/web .

@burodepeper let me know if you are able to address the last two items in the OP TODO list. If not, I will get back to it once I wrap up some other things. Thanks for your input on the subsets.

burodepeper commented 7 years ago

Results of manually proofing the glyphs. I used the regular subset (assuming that there's no difference between regular and the other variants) and found that the following glyphs are missing:

chrissimpkins commented 7 years ago

👍 thank you! To clarify, these are included in ASCII + Latin-1 + Latin-A spec, but they are not present in the regular web font subset?

chrissimpkins commented 7 years ago

and if above is a valid statement, anything that you see in the missing collection that seems mandatory? I will need to look at the tables to confirm what glyphs they are. Difficult to see some of them in Github formatted Markdown text.

burodepeper commented 7 years ago

I took the characters from this site: http://jrgraphix.net/r/Unicode/0100-017F (and subsequent pages)

The first six might be used in some Eastern European language, though I would have no idea which. The last glyph (20BF), I believe, is the Bitcoin sign, which I think can make a valid appearance in 'normal' content. Besides those, 201F and 2022 seemed like glyphs that you would expect to be present. I believe we can do without the rest.

chrissimpkins commented 7 years ago

The first six might be used in some Eastern European language, though I would have no idea which. The last glyph (20BF), I believe, is the Bitcoin sign, which I think can make a valid appearance in 'normal' content. Besides those, 201F and 2022 seemed like glyphs that you would expect to be present. I believe we can do without the rest.

OK, thanks David. I will check the source against the values that I entered in the lib.plist files to confirm that this is not a mistake on my end. Some are likely to be missing from the source altogether so will not compile. If we are missing some that are in the source, and are in the subset definition files, then we have a new problem to address. I don't see any ASCII in there which is promising :)

chrissimpkins commented 7 years ago

@burodepeper @jublo web font (full set) and web font subset builds (both woff and woff2) are available in :

https://github.com/source-foundry/Hack/tree/dev-build-scripts/build/web

Build: Version 3.000; 2a97b0d-dev

The fonts and css directories are outdated. Everything new exists in the top level of that directory and the naming scheme is:

Full set: hack-[VARIANT].woff(2) Subset: hack-[VARIANT]-subset.woff(2)

We expect the subset builds to contain all ASCII + Latin-1 + Latin Ext A that are included in the Hack source files. Source files lack some of the glyphs in the extended sets.

We need to know:

If you happen to need them for comparison, desktop ttf builds are in the same branch build/ttf directory. Version string tagged as Version 3.000; 2a97b0d-dev (and appropriately built off of source as of that commit sha1)

See David's work above on contents of web font builds.

related IR: https://github.com/source-foundry/Hack/issues/307

chrissimpkins commented 7 years ago

@jublo can you let me know if https://github.com/source-foundry/Hack/issues/286#issuecomment-329308449 is something you can support with your charset grid tool? This would be extremely useful to complete the work in this thread and is next up on my plate to address (manually if I need to do it).

mynetx commented 7 years ago

@chrissimpkins The following charts are based on your recent WOFF2/WOFF builds.

bildschirmfoto 2017-09-17 um 01 45 45 bildschirmfoto 2017-09-17 um 01 46 09 bildschirmfoto 2017-09-17 um 01 46 26 bildschirmfoto 2017-09-17 um 01 47 14 bildschirmfoto 2017-09-17 um 01 47 44 bildschirmfoto 2017-09-17 um 01 48 30 bildschirmfoto 2017-09-17 um 01 48 47 bildschirmfoto 2017-09-17 um 01 49 01

chrissimpkins commented 7 years ago

:1st_place_medal:

This is absolutely fantastic @jublo! This looks like the complete web font sets. It looks like we are missing the middle dot in the bold italic set?

Is it possible to show Latin-1, Latin Ext A for the web font subsets to confirm our coverage there? This will confirm that the subsetting process that is part of the current dev-build-scripts branch changes is working as intended.

According to these images, we have one glyph in bold italic to complete Latin-1 and a handful of glyphs across the four font variants to reach complete coverage of Latin Ext A. I wonder what languages the missing glyphs support?

The long S in the Latin A set is of historic interest only (but interestingly there was a IR a couple of months ago for the glyph from someone who was doing work with historic texts). It is not critical, but would address that user's request if there is interest in drawing it.

chrissimpkins commented 7 years ago

Also handful missing for complete coverage of MES 1. Lots missing for MES 2. It looks like some of the MES 2 set are greek glyphs? I know that we have these in there. I wonder if these are different Unicode code points for similar shapes?

chrissimpkins commented 7 years ago

Also confirming that these are the builds in this branch? https://github.com/source-foundry/Hack/tree/dev-build-scripts/build/web

We should be using the fonts in the top level of that directory. The sub directories are of historic interest and have old builds. They will be going away (will remove them now in that branch since this is a good reminder for me...)

mynetx commented 7 years ago

@chrissimpkins 

@font-face {
  font-family: 'Hack Test Regular';
  src: url('https://github.com/source-foundry/Hack/blob/dev-build-scripts/build/web/hack-regular-subset.woff2?raw=true') format('woff2');
  font-weight: 400;
  font-style: normal;
}
mynetx commented 7 years ago

Is it possible to show Latin-1, Latin Ext A for the web font subsets to confirm our coverage there?

@chrissimpkins What do you mean – the first two images are for Latin-1 and Latin Ext A from the -subset files, see CSS above?

chrissimpkins commented 7 years ago

Ah, got it. So top two images indicating Latin-1 and Latin Ext A are displayed from hack-*-subset.woff2 and the rest of the images are from hack-*.woff2?

If so we are in good shape

mynetx commented 7 years ago

Actually all of them should be from the subset files. Let me just compare!

chrissimpkins commented 7 years ago

Possible to add the ranges for the general punctuation glyphs and the currency glyphs?

chrissimpkins commented 7 years ago

Actually all of them should be from the subset files. Let me just compare!

Hmmm... our subsetter may not be working properly then. We should not be getting Latin Ext B in them.

chrissimpkins commented 7 years ago

The subsets as defined at the moment should include the following

chrissimpkins commented 7 years ago

The definitions are in the lib.plist files under the XML key <key>com.schriftgestaltung.Keep Glyphs</key>

located in this directory:

https://github.com/source-foundry/Hack/tree/dev-build-scripts/source/subset-lib

I entered these by hand so it is possible that there are errors there. Unfortunately we have to use glyph names and the names are not necessarily their Unicode code points (were defined by Glyphs). Some are pseudonames, some are Unicode code point names. There could be some missing due to error, but I know that there are not extras that should include the Latin-B extensions outside of the sets defined above.

mynetx commented 7 years ago

More details on the differences between Subset and full fonts (WOFF2 and TTF from your GH build folder):

bildschirmfoto 2017-09-17 um 03 49 28 bildschirmfoto 2017-09-17 um 03 49 35 bildschirmfoto 2017-09-17 um 03 49 46 bildschirmfoto 2017-09-17 um 03 50 30 bildschirmfoto 2017-09-17 um 03 53 28 bildschirmfoto 2017-09-17 um 03 53 44 bildschirmfoto 2017-09-17 um 03 56 17 bildschirmfoto 2017-09-17 um 03 56 43

chrissimpkins commented 7 years ago

@burodepeper thoughts on the current coverage in the web font subsets David? Looks like we are in fairly good shape for the expected coverage based upon the images that @jublo posted in https://github.com/source-foundry/Hack/issues/286#issuecomment-330007092

chrissimpkins commented 7 years ago

For some reason, middle dot seems to be missing from the bold italic subset but is included in the full set. This is likely my error in the subset list. I will confirm this.

mynetx commented 7 years ago

@chrissimpkins I could do with a list of code points to add that are missing in either font part, so I could look at drawing/adapting them?

chrissimpkins commented 7 years ago

@jublo fix for middle dot (U+00B7) is now available in https://github.com/source-foundry/Hack/pull/320

New build files are part of that PR (and currently located in dev-middledot-patch branch so that we can examine the sets to confirm that we now have all Latin-1 glyphs in all sets. If no issues with this PR, will plan to merge to dev-build-scripts.

chrissimpkins commented 7 years ago

@jublo glyphs missing from all variant sets that are necessary to complete Latin Ext A: