squidfunk / mkdocs-material

Documentation that simply works
https://squidfunk.github.io/mkdocs-material/
MIT License
20.93k stars 3.54k forks source link

Currently not GDPR compliant - reason external fonts #739

Closed DsrMedia closed 6 years ago

DsrMedia commented 6 years ago

Description

As in EU the GDPR starts to be active from May 25th on. It would be good if all external fonts and services are part of the bundle and do not link to the CDN version. Otherwise we have to display a cookie consent and a privacy policy for a "documentation".

Expected behavior

Please include the Material Icons and Roboto Font directly (and other external services) into the bundle at least everything which is needed for the default material theme. Currently it is not GDPR compliant how it is right now.

Actual behavior

It's loaded from a CDN.

Steps to reproduce the bug

In the Base.html (maybe in other files as well) from the material design there are links to the CDN services.

wopian commented 6 years ago

Google Fonts don't use Cookies though?

squidfunk commented 6 years ago

I'm a European citizen and highly critize the EU for issueing such a half-baked law. It will cost companies millions in implementation and legal fees. It's impractical to host everything by yourself. Is it even allowed to self-host Google Fonts per their license? Don't really know if I want to take any action here.

DsrMedia commented 6 years ago

@wopian yes thats true, they are not using cookie, but they get the IP-address from the user. But the font awesome icons are setting a cookie, as there are hosted on cloudflare (partials/social.html).

@squidfunk Yes I aggree, but still as company we have to deal with it as it is. For our side we adapted the base.html and the social.html and downloaded the necessary fonts to the assets folder.

squidfunk commented 6 years ago

@DsrMedia: I totally understand.

However, self-hosting all assets would mean we would need to keep everything in sync. Also when self-hosting Google Fonts we may loose rendering quality because Google's CDN serves distinct files for Windows and macOS for optimization reasons. You can disable Google Fonts entirely as described in the getting started guide.

We could host FontAwesome by ourselves, as it's pinned to a specific version and upgrading to 5 would mean a major release of the theme because class names / qualifiers changed. Would that help?

max-ci commented 6 years ago

@squidfunk FYI http://michaelboeke.com/blog/2013/09/10/Self-hosting-Google-web-fonts/

squidfunk commented 6 years ago

Oh and we would loose the ability to easily change the font face per configuration or would have to bundle all fonts.

nikramakrishnan commented 6 years ago

Edit: This is not legal advice, and does not mean you can continue using Google Fonts in your project.

I did some research and found a few links to point that it should not be a problem to use Google Fonts even after GDPR takes effect.

  1. Google Fonts FAQ Section on Privacy

  2. Discussion on GDPR Compliance on the Google Fonts GitHub Repository

  3. Google GDPR Compliance Notice

I strongly believe that we should continue using Google Fonts, because it offers fast content distribution and optimizations as mentioned by @squidfunk

Google should be releasing a statement of compliance soon.

squidfunk commented 6 years ago

Yes, we will definitely be using Google Fonts at least until May 25, and then reconsider how we use/advertise it depending on the statement of compliance Google will hopefully be issuing soon.

I will close this issue for now because there's nothing that can be done right at the moment. Furthermore, Google Fonts can be disabled easily by setting the following parameter in mkdocs.yml:

theme:
  font: false
justjanne commented 6 years ago

@squidfunk Considering your completely immature response to this, which leaves users up to liability, I am very disappointed in this project.

For my own use cases I have now created a fork that does not use Google Fonts.

squidfunk commented 6 years ago

@squidfunk Considering your completely immature response to this, which leaves users up to liability, I am very disappointed in this project.

If you mean my initial reaction: I think GDPR is great in theory but is an entire gray area as there are no official judgments or guidelines what is okay and what is not.

If you mean my intention to wait until it's clear what to do I cannot understand your anger. Google Fonts can be easily disabled and replaced by self hosted fonts (see my last comment) and there's definitely a plan to include FontAwesome into the distribution files (but that's only CSS, so probably not as critical as Google Analytics).

justjanne commented 6 years ago

@squidfunk I mean your entire attitude towards a privacy and legal issue, which is "I don’t care". Completely disregarding the users that are affected by it.

I can’t trust a project that might end up sending user’s data to who-knows-where because "it can be easily disabled and replaced".

squidfunk commented 6 years ago

Okay, I see your point. Actually it's not that I don't care, I'm very aware of the issue. The problem is that many people are using Material and the changes that we discussed would have quite an impact. My aim is to keep things as stable and constant as possible. For this reason we decided to settle until we know how to implement this with maximum compatibility and optimal usability. I'm very concerned about usability.

Could you make a constructive proposal how we could comply with GDPR and keep Material easy to use? Also, is Google Fonts not GDPR compliant? Did someone have the time to study this topic?

squidfunk commented 6 years ago

I think we should issue a statement in the docs on GDPR compliance of Material and how to achieve it.

nikramakrishnan commented 6 years ago

There's been a lot of debate about this on the Google Fonts repository issue that I linked before. Although Google has issued several official statements (prefixed to the first comment), but these statements do not directly address the problem, and do not make it clear as to whether we can use Fonts in projects.

This discussion on CIDRAM proposes to disable use of Google Fonts by default and add a clause to the Privacy Policy about enabling it.

Again, this is not legal advice, and we need to be careful!

squidfunk commented 6 years ago

Thanks! What about if we include the default Roboto font family with the Material distribution, so by default Material would not query the Google Fonts CDN. If the fonts are set via mkdocs.yml, we would link to Google Fonts and add an Admonition to the docs that this possibly breaks GDPR compliance. Font Awesome will be directly included. This way all external dependencies could be eliminated by default and the UX would stay the same.

Any thoughts?

nikramakrishnan commented 6 years ago

I believe this is what we should do till Google and other providers clear the issue.

Can the font parameter be programmed to enable/disable CDN for Font Awesome, too? There may be developers who may decide not to disable CDN, for faster loading.

justjanne commented 6 years ago

That would be appreciated, yes. For now I’ve taken down my sites that use this until the issue is resolved. (But I’d want to keep an option to stick with entirely self-hosted everything even if Google sorts this out, and especially without any analytics or tracking).

squidfunk commented 6 years ago

Can the font parameter be programmed to enable/disable CDN for Font Awesome, too? There may be developers who may decide not to disable CDN, for faster loading.

That makes it more complicated so I would favor always including it. However, you could always override the theme.

squidfunk commented 6 years ago

@justjanne: just disable Google Fonts and you are fine! No need to take it down.

justjanne commented 6 years ago

@squidfunk If I disable Google Fonts, all icons will be missing and more.

I already am running a heavily modified version here so I can get KaTeX support (instead of MathJax – ideally I’d even be able to get KaTeX at build time, but that’s a bit more work), and so I can change a few other minor details (e.g. making the icon at the top of the docs link back to the main site of the project instead of the docs home)

So, I’ve taken it all down for now, and then I’ll wait until the fix, merge it into my version, and run that.

Omeryl commented 6 years ago

Wouldn't having the information be available, albeit not pretty, be much preferred to unavailable completely? @justjanne

squidfunk commented 6 years ago

My current plan to ensure GDPR compliance is the following:

This should ensure GDPR compliance for the default configuration - meaning no custom fonts, no analytics – no third-party at all. I will also add documentation for this. If the user changes the fonts, Google Fonts will be used. This will also be noted in the docs. This will make the following option obsolete:

theme:
  font: false

It will be deprecated with 3.0, as no fonts are loaded by default.

Any thoughts?

vbd commented 6 years ago

I think it would also be helpful to briefly explain how to include other fonts without using google fonts.

squidfunk commented 6 years ago

So this is turning out harder than I thought. The problem is the way Google Fonts serves web fonts - they analyze the user agent and serve the ideal compatible web font format. For each requested font weight they serve 7 font face definitions for different charsets, e.g. latin, cyryllic, etc:

/* cyrillic-ext */
@font-face {
  font-family: 'Roboto';
  font-style: italic;
  font-weight: 400;
  src: local('Roboto Italic'), local('Roboto-Italic'), url(https://fonts.gstatic.com/s/roboto/v18/KFOkCnqEu92Fr1Mu51xFIzIXKMnyrYk.woff2) format('woff2');
  unicode-range: U+0460-052F, U+1C80-1C88, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F;
}
/* cyrillic */
@font-face {
  font-family: 'Roboto';
  font-style: italic;
  font-weight: 400;
  src: local('Roboto Italic'), local('Roboto-Italic'), url(https://fonts.gstatic.com/s/roboto/v18/KFOkCnqEu92Fr1Mu51xMIzIXKMnyrYk.woff2) format('woff2');
  unicode-range: U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116;
}
/* greek-ext */
@font-face {
  font-family: 'Roboto';
  font-style: italic;
  font-weight: 400;
  src: local('Roboto Italic'), local('Roboto-Italic'), url(https://fonts.gstatic.com/s/roboto/v18/KFOkCnqEu92Fr1Mu51xEIzIXKMnyrYk.woff2) format('woff2');
  unicode-range: U+1F00-1FFF;
}
/* greek */
@font-face {
  font-family: 'Roboto';
  font-style: italic;
  font-weight: 400;
  src: local('Roboto Italic'), local('Roboto-Italic'), url(https://fonts.gstatic.com/s/roboto/v18/KFOkCnqEu92Fr1Mu51xLIzIXKMnyrYk.woff2) format('woff2');
  unicode-range: U+0370-03FF;
}
/* vietnamese */
@font-face {
  font-family: 'Roboto';
  font-style: italic;
  font-weight: 400;
  src: local('Roboto Italic'), local('Roboto-Italic'), url(https://fonts.gstatic.com/s/roboto/v18/KFOkCnqEu92Fr1Mu51xHIzIXKMnyrYk.woff2) format('woff2');
  unicode-range: U+0102-0103, U+0110-0111, U+1EA0-1EF9, U+20AB;
}
/* latin-ext */
@font-face {
  font-family: 'Roboto';
  font-style: italic;
  font-weight: 400;
  src: local('Roboto Italic'), local('Roboto-Italic'), url(https://fonts.gstatic.com/s/roboto/v18/KFOkCnqEu92Fr1Mu51xGIzIXKMnyrYk.woff2) format('woff2');
  unicode-range: U+0100-024F, U+0259, U+1E00-1EFF, U+2020, U+20A0-20AB, U+20AD-20CF, U+2113, U+2C60-2C7F, U+A720-A7FF;
}
/* latin */
@font-face {
  font-family: 'Roboto';
  font-style: italic;
  font-weight: 400;
  src: local('Roboto Italic'), local('Roboto-Italic'), url(https://fonts.gstatic.com/s/roboto/v18/KFOkCnqEu92Fr1Mu51xIIzIXKMny.woff2) format('woff2');
  unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD;
}

As Material is a theme used in many different languages, we cannot possibly exclude any of the definitions. This means that for each weight, we would need to include 7 font face definitions. Additionally, this is only woff2 format - we would also need to serve woff and ttf at least (possibly eot if we want to support older IEs, but let's just forget about that). Material currently loads the weights 300, 400, 400i, 700 for Roboto and 400 for Roboto Mono, so 5 web fonts in total.

The total amount of webfonts to be bundled with Material is:

5 (weights) x 7 (charsets) x 3 (formats) = 105 web font files. Impractical.

I don't really know how we should proceed. I'm currently thinking of dropping Roboto for the default configuration (however still to be enabled via mkdocs.yml to use Google Fonts) and fall back to Helvetica or the respective system font. This would make Material GDPR compliant in its default configuration and still leave the possibility to use webfonts.

Personally, I think, from a user perspective, it may be better to leave the Google Font loading as-is (to be disabled via theme.font: false) and bundle all icons fonts. This way GDPR compliance can be ensured with a single line, but the default user experience is not crippled as it would with falling back to system fonts.

Any thoughts?

squidfunk commented 6 years ago

I think it would also be helpful to briefly explain how to include other fonts without using google fonts.

@vbd: you can just override the fonts block with your own link and style tags, see: https://squidfunk.github.io/mkdocs-material/customization/#overriding-template-blocks

This is the default code generated by the block - you would have to provide your own definitions here:

https://github.com/squidfunk/mkdocs-material/blob/2fbeb28d9dbda3e27fbe78dfa7cd53e76db8dab2/src/base.html#L116-L138

nikramakrishnan commented 6 years ago

That is a lot of font files!

More discussion about Google Fonts

Your idea of leaving the Google Fonts loading as-is makes much more sense now. Roboto is at the heart of Material Design, after all.

If that is being done, clear warnings could be added to the docs, along with the directive required to disable it.

squidfunk commented 6 years ago

Yes, I will provide an entire section on GDPR compliance and how to achieve it with Material

squidfunk commented 6 years ago

2.8.0 was just released which bundles both icon fonts. This means that the font configuration option and GA and Disqus integration are the only things that break GDPR compliance. The first must be explicitly disabled, the latter both are disabled by default. See the new section on compliance and feel free to ask any questions so we can mark this topic as done.

See https://squidfunk.github.io/mkdocs-material/compliance/

squidfunk commented 6 years ago

Closing this as fixed. In case any issues or problems arise, feel free to re-open.

sheldonhull commented 5 years ago

@squidfunk As a lurker just have to saw your communication and resolution was stellar. Thanks for this?

dirkroorda commented 5 years ago

Thank goodness that you kept the Google Font config option. Whether laws are good or bad, too much respect for the letter of the law is the end of civil society.

squidfunk commented 5 years ago

@dirkroorda one year after the advent of GDPR and we're still alive. There's a German article which sums it up pretty well. A few fines were imposed, but all-in-all pretty smooth.

seekM commented 4 years ago

If I'm not mistaken it was said in this discussion that material bundles font files for GDPR compliance.

However, using this simple configuration

site_name: Test
nav:
    - Test: index.md

theme:
  name: material

I noticed that it seems that requests at fonts.googleapis.com and fonts.gstatic.com are made:

gfont

I didn't expect this to happen - am I missing something here?

Note that https://squidfunk.github.io/mkdocs-material/compliance/ isn't working anymore.

martinbira commented 4 years ago

@seekM Please check this page: https://squidfunk.github.io/mkdocs-material/data-privacy/

ajh0912 commented 2 years ago

@squidfunk It might be worth re-visiting this topic now that Google Fonts is in the spotlight again. 1, 2, 3

Even if no changes are made directly to Material for MkDocs, maybe elaborating on the docs to points people in the right way for self-hosting the Google Fonts specifically? Although I would understand if maybe you'd prefer not to link to a third party.

For my uses I self-hosted the fonts that I needed manually as per https://squidfunk.github.io/mkdocs-material/setup/changing-the-fonts/#additional-fonts, with font: false in mkdocs.yml. The CSS and font files were generated via https://google-webfonts-helper.herokuapp.com/fonts

squidfunk commented 2 years ago

Google Fonts can be easily disabled. Self-hosting is always an option and as you mentioned, all information is described in the docs. I don't see anything else that needs to be added to the documentation, or is there?

ajh0912 commented 2 years ago

Please let me know if I'm missing something, but would it not be possible bundle Roboto in weights 300 400 400i 700 and Roboto Mono weight 400 both in charsets vietnamese latin-ext latin greek cyrillic-ext cyrillic with the format of woff2 and still cover all browsers currently supported? I believe it would involve 5 font files of total ~240KB. Including woff as well would make it a total of 10 font files and ~540KB.

It appears woff2 is currently supported by ~94.9% of user's browsers, with woff at 96.7%.

Roboto, Roboto Mono (note the 'Modern Browsers' option).

Only reason I'm mentioning that is, it's much more practical than the previously mentioned bundling of 105 font files and their corresponding CSS. I understand that the browsers you support has probably modernised since 2018 and that's likely why we can get away with only 1 or 2 font formats now, that and the combining of multiple charsets into a single font file.

I would understand if you still preferred not to bundle these, as I know it would be another dependency when building (although updates to the font itself are infrequent), and the increase of either 240KB or 540KB for the project. Another aspect is the request from the browser itself to the web server will now be asking for more data directly, whereas it would have got a quick CDN or browser cache hit if using Google Fonts.

squidfunk commented 2 years ago

@ajh0912 this is exactly what I analyzed it https://github.com/squidfunk/mkdocs-material/issues/739#issuecomment-396045729. You would have two options.

1. Bundle all variants

= 48 font files. At least. The numbers in the linked comment are higher because I included more weights and charsets. Remember that Material for MkDocs is used in 50+ languages, so we cannot possibly loose support for some languages by removing some character sets, we need to support all of them to some degree.

2. Bundle woff/ttf including all charsets

Roboto-Regular.ttf weighs 164kb. Take that times 6 (4 + 2 weights), you're at 984kb. woff and woff2 are possibly a little smaller than ttf, but not an order of magnitude. Currently, Material for MkDocs without fonts weighs 58kb (gzipped), i.e. all CSS and JS sources. woff and woff2 cannot be gzipped, as they're binary formats. Each font loaded from Google fonts adds 11kb per font, so a total of around 66kb for all fonts (for the regular latin charset), leading to:

That's an order of magnitude, and only for Roboto, so you would be locked in to use this font.

Recommendation

Turn font loading off and fall back to system fonts. If that's not an option, you can download the font files and bundle them yourself, but your website will likely receive worse scores on Lighthouse and/or Google.

Personal opinion

This ruling is pretty interesting because it would mean that all use of CDNs without the visitor's consent could be considered "leaking the visitors IP", at least as I read it. Thus, all use of external assets (not services, assets!), be it from third-party cookie consent providers, analytics integrations, etc., basically, everything hosted on external servers must be considered problematic at least. I believe that this ruling will not be persistent. Thus, I would be patient and wait for how this is handled.

Note that this is my personal opinion and I'm not an expert in legal affairs, but it might help to understand my argument.


Disclaimer: those numbers are estimates, so the actual numbers might be a little off, but the ratio roughly stays the same.

ajh0912 commented 2 years ago

@squidfunk The weights and charsets I had referenced were entirely based on what you had mentioned in that comment from 2018, apologies I could have made it clearer that I was basing things from it (that's where the '105 font files' reference came from).

The main point was if self-hosting the fonts, could it not be reduced down to only using woff2 and still work with all the browsers currently outlined as supported? If there's no problems solely using woff2, then the self-hosting of fonts only uses 6 font files of total ~273KB. No it's not as slim as Google Fonts that's for sure - but just checking there's no immediate flaws with using:

Roboto in weights 300 400 400i 700 Roboto Mono weights 400 700 Both in charsets vietnamese latin-ext latin greek cyrillic-ext cyrillic with solely the format woff2.

But for example if you bolded and italicised something, it would be missed by those weights right? If so the variant 700i might need included depending on each site's uses.

Thanks for mentioning about the 700 in Roboto Mono as well, things like this are why I'd appreciate a simple mention in the docs for the bare minimum required without Google Fonts. I understand you won't want to bundle these into the project itself as it bloats it considerably, but I was thinking a simple mention of 'for example self-hosting the fonts for this documentation, the minimum viable would be x, although each use case may differ'.

squidfunk commented 2 years ago

You can open a PR if you like.

My main concern is that we overcomplicate the setup, which might cause more trouble than help. It might be more issues for me to answer after all, so if we can find something short and concise, I'm happy to merge it, but I doubt it, due to how use case-specific font setup might be, given your target language and demographic.

justjanne commented 2 years ago

My main concern is that we overcomplicate the setup, which might cause more trouble than help

The setup for a GDPR-compliant mkdocs installation is extremely overcomplicated, requiring manual customization and importing files.

The goal would be to uncomplicate a GDPR-compliant mkdocs installation

This ruling is pretty interesting because it would mean that all use of CDNs without the visitor's consent could be considered "leaking the visitors IP", at least as I read it. Thus, all use of external assets (not services, assets!), be it from third-party cookie consent providers, analytics integrations, etc., basically, everything hosted on external servers must be considered problematic at least. I believe that this ruling will not be persistent. Thus, I would be patient and wait for how this is handled.

As I already wrote in 2018, this ruling was to be expected and is a natural consequence of BDSG (2006) and GDPR (2016). This has been in court countless times thanks to noyb and Max Schrems, waiting it out won’t solve it.

squidfunk commented 2 years ago

The setup for a GDPR-compliant mkdocs installation is extremely overcomplicated, requiring manual customization and importing files.

theme:
  font: false

That's everything you need. Google Analytics itself is paired with the cookie consent, so if you use the cookie consent, you can also safely use Google Analytics. Furthermore, cookies for third-parties can be easily integrated.

squidfunk commented 2 years ago

@justjanne (comment deleted afterwards)

Does that get me the same fonts and results as the google font-included version? No. I need to manually add them back, which is significantly more complicated.

Correct. Please refer to https://github.com/squidfunk/mkdocs-material/issues/739#issuecomment-1027713955 to understand why it's (currently) not an option.

And considering France and the Netherlands now have also followed Austria and declared use of third-party services or assets without manual consent illegal, so you shouldn’t expect this to change soon (and should instead provide an option where mkdocs downloads the required fonts during build time, potentially even subsetting only the required parts depending on which text is actually used in the project)

You're invited to create a PR. If you manage to pull this off, I'm happy to merge it. However, doing this during build time would mean that we would need to implement this as part of a plugin (otherwise we cannot hook into the build process), in order to pull out the glyphs for the configured language(s). I suspect that it's significantly more complicated than just switching off the fonts and falling back to system fonts.

If you wish to include Roboto yourself, it's not complicated at all. You don't even need theme extension – just add a custom CSS file, a @font-face definition and put the relevant *.woff files into your assets folder.

If you don't wish to do it, you're free to use any other static site generator you like.

squidfunk commented 2 years ago

If anyone wishes to self-host – here's a great site that helps download a customized version for your target languages: https://google-webfonts-helper.herokuapp.com/fonts/roboto

  1. Select the character sets you really need
  2. Select styles 300, 400, 400i and 700
  3. Copy the CSS and add it in an additional stylesheet
  4. Download the fonts and put them in your docs_dir relative to the CSS file

Repeat for the monospaced font with the weights you need (probably 400 and 700)

Sadly, this is a manual process and not automatable. Again, if we would automate it, we would either need to determine the necessary charsets (very complicated) or bundle all charsets (too big, your Lighthouse score might suffer). Turns out it is automatable, albeit it was quite tough. See my next comment.

squidfunk commented 2 years ago

Insiders 4.9.0 adds a new plugin: the built-in privacy plugin – the plugin will automatically download all external scripts, style and web fonts, so you can self-host them as part of your documentation without manual effort. The idea is to make compliance with GDPR easier, given the latest developments when serving external assets to users without prior consent.

Activate it via mkdocs.yml with:

plugins:
  - privacy

This was really tough. I'm very interested in your feedback, but the plugin should at least solve the font hosting problem once and for all. I've investigated and it seems that we only need *.woff2, since this covers the whole range of browsers that we are officially supporting.

Of course, it's still possible to achieve self-hosting manually without the use of the plugin, which I described in my last comment. The cool thing is that the plugin will download all assets regardless of where they are defined – in the template, as additional JavaScript or style sheets, or in Markdown – when they're external, they're downloaded.

MaxWinterstein commented 1 year ago

If anyone wishes to self-host – here's a great site that helps download a customized version for your target languages: https://google-webfonts-helper.herokuapp.com/fonts/roboto

  1. Select the character sets you really need
  2. Select styles 300, 400, 400i and 700
  3. Copy the CSS and add it in an additional stylesheet
  4. Download the fonts and put them in your docs_dir relative to the CSS file

Repeat for the monospaced font with the weights you need (probably 400 and 700)

~Sadly, this is a manual process and not automatable. Again, if we would automate it, we would either need to determine the necessary charsets (very complicated) or bundle all charsets (too big, your Lighthouse score might suffer).~ Turns out it is automatable, albeit it was quite tough. See my next comment.

For me, this still sends a request to google, see https://github.com/squidfunk/mkdocs-material/blob/master/material/base.html#L57

only by using

mkdocs.yaml

...
theme:
  font: false
...

extra.css

:root > * {
  --md-text-font:"Roboto";
  --md-code-font:"Roboto Mono"
}

It works and there are zero outgoing connections.