🤗 Static files compression with brotli

[suntong]

Question description

Following up with https://github.com/gofiber/fiber/issues/699, how to make fiber to compress with brotli?

Code snippet Optional

Please take a look at https://github.com/suntong/fiber_demo/blob/612a8f828cd8d4c0ab73d9f57215029a587e62f2/app/static-br.go#L23-L25

i.e.,

    app.Static("/sample-br0", "../web/sample0", fiber.Static{
        Compress: true,
    })

when I view it with my chrome, which suport deflate, gzip and brotli, I see only gzip is used.

More over, the next block https://github.com/suntong/fiber_demo/blob/612a8f828cd8d4c0ab73d9f57215029a587e62f2/app/static-br.go#L26-L28, it should works with my chrome as well, but it doesn't. I checked the http header it is still saying gzip. What could be wrong? How to fix it?

Again, the go code serves directly to http://localhost:3000/, and I visit http://localhost:3000/ directly with my chrome from my local machine. There is no other intermediate intervention (e.g. proxy) involved.

[thomasvvugt]

Hi @suntong,

The router registers the Static route in these lines, copying over FastHTTP's FS struct; https://github.com/gofiber/fiber/blob/master/router.go#L263-L288

After taking a quick look at FastHTTP's source, I can conclude that FastHTTP only allows GZip compression for Static routes depending on the Compress setting; https://github.com/valyala/fasthttp/blob/master/fs.go#L989

We supply a custom Compression middleware within Fiber to compress your requests with Brotli; https://github.com/gofiber/fiber/blob/master/middleware/compress.md

You can see our configuration's related settings on Fasthttp's end on these lines in the source; https://github.com/gofiber/fiber/blob/e98a1b1bb367934a79bdf2ec7418f461c4b379ee/middleware/compress.go#L30-L35

Example code;

// Default compression config
app.Use(middleware.Compress())

// Register static route
app.Static("/sample-br0", "../web/sample0")

Since we already use the compression middleware to compress any requests going in, we can ignore the Compress setting while registering the static route.

Let me know if the above example helps you out, thanks for bringing up the issue! 🚀

[suntong]

Thanks for the fast respose, @thomasvvugt

Yep, I'm seeing Content-Encoding: br now, after following your sample.

Now to the second part of my question, you said,

FastHTTP only allows GZip compression for Static routes depending on the Compress setting... We supply a custom Compression middleware within Fiber to compress your requests with Brotli

That means the Static files can only be pre-compressed with GZip, not Brotli, at least for now, right? Just want to confirm that. Feel free to close it for now, or wait till Brotli pre-compressed Static files are supported to close.

[suntong]

Ah, both work! Thanks again!

[suntong]

That means the Static files can only be pre-compressed with GZip, not Brotli, at least for now, right? Just want to confirm that.

Hmm...

How can I make sure it is serving from pre-compressed Brotli files?

This is my index.html:

-rw-rw---- 1   0 1970-01-23 04:56 index.html
-rw-rw---- 1 241 2020-06-03 21:46 index.html.br

When I visit it, I get a blank page instead contents from index.html.br.

[leopku]

That means the Static files can only be pre-compressed with GZip, not Brotli, at least for now, right? Just want to confirm that.

Hmm...

How can I make sure it is serving from pre-compressed Brotli files?

This is my index.html:

-rw-rw---- 1   0 1970-01-23 04:56 index.html
-rw-rw---- 1 241 2020-06-03 21:46 index.html.br

When I visit it, I get a blank page instead contents from index.html.br.

You can testing it by just specifying Accept-Encoding in request header. For example:

❯❯❯ curlie :4455/api/v1/users 'Accept-Encoding: br'                                                                ✘ 130 master ✱ ◼
HTTP/1.1 200 OK
Date: Sun, 16 Aug 2020 07:31:57 GMT
Content-Type: application/json
Content-Length: 182
Content-Encoding: **br**

+-----------------------------------------+
| NOTE: binary data not shown in terminal |
+-----------------------------------------+

or by curl

❯❯❯ curl -i -H 'Accept-Encoding: br' http://localhost:4455/api/v1/users                                                  master ✱ ◼
HTTP/1.1 200 OK
Date: Sun, 16 Aug 2020 07:33:02 GMT
Content-Type: application/json
Content-Length: 182
Content-Encoding: **br**

��[3忞yMv9��j    �
np��X�  %���,�/>�<$�7�v�۞��HD����@��<�n�pn8�u�f:/J(t[�
                                                      �T�1Uk-����v� ��4���$�P:����k8�8����
>l��˜5�&������
              ����8��)<B�dew�_�L�`�%~�H��%

You can find the Content-Encoding in response headers.

[thomasvvugt]

Hi @suntong,

You indeed have to make sure the client is able to handle Brotli responses using the Accept-Encoding: br header like @leopku showed.

Also, I would like to point out that pre-compressing files using GZIP and serving them using Broli, takes up a fairly large amount of CPU power twice to compress your data.

A general starting point is that Brotli is better at compressing static data because of its superior compression ratio whereas GZIP is better at compressing dynamic data because of its often superior compression speed. (In-depth info here and here)

Therefore, I would recommend using Broli to compress your files served using app.Static(), whereas GZIP would outperform Broli using dynamic requests, such as app.Get('/home', func(c *fiber.Ctx) { c.Render("home", fiber.Map{ "User": "Some dynamic content" }) })

Let me know if that helps you out!

[suntong]

Thanks for the reply, @leopku & @thomasvvugt, but let's make sure we are talking about the same thing first. As I have already confirmed:

Yep, I'm seeing Content-Encoding: br now, after following your sample.

So honestly, I don't quite understand what point @leopku's reply is trying to make here. Don't get me wrong, I appreciate the reply, but I don't know how it can solve my specific question and what problem the reply solves as I've already been seeing Content-Encoding: br header.

Also,

I would like to point out that pre-compressing files using GZIP and serving them using Broli, takes up a fairly large amount of CPU power twice to compress your data.

I don't understand where this comment is coming from either as I've said:

This is my index.html:

-rw-rw---- 1   0 1970-01-23 04:56 index.html
-rw-rw---- 1 241 2020-06-03 21:46 index.html.br

I.e.,

I'm not pre-compressing files using GZIP but Broli instead.

So let me ask my question again,

How can I make sure it is serving from pre-compressed Brotli files?

This is my index.html:

-rw-rw---- 1   0 1970-01-23 04:56 index.html
-rw-rw---- 1 241 2020-06-03 21:46 index.html.br

When I visit it, I get a blank page instead contents from index.html.br, giving me a strong indication that fiber is serving from the empty index.html file instead of the pre-compressed Brotli file index.html.br.

Comments?

[Fenny]

The FileServer that is used within Static only supports deflate and gzip, you could write a middleware in front of it that appends .br to the path c.Path(c.Path() + ".br")

[Fenny]

@suntong here is a trick to compress static files with brotli only.

// v1.14.5
package main

import (
    "github.com/gofiber/fiber"
    "github.com/gofiber/fiber/middleware"
)

func main() {
    app := fiber.New()

    app.Use(middleware.Compress())

    app.Use(func(c *fiber.Ctx) {
        c.Fasthttp.Request.Header.Set(fiber.HeaderAcceptEncoding, "br")
        c.Next()
    })

    app.Static("/", "./public")

    app.Listen(":3000")
}

Let us know if you need further assistance, else you could close the issue <3

PS: v2.0.0 will be released on September 15th and contains some breaking changes. The following code will work for v2.0.0

// v2 ( September 15th )
package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/compress"
)

func main() {
    app := fiber.New()

    app.Use(compress.New())

    app.Get("/", func(c *fiber.Ctx) error {
        c.Request().Header.Set(fiber.HeaderAcceptEncoding, "br")
        return c.Next()
    })

    app.Static("/", "./public")

    app.Listen(":3000")
}

[suntong]

Thanks Fenny! Allow me to wait until September 15th to verify with v2 as well, then close it.

[Fenny]

package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/compress"
)

func main() {
    app := fiber.New()

    app.Use(compress.New())

    app.Get("/", func(c *fiber.Ctx) error {
        c.Request().Header.Set(fiber.HeaderAcceptEncoding, "br")
        return c.Next()
    })

    app.Static("/", "./public")

    app.Listen(":3000")
}

The above example will enforce br encoding.

before:

after:

[suntong]

Thanks @Fenny,

Any possibility to allow users to pre-compress files with Brotli first? -- It'll allowing doing Brotli compression only once instead of fiber doing it every time when serving them.

The above code works, but it seems to be doing compression every time when serving Static files -- This is my index.html:

-rw-rw---- 1   0 1970-01-23 04:56 index.html
-rw-rw---- 1 241 2020-06-03 21:46 index.html.br

When I visit it, I get a blank page instead contents from index.html.br, giving me a strong indication that fiber is serving from the empty index.html file instead of the pre-compressed Brotli file index.html.br.

Confirmed:

my index.html:

-rw-rw---- 1   4 1970-01-23 04:56 index.html
-rw-rw---- 1 241 2020-09-19 23:19 index.html.br

$ cat index.html
abc

When I visit the site, I get abc instead contents from index.html.br.

[rkakrik]

Same problem with pre-compressed files. Is there some way to do this?

[ReneWerner87]

Same problem with pre-compressed files. Is there some way to do this?

Maybe with the file suffix https://github.com/gofiber/fiber/blob/master/app.go#L233

If you create the files with the correct names, these should be used

[ReneWerner87]

Hmm maybe we have to support other suffixes for other encodings https://github.com/valyala/fasthttp/blob/master/fs.go#L320

[ReneWerner87]

If that helps, just create a feature request report to allow us to define the file suffix for other encodings.