Optimize Error Pages - Reduce content size

[ferristocrat]

Summary

• Our error pages are really large so even though we are implementing rate limiting when we serve the error page (such as 429s) we are still suffering from too much load
• We're serving 16gbps of error pages

Acceptance Criteria

• Identify and implement optimizations to error pages to reduce the size

Possible Solutions

• Stripping away unnecessary bootstrap (use only what we need if we are just using the entirety of bootstrap)
• Serve bootstrap (and any other large files) from CDN
• Don't use bootstrap
• Get rid of background image

[storj-gerrit[bot]]

Change pkg/linksharing/sharing: optimize 429 responses mentions this issue.

[storj-gerrit[bot]]

Change pkg/linksharing/web: serve bootstrap from CDN mentions this issue.

[halkyon]

Should we keep this open for a few more improvements? The static background could be another thing to optimize.

@amwolff, did you intend for it to close after that change?

[egonelbre]

Are we missing caching headers for the content? I don't see a reason why they should be such a big impact.

Also, are we missing compression for the assets?

[halkyon]

Few other points mentioned in a Slack thread:

• Make sure cache headers are correct (this will probably solve a lot of the issue)
• Compress assets?
• Strip down bootstrap to only whatever it is we use.
• remove the need for leaflet.js and css (is it used, can it be replaced for the map?)
• Map loading is not async
• collapse.svg / expand.svg are served, but these could be replaced by symbols in the font

See the PageSpeed report here: https://pagespeed.web.dev/analysis/https-link-storjshare-io-s-jwur4oa5atf5ireddlrxtk4g7yqq-uptime-checks-hello-world-txt/kny8inzxcm?form_factor=mobile

Generally it might be good to check out the network tab in the browser and see if we can spot anything else that looks like a big download or not optimal.

[storj-gerrit[bot]]

Change pkg/linksharing: enable caching for static files mentions this issue.

[halkyon]

I found https://github.com/vearutop/statigz that looked good, but it didn't have cache busting features.

Egon had a great idea (thanks!), that looks something like this:

type StaticCache struct {
    prefix string // if needed
    data fs.FS
    mu   sync.Mutex
    comp map[string]Entry
}

type Entry struct {
    Path        string
    Etag        string
    ContentType string

    Data []byte
    Gzip []byte
}

func (cache *StaticCache) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Allow", http.MethodGet + "," + http.MethodHead)
    if r.Method != http.MethodGet || r.Method != http.MethodHead {
        http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
        return
    }

    entry, ok := cache.Entry(r.URL.Path)
    if !ok {
        http.Error(...)
        return
    }

    w.Header().Add("Content-Type", entry.ContentType)
    w.Header().Add("Etag", entry.Etag)
    // add caching header

    data := entry.Data
    if acceptsGzip(r) && len(entry.Gzip) > 0 {
        w.Header().Add("Content-Encoding", "gzip")
        data = entry.Gzip
    }

    if r.Method == http.MethodHead {
        return
    }

    w.Write(data)
}

func (cache *StaticCache) Link(urlpath string) template.HTML {
    // return canonicallink + "?etag" + etag
}

func (cache *StaticCache) Entry(urlpath string) (Entry, bool) {
    target := path.Clean(urlpath)

    cache.mu.Lock()
    entry, ok := cache.comp[target]
    cache.mu.Unlock()

    if !ok {
        entry, ok = cache.load(target)
        if !ok {
            return Entry{}, false
        }

        cache.mu.Lock()
        cache.comp[target] = entry
        cache.mu.Unlock()
    }

    return entry, true
}

func (cache *StaticCache) load(target string) (Entry, bool) {
    // load data
    // compute some hash for etag
    // compress gzip
    // determine content type based on ext
}

func acceptsGzip(r *http.Request) bool {
    // todo:
}

[ferristocrat]

@boshevski - let's take a look at the Linksharing UI and discuss if it needs a refresh to make it more consistent with the Satellite UI.

[boshevski]

@ferristocrat I made a pr that optimizes the logo and the background image 3x. I'm working on updating the ui designs next

[egonelbre]

AFAIK, https://github.com/storj/gateway-mt/issues/361#issuecomment-1718344528 has not yet been implemented or will that be handled in a separate issue?

[halkyon]

yes, this isn't complete. I reopened it.