Support compressed cmap files

[apigatto]

It would be convenient if cmap-js could read .gz files without the user having to decompress them. Could possibly be implemented with nodejs zlib.

[nathanweeks]

Minor correction: this could be done on the client using the Compression Streams API (which is unfortunately currently not supported by FireFox and Safari).

[nathanweeks]

Now it's only Firefox that doesn't support the Compression Steams API. But an alternative approach might be to force the Content-Type to text/plain, while setting the Content-Encoding to gzip.

cmap-js loads a data source with a Mithril.js request API:

https://github.com/LegumeFederation/cmap-js/blob/35212878a66e8eca61c7754e5f25149d0af1cd43/src/model/DataSourceModel.js#L72-L74

request() supports overriding HTTP headers such as Content-Type; see example at: https://mithril.js.org/request.html#non-json-responses

[nathanweeks]

Probably my lack of JS experience, but it seems difficult to use the (asynchronous) Compression Streams API within a DataSourceModel method. What doesn't work (in src/DataSourceModel.js): setting xhr.responseType = "blob" for .gz data files, and defining a Mithril request extract() method that decompresses the xhr.response; e.g.:

constructor(...) {
...
this.config = config || (url.endsWith(".gz")) ? function(xhr) { xhr.responseType = "blob"; } : "";
...
}
...
  extract(xhr, options) {
    if (xhr.responseType == 'blob') {
      const ds = new DecompressionStream('gzip')
      const response = new Response(xhr.response.stream().pipeThrough(ds));
      // does not work; await cannot be used within extract()
      xhr.responseText = await response.text();
    }
    return this.deserialize(xhr.responseText);
  }

(side node: I've only found stream/consumers documented for Node.js, but it seems to work in the browser as well?) (EDIT: used more-standard Response() instead of stream/consumers)

[nathanweeks]

Now it's only Firefox that doesn't support the Compression Steams API. But an alternative approach might be to force the Content-Type to text/plain, while setting the Content-Encoding to gzip.

Instead of setting the request header Content-Type: text/plain, it looks as though the way to force the response header Content-Type: text/plain might be to use the XMLHttpRequest overrideMimeType method (xhr.overrideMimeType("text/plain")) before the request is sent. I'm not sure how to do this with the Mithril request API, though.

[nathanweeks]

Deferring this for now in favor of an expedient solution where the http server is configured to set Content-Type: text-plain when the display=text query parameter is set; e.g.:

    <FilesMatch "\.gz$">
        <If "%{QUERY_STRING} =~ /display=text/">
          # https://httpd.apache.org/docs/2.4/env.html#special
          SetEnv no-gzip
          Header set Content-Encoding gzip
          Header set Content-Type text/plain
        </If>
    </FilesMatch>