bigwig file fetch error : Uncaught RangeError: byte length of Int16Array should be a multiple of 2

[jinghuage]

Hi,

I am trying to load a bigwig file to the dalliance browser (v0.13dev, chrome). I can load the file from local file system and configure it to use lineplot and looks all nice. But when I put it with the server, I got this error:

Uncaught RangeError: byte length of Int16Array should be a multiple of 2 bigwig.js:761 (anonymous function) bin.js:181 URLFetchable.fetch.req.onreadystatechange

I checked the file size, it's 26259 bytes. I have two other files, they both can load from server with no problem. their sizes are 26176 and 26180 bytes. I created these files using USCS tool wigToBigWig. The data values are float-point.

Help is greatly appreciated! Thanks much!

Jinghua

[dasmoth]

Dear Jinghua, sorry to hear you're having problems.

This kind of issue is relatively hard to debug without access to the files in question. However, I do have a suspicion. Biodalliance relies on being able to fetch specific byte ranges of files like bigBeds, using the HTTP Range header. What I think is happening is that your server is ignoring the Range header and returning the whole file -- which is triggering an exception because it's not the size we expect.

Are you able to tell me which web server you're using?

If you've got some experience with your browser's debugging tools, it might be worth taking a closer look at the network traffic between Biodalliance and your server. In particular, if you look at the first GET request which is made to a bigBed file (ignore any OPTIONS requests), Biodalliance will always request the first 512 bytes of the file. The response should have a Content-Range header indicating this:

If the Content-Range header is missing, you may need to either reconfigure or change web servers.

[jinghuage]

Dear Thomas,

You are absolutely right about this, I have been using a simple python server to test the embedding of biodalliance, so the range header is not supported there. After I configure and switch to my yosemite webserver apache 2.4.9, everything works great, I see partial content response (code 206) of 512 byte each time.

Then I begin to switch to my department web server so everybody can see it, I got some strange behavior on there. I see some GET request e.g. this one: http://avsl.cct.lsu.edu/files/example-browsers/icmBigWig-twist-hg19.bw?salt=8OAPH6hJ45Ps2grOsAzpip25Yv4 which returned code 206 but with wrong byte size such as 528

Then, the direct request : http://avsl.cct.lsu.edu/files/example-browsers/icmBigWig-twist-hg19.bw return error code net::ERR_CONTENT_DECODING_FAILED

A HEAD request to this file on the server returns: Date: Wed, 11 Feb 2015 15:28:25 GMT Content-Encoding: gzip Last-Modified: Tue, 10 Feb 2015 18:41:07 GMT Server: Apache ETag: "7158b1-6693-50ec03ad376c0" Vary: Accept-Encoding Content-Type: text/plain; charset=UTF-8 Connection: Keep-Alive Accept-Ranges: bytes Keep-Alive: timeout=2, max=1500 Content-Length: 1593

which seems quite wrong. I am kinda lost here, is it trying to do a gzip encoding/decoding thing? Do you have any insight on this issue? Thanks so much with any advice!

Jinghua

On Tue, Feb 10, 2015 at 2:47 PM, Thomas Down notifications@github.com wrote:

Dear Jinghua, sorry to hear you're having problems.

This kind of issue is relatively hard to debug without access to the files in question. However, I do have a suspicion. Biodalliance relies on being able to fetch specific byte ranges of files like bigBeds, using the HTTP Range header. What I think is happening is that your server is ignoring the Range header and returning the whole file -- which is triggering an exception because it's not the size we expect.

Are you able to tell me which web server you're using?

If you've got some experience with your browser's debugging tools, it might be worth taking a closer look at the network traffic between Biodalliance and your server. In particular, if you look at the first GET request which is made to a bigBed file (ignore any OPTIONS requests), Biodalliance will always request the first 512 bytes of the file. The response should have a Content-Range header indicating this:

[image: image] https://cloud.githubusercontent.com/assets/209047/6136090/d4b78154-b165-11e4-8777-be0e2453267f.png

If the Content-Range header is missing, you may need to either reconfigure or change web servers.

Reply to this email directly or view it on GitHub https://github.com/dasmoth/dalliance/issues/139#issuecomment-73780423.

[dasmoth]

Glad to hear you're making progress.

I think the problem here is the Content-Encoding: gzip header. Even though parts of bigwig files are encoded using the same compression algorithm as gzip, it's not a gzip file, and the web browser interpreting it as such will cause problems. Unfortunately, there's nothing a Javascript client can do to override this, so you'll need to investigate what is causing your server to emit this header and disable it.

If it's an Apache server, it's possible that mod_mime_magic is to blame. I've never seen this problem before with bigWigs or bigBeds, but have seen it a couple of times with BAMs. A similar problem (also with BAMs) is described here.

[jinghuage]

Hi Thomas,

Thanks so much for your reply! Now I understand this issue better, I will investigate it further.

Jinghua

P.S, the server does let me download the file correctly, with this message:

Resource interpreted as Document but transferred with MIME type application/octet-stream: " http://avsl.cct.lsu.edu/files/example-browsers/icmBigWig-roll-hg19.bw".

On Wed, Feb 11, 2015 at 3:09 PM, Thomas Down notifications@github.com wrote:

Glad to hear you're making progress.

I think the problem here is the Content-Encoding: gzip header. Even though parts of bigwig files are encoded using the same compression algorithm as gzip, it's not a gzip file, and the web browser interpreting it as such will cause problems. Unfortunately, there's nothing a Javascript client can do to override this, so you'll need to investigate what is causing your server to emit this header and disable it.

If it's an Apache server, it's possible that mod_mime_magic http://httpd.apache.org/docs/current/mod/mod_mime_magic.html is to blame. I've never seen this problem before with bigWigs or bigBeds, but have seen it a couple of times with BAMs. A similar problem (also with BAMs) is described here http://wiki.bioinformatics.ucdavis.edu/index.php/Bam_file_browser/web_server_problem .

Reply to this email directly or view it on GitHub https://github.com/dasmoth/dalliance/issues/139#issuecomment-73965242.

[jinghuage]

Hi Thomas,

I think I understand the error now so I thought to give you an update:

My department's web server has content-encoding: gzip enabled by the deflate module. When making a 'Range' header request to an Apache server which has enforced GZIP compression, it compress the entire file first then return the compressed data range according to the 'Range' header. Of course the partial file can't be de-compressed at the browser side. The correct behavior we'd like to have is for the 'Range' to be taken from the uncompressed data, then compressed and returned. But I can't find any solution that can enable this.

So I think for the "ranged" fetch to work, the web server has to remove its content-encoding module. Unfortunately it also disables the neat compression feature.

Jinghua

On Wed, Feb 11, 2015 at 3:46 PM, Jinghua Ge jinghuage@cct.lsu.edu wrote:

Hi Thomas,

Thanks so much for your reply! Now I understand this issue better, I will investigate it further.

Jinghua

P.S, the server does let me download the file correctly, with this message:

Resource interpreted as Document but transferred with MIME type application/octet-stream: " http://avsl.cct.lsu.edu/files/example-browsers/icmBigWig-roll-hg19.bw".

On Wed, Feb 11, 2015 at 3:09 PM, Thomas Down notifications@github.com wrote:

Glad to hear you're making progress.

I think the problem here is the Content-Encoding: gzip header. Even though parts of bigwig files are encoded using the same compression algorithm as gzip, it's not a gzip file, and the web browser interpreting it as such will cause problems. Unfortunately, there's nothing a Javascript client can do to override this, so you'll need to investigate what is causing your server to emit this header and disable it.

If it's an Apache server, it's possible that mod_mime_magic http://httpd.apache.org/docs/current/mod/mod_mime_magic.html is to blame. I've never seen this problem before with bigWigs or bigBeds, but have seen it a couple of times with BAMs. A similar problem (also with BAMs) is described here http://wiki.bioinformatics.ucdavis.edu/index.php/Bam_file_browser/web_server_problem .

Reply to this email directly or view it on GitHub https://github.com/dasmoth/dalliance/issues/139#issuecomment-73965242.

[dasmoth]

Depending on the exact details of the server, it may well be possible to make the compression more selective. For example, Apache's mod_deflate can certainly be configured to compress selectively based on MIME times and/or which directories the files are in.

[jinghuage]

Thanks! I think this is the solution to my problem.

Jinghua

On Sat, Feb 14, 2015 at 8:58 AM, Thomas Down notifications@github.com wrote:

Depending on the exact details of the server, it may well be possible to make the compression more selective. For example, Apache's mod_deflate http://httpd.apache.org/docs/2.2/mod/mod_deflate.html can certainly be configured to compress selectively based on MIME times and/or which directories the files are in.

Reply to this email directly or view it on GitHub https://github.com/dasmoth/dalliance/issues/139#issuecomment-74378113.