Multipart response / byte range(s) decoding bug? `Cannot decode header in multipart message`

[ryuwd]

Hi there! I am attempting to use the JSROOT v7.7.4 HierarchyPainter in my NextJS app like this:

https://gitlab.cern.ch/lhcb-dpa/analysis-productions/LbAPWeb/-/blob/main/components/pipelines/JSROOTgui.jsx?ref_type=heads#L32

to render a ROOT file browser in our webapp. I am running into a error, where the openRootFile call returns false. When I open the file directly with openFile(url) from jsroot/io I get the following:

Error: Cannot decode header in multipart message

We recently switched from putting the ROOT files on S3 (where they could be read by JSRoot with a signed URL generated by S3) to reading the files from EOS routing the output through our API to sidestep CORS problems.

The requests appear to fetch ranges from the ROOT file and are marked successful in the browser console. I am also able to download the ROOT file normally with my browser by going directly to the link, and browse in a regular TBrowser.

I've managed to trace the problem to possibly how JSROOT is decoding the multipart response, i.e. where the content type is

content-type: multipart/byteranges; boundary=EOSMULTIPARTBOUNDARY

as I only seem to run into the issue when JSROOT requests multiple byte ranges rather than single ranges. When I set:

      settings.MaxRanges = 1;

the ROOT file does then start load in the browser and things do function as normal, although slower (which is understandable due to forcing 1 request per range).

I start to wonder now if it's because of how jsroot interprets the multipart boundary and multipart/byteranges content types? Or indeed it could be because EOS / xrootd is using EOSMULTIPARTBOUNDARY as a boundary rather than a lowercase random-ish alphanumeric string which seems to be more standard. The RFC's defining the boundaries for multipart responses doesn't seem to exclude this kind of boundary, though.

Any advice much appreciated. Let me know if I can help and provide more information.

[linev]

Hi,

I already saw problems with different http servers not correctly implemented multi-range requests. Most easy will be if I can debug your http server myself.

Or you can check if your server correctly implements such multi-range request as described here:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests

I do not think that usage of EOSMULTIPARTBOUNDARY identifier is a problem. But does your server adds -- before this identifier in message body?

There was issue with EOS - see https://github.com/root-project/root/issues/13018. May be you have also related problem?

[ryuwd]

Thanks for the speedy response! It's entirely possible that the XRootD http server has a nonstandard implementation of multipart responses - I'd assumed it was fine as my browser was able to download the file, but I realise now that probably wouldn't involve multirange requests.

But does your server adds -- before this identifier in message body?

Yes it does - access to our server via URL and eos token alone should be sufficient, but then I'm unsure about that because it's collaboration internal data. If I put a dummy/opendata file somewhere in our EOS area then we might be able to get around that. In the meantime I can provide the headers and response, which I hope can help in the meantime:

$ cat test_response.txt | base64 -d

--EOSMULTIPARTBOUNDARY
Content-Type: application/octet-stream
Content-Range: bytes 29161410-29165476/1762874522

<illegible binary output>
--EOSMULTIPARTBOUNDARY
Content-Type: application/octet-stream
Content-Range: bytes 1762872720-1762874096/1762874522

<illegible binary output>
--EOSMULTIPARTBOUNDARY--

response headers:

HTTP/1.1 206 Partial Content
date: Thu, 24 Oct 2024 HH:MM:SS
server: uvicorn
last-modified: Thu, 24 Oct 2024 HH:MM:SS
content-length: 5702
etag: "..snip.."
access-control-expose-headers: Accept-Ranges, Content-Type, Content-Length, Content-Range, Content-Encoding
accept-ranges: bytes
content-type: multipart/byteranges; boundary=EOSMULTIPARTBOUNDARY
access-control-allow-origin: https://lhcb-productions.web.cern.ch
vary: Origin
set-cookie: ...snip...
cache-control: private

request and headers

GET /eos_proxy/browse_artifact/a/b/c/d/e/f.g.root?xrd.wantprot=unix&authz=EOS_TOKEN_FOR_FILE HTTP/1.1
Host: lbap.app.cern.ch
User-Agent: Mozilla/5.0 ....snip....
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd, identity
Range: bytes=1762872720-1762874096,29161410-29165476
Origin: https://lhcb-productions.web.cern.ch
Connection: keep-alive
Referer: https://lhcb-productions.web.cern.ch/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
DNT: 1
Sec-GPC: 1
Pragma: no-cache
Cache-Control: no-cache

[linev]

Any test ROOT file - like hsimple.root - will be sufficient for me.

If you can explain how I can configure xrootd server myself - I will try to test it directly on my machine.

May be Content-Type: application/octet-stream is problem for JSROOT.

[chrisburr]

I'll contact your privately to provide a reproducer URL.

[linev]

@chrisburr

Can you provide me working link? I trying to understand why JSROOT not working with EOS. With plain xrootd I can read data - but only from node.js

[linev]

Problem was identified.

It is reordering of fragments in reply of EOS server. Seems to be it is allowed by RFC 7233, but was not handled by JSROOT.

Fix provided in master and 7.7 branches.

Thanks for reporting it.

[ryuwd]

Fantastic! Thank you @linev

[linev]

Fix published in https://github.com/root-project/jsroot/releases/tag/7.7.5

[chrisburr]

Thank you for dealing with this so quickly (as usual!)

I've updated our install of JSROOT and confirmed 7.7.5 fixes the issue.