manthey
commented
7 months ago DSA uses a variety of libraries to read images. Many of these libraries require file-like access to the images, so rather than fetching them directly from S3, we expose the data in girder in a FUSE file system and the image libraries read those files. Some of the image libraries (notably openslide) do a lot of open-seek-read-close on small fragments for files (especially on ndpi files) and these manifest as range requests.
Internally, the FUSE file system uses the python requests library to fetch from s3 with a get call with stream=True. I don't see any code where we explicitly end that call; we let the python garbage collector do whatever it does and trust the requests library to do the right thing. Perhaps this should be explicitly closed, but it isn't.
There is an option to enable diskcache on the mount (see https://github.com/DigitalSlideArchive/digital_slide_archive/blob/master/devops/dsa/docker-compose.yml#L26-L28); this will make 128kB range requests (I think) rather than really small range requests and cache the results so you end up with vastly fewer partial requests.
I would expect reading the first bytes of the file, even if the image library doesn't mean to read the whole thing, will appear as a full object request (since from a file system level it is an open without seek). This is dependent on the image library doing the reading as well as the file format.
I'm not sure how to proxy requests to one S3 server to another.
codybum
We are experimenting with the use of a locally deployed java-based (https://github.com/mindmill/ladon-s3-server) S3 server with DSA. We have made numerous updates to the base code, which now supports most boto3 (https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) and CyberDuck (https://cyberduck.io/) commands. We are testing with a single 800M Aperio SVS file "test.svs".
We are able to successfully add the local S3 server as an Assetstore without issue. Girder PUT and DELETE test complete successfully without error. Import of the test file is slow, but it completes successfully according to DSA. We are able to access the test file in a Collection and view the slide in HistomicsUI. While things mostly work, tile updates are very slow and sporadic.
On the S3 side there are no errors until slide import. In network captures we can see ~50 HTTP request to the S3 server during the single file import process. We see DSA request the list of objects in the bucket, then a complete object request for test.svs. The S3 server starts the transfer to DSA for the complete object request, we see the window size on the DSA size for the transfer +49k until ~10 packets, at which point it is reduced to 8192, and then in the next packet we see a TCP reset (RST,ACK) from DSA. A (RST,ACK) from DSA would typically indicate that the socket on the DSA side has been closed. We observe a write error on the S3 side that also throws an error saying the socket is already closed.
The packet capture can be seen here:
In subsequent DSA request the "Range: bytes=[start_byte-end_byte]" header is set, but as with the full object request after some number of packets we see the the TCP window size eventually decrease and we see a reset from the DSA side. The range/partial requests are not the same, the offset (start_byte) changes while the end_byte remains the same. All subsequent ranged request are toward the end of the test.svs file.
In attempts to debug this issue, I partially re-implemented the downloadFile function (https://girder.readthedocs.io/en/latest/_modules/girder/utility/s3_assetstore_adapter.html#S3AssetstoreAdapter.downloadFile) from Girder in my test code. I took the HTTP request from the import and re-ran the request, confirming that the MD5 hashes provided by the S3 server are identical to the local file. The transfers end with (FIN,ACK), as expected.
It is far more likely there is something wrong with the S3 implementation than with DSA, but I am having a hard time reproducing these issues outside of DSA.
A few questions to help me along: