HTTPFileSystem breaks when range requests are not supported

fsspec / filesystem_spec

A specification that python filesystems should adhere to.

BSD 3-Clause "New" or "Revised" License

949 stars 343 forks source link

HTTPFileSystem breaks when range requests are not supported #1626

Open dholth opened 3 weeks ago

dholth commented 3 weeks ago

from fsspec.core import url_to_fs

ufs, url = url_to_fs("https://example.org/")
f = ufs.open("https://example.org/no-range-requests/data.txt")
f.seek(8192)
f.read(1)

Will raise # ValueError: The HTTP server doesn't appear to support range requests.

I was having trouble with cache options, but it turns out I was sending simplecache:: to the wrong place; this does work, but I don't really want to keep the seekable file around after the file is closed. Trying to be careful to open the file only once.

martindurant commented 3 weeks ago

Specifically:

HTTPFile, which supports seeking, sends range requests and is refused by the server
HTTPStreamFile does not use range, but does not support seek().
HTTPStreamFile ought to support some caching modes, particularly "all" and "first", but is hard-coded to "none".

dholth commented 3 weeks ago

IIUC, for my use case I could

use simplecache:: or filecache::
enhance those so that the cache only keeps a file or two if I'm worried about disk space, like python TemporaryFile's delete-file-on-close
Edit HTTPStreamFile so that it can fetch a range, if that range is right after whatever we've fetched so far, and use the file-level "all" cache instead of the filesystem wrappers simplecache::

martindurant commented 2 weeks ago

cf https://github.com/fsspec/filesystem_spec/discussions/1629