astronomy-commons / hipscat

Hierarchical Progressive Survey Catalog
https://hipscat.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15 stars 3 forks source link

Support more fsspec filesystems #307

Open hombit opened 1 month ago

hombit commented 1 month ago

Bug report

We currently support only few filesystems, while community have very different setups for network filesystems, which may include HTTP, FTP, Google Cloud Block Storage, SSHFS, etc. The interface we have do not support any user-defined fsspec filesystems, I propose here to support them.

This is related and complimentary to #301

Before submitting Please check the following:

hombit commented 1 month ago

This is how you can use google drive filesystem

Install the package, it is not on PyPI (there is a package with the same name there, but it is not that one)

pip install git+https://github.com/fsspec/gdrivefs

Create a fs and read some files

import gdrivefs
import pyarrow

# Use token='browser' first time to create a token and store it locally, then use token='cache' 
gdfs = gdrivefs.GoogleDriveFileSystem(token='cache', root_file_id='1mocyakfy_8OgFGOIQ813S7POqwdDtfX_')
with gdfs.open('Norder=6/Dir=20000/Npix=28672.parquet') as f:
    metadata = pq.read_metadata(f)
print(metadata)