fsspec / filesystem_spec

A specification that python filesystems should adhere to.
BSD 3-Clause "New" or "Revised" License
1k stars 354 forks source link

Inconsistencies: `ls` may have `detail=True` or `detail=False` #1205

Open lhoestq opened 1 year ago

lhoestq commented 1 year ago

I saw the AbstractArchiveFileSystem switched to detail=True in https://github.com/fsspec/filesystem_spec/commit/0f3ecd8e629043646ab19b1a2b00d895f0553a81.

There are some inconstencies with other filesystems which have detail=False (e.g. the local filesystem).

with fsspec==2023.3.0

Currently:

>>> import fsspec
>>> url = "https://huggingface.co/datasets/lhoestq/test_zip_txt/resolve/main/atoz.zip"
>>> fs = fsspec.filesystem("zip", fo=url)
>>> fs.ls("")
[{'orig_filename': 'atoz/',
  'filename': 'atoz/',
  'date_time': (2021, 12, 8, 1, 15, 4),
  'compress_type': 0,
  '_compresslevel': None,
  'comment': b'',
  'extra': b'\n\x00 \x00\x00\x00\x00\x00\x01\x00\x18\x00\x99;\xf6\xa4\xc8\xeb\xd7\x01\x99;\xf6\xa4\xc8\xeb\xd7\x01i\x9b\x84\xa7e\xe7\xd7\x01',
  'create_system': 0,
  'create_version': 63,
  'extract_version': 20,
  'reserved': 0,
  'flag_bits': 0,
  'volume': 0,
  'internal_attr': 0,
  'external_attr': 16,
  'header_offset': 0,
  'CRC': 0,
  'compress_size': 0,
  'file_size': 0,
  '_raw_time': 2530,
  'name': 'atoz/',
  'size': 0,
  'type': 'directory'}]

Previously:

>>> fs.ls("")
['atoz/']

cc @martindurant

Question: should the local file filesystem implementation have detail=True as well ?

(And is there a changelog somewhere with communication about breaking changes ?)

related: https://github.com/fsspec/filesystem_spec/issues/899

martindurant commented 1 year ago

Actually, AbstractFileSystem (the base class) has detail=True, so it was an effort to be consistent with that. Other filesystems should migrate to detail=True.