danielfrg / s3contents

Jupyter Notebooks in S3 - Jupyter Contents Manager implementation
Apache License 2.0
248 stars 88 forks source link

glob pattern in filename is unexpectedly expanded #165

Open toslunar opened 1 year ago

toslunar commented 1 year ago

Because s3fs (fsspec) supports some glob patterns (*, **, ?, and some of []), there are file names that should not be passed to self.fs.cp or self.fs.rm without escape.

To reproduce

Example 1.

Example 2.

To fix the issue

For now, adding glob.escape doesn't fix the issue because fsspec does not interpret the pattern [?] correctly. https://github.com/fsspec/filesystem_spec/blob/2023.5.0/fsspec/asyn.py#L648

Instead, I configured the fs not to use glob by

class S3FileSystemNoGlob(s3fs.S3FileSystem):
    async def _expand_path(self, path, recursive=False, maxdepth=None):
        ...

and turning off the if-branch https://github.com/fsspec/filesystem_spec/blob/2023.5.0/fsspec/asyn.py#L763-L772.