Context not preserved when open file with compression, leading exception to be ignored

[heiseish]

Context

• When providing the compression parameter to fsspec.open, the underlying Filesystem object is not properly cleaned up (even with context).
• This causes any errors related to the Filesystem object cleanup to be suppressed and show up during garbage collection instead.

Details

Reproducible code

import fsspec

def foo():
  with fsspec.open('s3://non-access-bucket/non-access-data.gz', mode = 'wb', compression = 'gzip') as f:
      f.write(b'hello world')

foo()

The above code snippet will happily run (assuming s3 credentials are setup properly etc) and function exits successfully, but later when the file object destructor get called, we will see the following exception

Exception

Exception ignored in: <funciton AbstractBufferredFile.__del__ at 0x7bc52f060040>
...
self.close
...
self.flush(force=True)
... snip ...
PermissionError: Access Denied

I have not tried with any other FS yet but I reckon as long as we can trigger the error during flush/close, we should be able to reproduce the error.

Temporary work around

• Separate fsspec open and compression library opener.

Relevant library code

            f = self._open(
                path,
                mode=mode,
                block_size=block_size,
                autocommit=ac,
                cache_options=cache_options,
                **kwargs,
            )
            if compression is not None:
                from fsspec.compression import compr
                from fsspec.core import get_compression

                compression = get_compression(path, compression)
                compress = compr[compression]
                f = compress(f, mode=mode[0]) # Here

[martindurant]

This sounds like a genuine bug. I know there are methods to chain contexts, but probably there is a simple fix available by patching the close method of the compression file-like object.

This kind of thing doesn't show up in tests because:

• this is windows specific behaviour (linux/bsd is happy to delete files that are open elsewhere)
• the exception doesn't actually cause any failures, as you have seen. So we should like to be able to make a test case, perhaps with an explicit del which exposes it.