FileExistsErrors when running the test suites.

[nbren12]

I cannot run the test suite on my laptop for some reason:

To reproduce:

pip install -r requirements.txt
pytest

Error message:

self = <tests.conftest.MockGCSFileSystem object at 0x7ff978207f70>, path = '/vcm-fv3config/data/initial_conditions/gfs_c12_example/v1.0', create_parents = True, kwargs = {}

    def mkdir(self, path, create_parents=True, **kwargs):
        path = self._strip_protocol(path)
        if path in self.store or path in self.pseudo_dirs:
>           raise FileExistsError
E           FileExistsError

../../../miniconda3/envs/fv3net/lib/python3.8/site-packages/fsspec/implementations/memory.py:99: FileExistsError

Maybe some fsspec caches are hitting us here? Do we need gcsfs specific tests? If there is no special handling of gcsfs in this package maybe we could just remove the mock.

[nbren12]

Ok. It is a version specific issue. It only happens if the constraints file is not used. Here is the version it fails on:

$ pip freeze | grep fsspec                                                                                                                                                                          (fv3config2)
fsspec==2022.3.0

[mcgibbon]

Sounds like the behavior of mkdir went from allow-existing to not. Also, it's using the in-memory cache, right? Hard to tell from what you posted, but it’s a MockGCS filesystem and the error is from implementations/memory.py. The test says “mock gcs” but I think really it’s just an fsspec compatibility test.

Ideally this can be fixed by passing some allow-existing flag or using a mkdirs function or something that allows existing directories.

If it's version-specific maybe we can ignore it, but this could reflect a change in the behavior of fsspec moving forward (meaning the test is giving us useful info here).

[nbren12]

We do use fv3config frequently in practice with the newer fsspec versions.

[mcgibbon]

Probably worth fixing the code for the failure then, unless we find this is actually a bug in fsspec (then we could fix it there).