aai-institute / lakefs-spec

An fsspec implementation for the lakeFS project.
http://lakefs-spec.org/
Apache License 2.0
37 stars 4 forks source link

`ls`: Make non-recursive by default, correctly update dircache #198

Closed AdrianoKF closed 7 months ago

AdrianoKF commented 7 months ago

This PR combines various fixes to the handling of the directory listing cache and ls behavior in general:

🚨 Breaking Change: Make ls non-recursive by default

The ls method in its previous implementation would always return items in subdirectories recursively, which is not an intuitive behavior.

This PR introduces an optional recursive kwarg, which is set to False by default.

ls dircache update

The ls command was not updating the dircache correctly, which could lead to incorrect results when listing the same directory multiple times.

In particular, the implementation would always add all results from the API call to the dircache for the directory, regardless of whether the results were already in the cache entry (essentially doubling the cache entry for each subsequent call).

This behavior would only be triggered by passing refresh=True, which bypasses the shortcut evaluation that returns a dircache entry directly.

rm dircache invalidation

Additionally, this PR fixes the invalidation of the dircache as part of rm.

Removing a file from the repository obviously makes any cache directory listing of its parent directory invalid, which is now handled correctly (and tested).

codecov[bot] commented 7 months ago

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (ee6d531) 88.04% compared to head (071fbea) 88.23%. Report is 1 commits behind head on main.

Files Patch % Lines
src/lakefs_spec/spec.py 86.66% 1 Missing and 3 partials :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #198 +/- ## ========================================== + Coverage 88.04% 88.23% +0.19% ========================================== Files 7 7 Lines 485 510 +25 Branches 70 79 +9 ========================================== + Hits 427 450 +23 Misses 31 31 - Partials 27 29 +2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.