pangeo-data / rechunker

Disk-to-disk chunk transformation for chunked arrays.
https://rechunker.readthedocs.io/
MIT License
163 stars 25 forks source link

check for nullness #64

Closed ocefpaf closed 3 years ago

ocefpaf commented 3 years ago

This should fix #59. It was failing b/c a fsspec.get_mapper() evaluates to False. I don't know enough about fsspec but I guess that makes sense if a directory is not created yet. Regardless we should always check for nullness instead of if var: do it in those cases.

codecov[bot] commented 3 years ago

Codecov Report

Merging #64 (6ca9ecf) into master (17053cc) will not change coverage. The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master      #64   +/-   ##
=======================================
  Coverage   97.75%   97.75%           
=======================================
  Files          10       10           
  Lines         445      445           
  Branches       88       88           
=======================================
  Hits          435      435           
  Misses          5        5           
  Partials        5        5           
Impacted Files Coverage Δ
rechunker/api.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 17053cc...6ca9ecf. Read the comment docs.

rabernat commented 3 years ago

I'm going to cc @martindurant on this. It seems unexpected that fsspec.get_mapper() evaluates to False. Martin, is this expected behavior or an fsspec bug?

EDIT: THANK YOU @ocefpaf for this PR! Depending on Martin's reply we will either fix upstream or more forward with this.

martindurant commented 3 years ago

Since it derives from MutableMapping, I think that bool(mapper) is essentially a test for emptiness, like bool(dict). The behaviour is not explicitly defined in fsspec.

rabernat commented 3 years ago

Ok great, so let's go forward with this.

@ocefpaf - clearly the situation in #59 was not covered by our tests. Could you manage to add a simple test for this scenario? I believe the right combo is

rabernat commented 3 years ago

I would add an additional parameterization here:

https://github.com/pangeo-data/rechunker/blob/17053cc2956ab8d417a666cc8675bcb4df1c5c56/tests/test_rechunk.py#L45-L54

...which parameterizes the store arguments (rather than setting them within the test function).

rabernat commented 3 years ago

Thanks Felipe! This is a fine solution.