search

google / Xee

An Xarray extension for Google Earth Engine
Apache License 2.0
240 stars 28 forks source link

Error when Xee attempts to cast image data to an expected dtype #146

Open KMarkert opened 6 months ago

KMarkert commented 6 months ago

In some rare cases the data extraction process from EE to local ends up with an error: ValueError: When changing to a smaller dtype, its size must be a divisor of the size of original dtype. It is not clear to me why this occurs but obviously something is going on between the return dtype from EE and the expected dtype from xee.

This issue was introduced by #113.

More investigation is need to work on a reproducible example that demonstrates why this occurs and provide a fix.

naschmitz commented 6 months ago

How frequently do you see this issue? Do you think it warrants a rollback?

KMarkert commented 4 months ago

This seems like a pretty isolated error so no rollback necessary in my opinion.

I did some testing and it seems like this was happening for me when there was an image that was all masked was being requested in the same chunk as non-masked images. This can create a mismatch in data types (floats and ints in this case) then the float values from the mask could not be cast as ints.

I was using xee v0.09. Upgrading to latest version (v0.0.12 as time of posting) seems to have fixed the issue and get the data types correct.

But just in case others run into this issue, one fix I tried was to manually set the chunk size where each image in time is requested individually (e.g. ds.chunk({"time": 1, "lon": x_dim, "lat":y_dim})) before requesting the data locally. That way there isn't a mismatch of dytpes in the array that is sent from Earth Engine.