Website errors for Chapters 5-8

[shriv]

Hi there, I cannot see Chapters 5-8. I get the following error.

[michaeldorman]

Thanks!

I guess that there is an error in the code of Chapter 5 on GitHub actions. Locally it works fine for me.

Where can we see the errors log of the Python code in GitHub actions to figure out what's different there? @Robinlovelace , @Nowosad, @anitagraser

🙏

[Nowosad]

 OverflowError                             Traceback (most recent call last)
File /usr/local/lib/python3.11/site-packages/numpy/ma/core.py:489, in _check_fill_value(fill_value, ndtype)
    488 try:
--> 489     fill_value = np.asarray(fill_value, dtype=ndtype)
    490 except (OverflowError, ValueError) as e:
    491     # Raise TypeError instead of OverflowError or ValueError.
    492     # OverflowError is seldom used, and the real problem here is
    493     # that the passed fill_value is not compatible with the ndtype.

OverflowError: Python integer 9999 out of bounds for uint8

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Cell In[33], line 1
----> 1 out_image, out_transform = rasterio.mask.mask(
      2     src_nlcd, 
      3     zion.geometry.to_crs(src_nlcd.crs), 
      4     crop=False, 
      5     nodata=9999
      6 )
      7 counts = np.unique(out_image, return_counts=True)
      8 counts

[Nowosad]

@michaeldorman you can find this error by clicking on "Render" at https://github.com/geocompx/geocompy/actions/runs/10767927445/job/29856118003 and then scrolling down.

[Robinlovelace]

Probably a Python package version issue, although surprised it didn't result in Actions failing.

Also affects other the gh-pages workflow: https://github.com/geocompx/geocompy/actions/runs/10767927441/job/29856117885#step:4:413

[Robinlovelace]

Logical next step would be to test locally with the python image:

https://github.com/geocompx/geocompy/blob/4f124ea35efce93ffa3340f3ce49211a9678e043/.github/workflows/netlify.yml#L10

Also in our devcontainer:

https://github.com/geocompx/geocompy/blob/4f124ea35efce93ffa3340f3ce49211a9678e043/.devcontainer.json#L2

[Robinlovelace]

Testing in GitHub codespaces which uses the image listed above.

Anyone can do by clicking this link, I think:

https://github.com/codespaces/new?hide_repo_select=true&ref=main&repo=447558863

[Robinlovelace]

Failing command:

https://github.com/geocompx/geocompy/blob/4f124ea35efce93ffa3340f3ce49211a9678e043/05-raster-vector.qmd#L449

[Robinlovelace]

Can reproduce in devcontainer:

[Robinlovelace]

Removing

    nodata=9999

Fixes it. Will put in PR.

[Robinlovelace]

I think it's numpy2 related:

  Cell 76/78: ''................................Done
  Cell 77/78: ''................................ERROR: 

An error occurred while executing the following cell:
------------------
image = rasterio.features.rasterize(
    distances,
    out_shape=r.shape,
    dtype=np.float_,
    transform=new_transform,
    fill=np.nan
)
image
------------------

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[109], line 4
      1 image = rasterio.features.rasterize(
      2     distances,
      3     out_shape=r.shape,
----> 4     dtype=np.float_,
      5     transform=new_transform,
      6     fill=np.nan
      7 )
      8 image

File /usr/local/lib/python3.11/site-packages/numpy/__init__.py:397, in __getattr__(attr)
    394     raise AttributeError(__former_attrs__[attr])
    396 if attr in __expired_attributes__:
--> 397     raise AttributeError(
    398         f"`np.{attr}` was removed in the NumPy 2.0 release. "
    399         f"{__expired_attributes__[attr]}"
    400     )
    402 if attr == "chararray":
    403     warnings.warn(
    404         "`np.chararray` is deprecated and will be removed from "
    405         "the main namespace in the future. Use an array with a string "
    406         "or bytes dtype instead.", DeprecationWarning, stacklevel=2)

AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead.

Can you try installing NumPy 2.0 @michaeldorman ?

[Robinlovelace]

In the container we have 2.0.1:

pip install numpy
Requirement already satisfied: numpy in /usr/local/lib/python3.11/site-packages (2.0.1)

[Robinlovelace]

And that matches our requirements.txt:

https://github.com/geocompx/geocompy/blob/4f124ea35efce93ffa3340f3ce49211a9678e043/requirements.txt#L6

So I think we need to update the offending code. Thanks for the report @shriv and do let us know anyone ideas of fixes. I imagine there are lots of issues in the rasterio GitHub related to NumPy 2.

[Robinlovelace]

Yes, there are lots of issues: https://github.com/search?q=repo%3Arasterio%2Frasterio+numpy&type=issues

Notably this one by @sgillies: https://github.com/rasterio/rasterio/issues/2819

Should we upgrade to 1.4.0 when ready?

[Robinlovelace]

Currently we're on 1.3:

https://github.com/geocompx/geocompy/blob/4f124ea35efce93ffa3340f3ce49211a9678e043/requirements.txt#L11

[michaeldorman]

@michaeldorman you can find this error by clicking on "Render" at https://github.com/geocompx/geocompy/actions/runs/10767927445/job/29856118003 and then scrolling down.

Got it, thanks @Nowosad !

[Robinlovelace]

Any comments on this very welcome @sgillies and great to see 1.4.0rc1 is due out today!

Do you have an ETA for 1.4.0, without the rc (release candidate I think) bit?

[michaeldorman]

I think it's numpy2 related:

  Cell 76/78: ''................................Done
  Cell 77/78: ''................................ERROR: 

An error occurred while executing the following cell:
------------------
image = rasterio.features.rasterize(
    distances,
    out_shape=r.shape,
    dtype=np.float_,
    transform=new_transform,
    fill=np.nan
)
image
------------------

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[109], line 4
      1 image = rasterio.features.rasterize(
      2     distances,
      3     out_shape=r.shape,
----> 4     dtype=np.float_,
      5     transform=new_transform,
      6     fill=np.nan
      7 )
      8 image

File /usr/local/lib/python3.11/site-packages/numpy/__init__.py:397, in __getattr__(attr)
    394     raise AttributeError(__former_attrs__[attr])
    396 if attr in __expired_attributes__:
--> 397     raise AttributeError(
    398         f"`np.{attr}` was removed in the NumPy 2.0 release. "
    399         f"{__expired_attributes__[attr]}"
    400     )
    402 if attr == "chararray":
    403     warnings.warn(
    404         "`np.chararray` is deprecated and will be removed from "
    405         "the main namespace in the future. Use an array with a string "
    406         "or bytes dtype instead.", DeprecationWarning, stacklevel=2)

AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead.

Can you try installing NumPy 2.0 @michaeldorman ?

Thanks @Robinlovelace 🙏

[Robinlovelace]

Use `np.float64` instead

Could be good idea from above.

[michaeldorman]

Interesting... I tried installing numpy 2.1.1, but it conflicts with matplotlib. If I try to install matplotlib or both, then pip downgrades to numpy 1.26.4

[Robinlovelace]

You need to upgrade a load of pkgs I think Michael. The Docker container has the following, as you'd be able to find out with

docker run -it ghcr.io/geocompx/docker:python /bin/bash

pip install matplotlib
Requirement already satisfied: matplotlib in /usr/local/lib/python3.11/site-packages (3.9.0)

[michaeldorman]

Thanks @Robinlovelace , seems to be solved. I also needed pip install -U pyarrow numexpr bottleneck

[michaeldorman]

Solved, thanks @Nowosad and @Robinlovelace !

The issue was using 9999 instead of src_nlcd.nodata in this code section:

out_image, out_transform = rasterio.mask.mask(
    src_nlcd, 
    zion.geometry.to_crs(src_nlcd.crs), 
    crop=False, 
    nodata=src_nlcd.nodata
)

https://github.com/geocompx/geocompy/commit/b4c5fc180ecfeae57f6088a697f9db3ae7bdcebf