jupyter notebooks have some API issues

[mbauer288]

I found a couple of probably out-of-date API examples/tests (search for ## NEEDS FIX?), but otherwise everything seems to work.

OS : Ubuntu 22.04 LTS Codename : jammy Kernel : 5.15.0-41-generic Architecture: x86_64

CPU : Intel i7-9700K @ 3.60GHz x 8 cores GPU : NVIDIA GeForce RTX 2080 SUPER RAM : 32 GB DDR4-2666 MHz (2 x 16 GB) HD: : 2.5 TB

Step 1: Install STAREPandas (https://github.com/SpatioTemporal/STAREPandas)

Step 1a: Clone STAREPandas repo

    $ cd 
    $ gh repo clone SpatioTemporal/STAREPandas
    $ cd STAREPandas

Step 1b: Install STAREPandas (be sure pyevn mystare being used).

    Step 1b1: Install some dependencies
        $ pip install sqlalchemy
        $ pip install dask
        $ pip install rasterio
        $ pip install importlib
        $ pip install ipython 
        $ pip install ipympl

        $ sudo apt -y install nodejs
        $ sudo apt -y install npm

        $ jupyter labextension install @jupyter-widgets/jupyterlab-manager
        $ jupyter labextension install jupyter-matplotlib
        $ jupyter nbextension enable --py widgetsnbextension

    $ pip install starepandas

    $ ls ./.pyenv/versions/3.10.5/envs/mystare/lib/python3.10/site-packages/starepandas
        datasets  __init__.py  io  staredataframe.py  stareseries.py  tools  _version.py

Step 1c: Tests
    $ cd STAREPandas 
    $ pytest
        ================================================= test session starts ==================================================
        platform linux -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
        rootdir: /home/mbauer/STAREPandas, configfile: pytest.ini
        plugins: anyio-3.6.1
        collected 92 items                                                                                                     

        starepandas/staredataframe.py ......................                                                             [ 23%]
        starepandas/datasets/__init__.py s                                                                               [ 25%]
        starepandas/io/pod.py .                                                                                          [ 26%]
        starepandas/io/granules/__init__.py ..                                                                           [ 28%]
        starepandas/tools/spatial_conversions.py ...........                                                             [ 40%]
        starepandas/tools/temporal_conversions.py .                                                                      [ 41%]
        starepandas/tools/trixel_conversions.py ..............                                                           [ 56%]
        tests/test_catalog.py .                                                                                          [ 57%]
        tests/test_dissolve.py .                                                                                         [ 58%]
        tests/test_enclaves.py .                                                                                         [ 59%]
        tests/test_geotiff.py .                                                                                          [ 60%]
        tests/test_guess_companion_path.py ......                                                                        [ 67%]
        tests/test_instantiation.py ..                                                                                   [ 69%]
        tests/test_intersects.py ...                                                                                     [ 72%]
        tests/test_parallel.py .......                                                                                   [ 80%]
        tests/test_plotting.py ...                                                                                       [ 83%]
        tests/test_readgranules.py .........                                                                             [ 93%]
        tests/test_shapely20.py .                                                                                        [ 94%]
        tests/test_trixels.py ..                                                                                         [ 96%]
        tests/test_type_persistance.py ...                                                                               [100%]

        =================================================== warnings summary ===================================================
        starepandas/io/folder.py:103
          /home/mbauer/STAREPandas/starepandas/io/folder.py:103: DeprecationWarning: invalid escape sequence '\.'         ## NEEDS FIX?
            granule_paths, s3 = starepandas.io.s3.s3_glob(path, '.*\.{ext}$'.format(ext=granule_extension))

        starepandas/io/folder.py:108
          /home/mbauer/STAREPandas/starepandas/io/folder.py:108: DeprecationWarning: invalid escape sequence '\.'         ## NEEDS FIX?
            pattern = '.*[^_stare]\.(nc|hdf|HDF5)'

        -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
        ====================================== 91 passed, 1 skipped, 2 warnings in 24.74s ======================================

    $ cd examples
    $ jupyter notebook
        catalog.ipynb            ## NEEDS FIX?
            ----> 9 n_america.set_trixels(inplace=True)
                TypeError: STAREDataFrame.set_trixels() missing 1 required positional argument: 'col'
        dask_df.ipynb          ## NEEDS FIX?
            ----> 1 client = dask.distributed.Client(n_workers=4)
                AttributeError: module 'dask' has no attribute 'distributed'
        dissolve.ipynb  
            Works           
        enclaves.ipynb           ## NEEDS FIX?
            ----> 3 rsa = starepandas.STAREDataFrame(rsa, add_stare=True, level=level, add_trixels=True)
                File ~/.pyenv/versions/3.10.5/envs/mystare/lib/python3.10/site-packages/starepandas/staredataframe.py:66, in STAREDataFrame.__init__(self, sids, add_sids, resolution, trixels, add_trixels, n_workers, *args, **kwargs)           
        geoTIFF.ipynb            ## NEEDS FIX?
            ----> 5 mask =
                SyntaxError: invalid syntax
        granules.ipynb  
            Works   
        indidivdual_geoms.ipynb          ## NEEDS FIX? 
            ----> 7 indices = starepandas.from_polygon(polygon, level=8, force_ccw=True)
                AttributeError: module 'starepandas' has no attribute 'from_polygon'            
        instantiation.ipynb          ## NEEDS FIX?    
            ---> 11 df.plot(ax=ax, column='City')   
                ValueError          
        intersection.ipynb          ## NEEDS FIX?
            ----> 7 stare = starepandas.stare_from_gdf(amazon, level=8, force_ccw=True)
                AttributeError: module 'starepandas' has no attribute 'stare_from_gdf'                 
        intersects.ipynb         ## NEEDS FIX?        
            ----> 8 fname = starepandas.datasets.get_path('MOD05_L2.A2019336.0000.061.2019336211522.hdf')
                File ~/.pyenv/versions/3.10.5/envs/mystare/lib/python3.10/site-packages/starepandas/datasets/__init__.py:30, in get_path(dataset)
                ValueError: The dataset 'MOD05_L2.A2019336.0000.061.2019336211522.hdf' is not available. Available datasets are             
        manual_join.ipynb         ## NEEDS FIX?        
             ---> 10 stare = starepandas.stare_from_xy(cities.Longitude, cities.Latitude, level=27)
            A   ttributeError: module 'starepandas' has no attribute 'stare_from_xy'
        parallel.ipynb          ## NEEDS FIX? 
             ---->  3621     return self._engine.get_loc(casted_key)
                KeyError: 'stare'                     
        plotting.ipynb            
            Works
        read_granules.ipynb          ## NEEDS FIX?
            ----> 2 vnp03dnb = starepandas.io.granules.VIIRSL2(path)
                AttributeError: module 'starepandas.io.granules' has no attribute 'VIIRSL2'                  
        shapely20.ipynb           
            Works
        sid_resolution.ipynb      
            Works
        spatial_relations.ipynb         ## NEEDS FIX?   
            ---> 10 stare = starepandas.stare_from_xy(cities.Longitude, cities.Latitude, level=27)
                AttributeError: module 'starepandas' has no attribute 'stare_from_xy'            
        split_antimeridian.ipynb
            Works
        stare_join.ipynb         ## NEEDS FIX?
             ---> 10 stare = starepandas.stare_from_xy(cities.Longitude, cities.Latitude, level=26)
            AttributeError: module 'starepandas' has no attribute 'stare_from_xy'           
        temporal.ipynb         ## NEEDS FIX?
            ----> 1 [datetime.datetime.utcfromtimestamp(m/1000) for m in ms]
                NameError: name 'ms' is not defined            
        to_database.ipynb         ## NEEDS FIX?
            ---> 10 stare = starepandas.stare_from_xy(cities.Longitude, cities.Latitude, level=27)
                AttributeError: module 'starepandas' has no attribute 'stare_from_xy'
        user_defined_granule_loader.ipynb         ## NEEDS FIX?
            ----> 1 from starepandas.io.granules import Modis
                ImportError: cannot import name 'Modis' from 'starepandas.io.granules' (/home/mbauer/.pyenv/versions/3.10.5/envs/mystare/lib/python3.10/site-packages/starepandas/io/granules/__init__.py)            
        wrapping.ipynb         ## NEEDS FIX?
            ----> 3 sdf.plot(ax=ax)
                AttributeError: No trixels set (expected in "trixels" column)

   # Generate documentation
        $ cd docs/
        $ make html
            Running Sphinx v5.1.1
            making output directory... done
            ...
            lots of warnings and errors
            ...
            done
            preparing documents... done
            writing output... [100%] readme/README                                                                                  
            generating indices... genindex done
            copying notebooks ... [100%] examples/notebooks/wrapping.ipynb                                                          
            writing additional pages... search done
            copying images... [100%] readme/figures/amazon.png                                                                      
            copying static files... done
            copying extra files... done
            dumping search index in English (code: en)... done
            dumping object inventory... done
            build succeeded, 12 warnings.

            The HTML pages are in build/html.

[NiklasPhabian]

This is great. Yes. plenty of notebooks out of date.

Dask[distributed], sqlalchemy, and rasterio actually have become hard (non-optional) dependencies of starepandas. We shall add them to the setup.cfg. I'd guess jupyter (related) things remain optional to execute the examples for now.

I will make an effort to have all notebooks being invoked when we call pytest which shall force us to fix all the notebooks. At this point, arguably jupyter will become a hard dependency as well. We need this for pystare as well. I will open a separate issue for this

[NiklasPhabian]

I fixed all notebooks and the STAREPandas dependencies. I also added the notebooks as tests through nbmake. They are run as GH actions, so we will always know when we break the API of our examples. One can also run them locally with:

pytest --nbmake --nbmake examples/*.ipynb