fastice / GrIMPTools

Information on GrIMP tools with links to other repositories
20 stars 3 forks source link

AttributeError around hvplot.xarray (grimpfunc) within jupyter lab #1

Open tbartholomaus opened 1 year ago

tbartholomaus commented 1 year ago

Hi Ian,

I'm somewhat loath to submit another issue here after my first, but now that I've got everything up and running on a machine at home (macbook pro 2016 intel chip) and my linux machine at work (CentOS 7), I really think this might be an actual bug.

I've installed the same greenlandMapping environment on my 2021 M1 Pro macbook pro, and proceeded without issue. At a python console, I have no problem importing grimpfunc:

(greenlandMapping) tbartholomaus@GEO-D618311 ~ % python
Python 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:55:37)
[Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import grimpfunc
>>>

However, when I do the same within jupyter lab, I get a fairly long set of errors, despite having run python -m ipykernel install --user --name=greenlandMapping several times, both with and without the greenlandMapping environment activated.

Within jupyter lab, with the greenlandMapping environment active and sitting idle (I can, for example, successfully import dask, geopandas, rioxarray, etc.), when I try and import grimpfunc I get an error with the boxPicker submodule that stems from the internal call to import hvplot.xarray

I'm copying this error below, and also attaching two screenshots, if they offer any more context.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [1], line 1
----> 1 import grimpfunc

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/grimpfunc/__init__.py:4
      1 __all__ = ['boxPicker', 'cmrUrls', 'Flowlines', 'get_urls', 'GrIMPSubsetter',
      2            'NASALogin', 'pointInspector']
----> 4 from grimpfunc.boxPicker import boxPicker
      5 from grimpfunc.cmrUrls import cmrUrls
      6 from grimpfunc.cmr import get_urls

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/grimpfunc/boxPicker.py:11
      9 import panel as pn
     10 import holoviews as hv
---> 11 import hvplot.xarray
     12 import grimpfunc as grimp
     13 import rioxarray

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/xarray.py:61
     57     xr.register_dataarray_accessor(interactive)(XArrayInteractive)
     59     post_patch(extension, logo)
---> 61 patch()

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/xarray.py:59, in patch(name, interactive, extension, logo)
     56 xr.register_dataset_accessor(interactive)(XArrayInteractive)
     57 xr.register_dataarray_accessor(interactive)(XArrayInteractive)
---> 59 post_patch(extension, logo)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/__init__.py:166, in post_patch(extension, logo)
    164 def post_patch(extension='bokeh', logo=False):
    165     if extension and not getattr(_hv.extension, '_loaded', False):
--> 166         hvplot_extension(extension, logo=logo)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/pyviz_comms/__init__.py:64, in extension.__new__(cls, *args, **kwargs)
     62 except Exception:
     63     pass
---> 64 return param.ParameterizedFunction.__new__(cls, *args, **kwargs)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/param/parameterized.py:3654, in ParameterizedFunction.__new__(class_, *args, **params)
   3652 inst = class_.instance()
   3653 inst.param._set_name(class_.__name__)
-> 3654 return inst.__call__(*args,**params)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/utilities.py:45, in hvplot_extension.__call__(self, *args, **params)
     40 def __call__(self, *args, **params):
     41     # importing e.g. hvplot.pandas always loads the bokeh extension.
     42     # so hvplot.extension('matplotlib', compatibility='bokeh') doesn't
     43     # require the user or the code to explicitely load bokeh.
     44     compatibility = params.pop('compatibility', None)
---> 45     super().__call__(*args, **params)
     46     backend = _hv.Store.current_backend
     47     if compatibility in ['matplotlib', 'plotly'] and backend != compatibility:

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/holoviews/ipython/__init__.py:149, in notebook_extension.__call__(self, *args, **params)
    147     completer.completions_sorting_key = self.completions_sorting_key
    148 if not p.allow_jedi_completion and hasattr(IPCompleter, 'use_jedi'):
--> 149     ip.run_line_magic('config', 'IPCompleter.use_jedi = False')
    151 resources = self._get_resources(args, params)
    153 Store.display_formats = p.display_formats

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/IPython/core/interactiveshell.py:2364, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2362     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2363 with self.builtin_trap:
-> 2364     result = fn(*args, **kwargs)
   2365 return result

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/IPython/core/magics/config.py:163, in ConfigMagics.config(self, s)
    159 from traitlets.config.loader import Config
    160 # some IPython objects are Configurable, but do not yet have
    161 # any configurable traits.  Exclude them from the effects of
    162 # this magic, as their presence is just noise:
--> 163 configurables = sorted(set([ c for c in self.shell.configurables
    164                              if c.__class__.class_traits(config=True)
    165                              ]), key=lambda x: x.__class__.__name__)
    166 classnames = [ c.__class__.__name__ for c in configurables ]
    168 line = s.strip()

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/IPython/core/magics/config.py:164, in <listcomp>(.0)
    159 from traitlets.config.loader import Config
    160 # some IPython objects are Configurable, but do not yet have
    161 # any configurable traits.  Exclude them from the effects of
    162 # this magic, as their presence is just noise:
    163 configurables = sorted(set([ c for c in self.shell.configurables
--> 164                              if c.__class__.class_traits(config=True)
    165                              ]), key=lambda x: x.__class__.__name__)
    166 classnames = [ c.__class__.__name__ for c in configurables ]
    168 line = s.strip()

AttributeError: type object 'CommManager' has no attribute 'class_traits'

As I mentioned at the top, I'm up and running on other systems, so this isn't urgent, but I thought I'd mention as a potential issue in case this comes up for anyone else, or if this is actually easily addressed or meaningful to you. Without debugging the call to import hvplot.xarray, specifically within a jupyter lab kernel, I'm not sure this is something I can get to the bottom of without a lot of effort.

Thanks again.

Screen Shot 2022-11-28 at 5 12 15 PM Screen Shot 2022-11-28 at 5 18 21 PM

fastice commented 1 year ago

I am out for the evening but will look at this tomorrow- remind me if you don’t hear by tomorrow pm. IanIan JoughinPolar Science CenterApplied Physics LabU Washington206-221-3177On Nov 28, 2022, at 5:22 PM, Timothy Bartholomaus @.> wrote: Hi Ian, I'm somewhat loath to submit another issue here after my first, but now that I've got everything up and running on a machine at home (macbook pro 2016 intel chip) and my linux machine at work (CentOS 7), I really think this might be an actual bug. I've installed the same greenlandMapping environment on my 2021 M1 Pro macbook pro, and proceeded without issue. At a python console, I have no problem importing grimpfunc: (greenlandMapping) @. ~ % python Python 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:55:37) [Clang 14.0.6 ] on darwin Type "help", "copyright", "credits" or "license" for more information.

import grimpfunc

However, when I do the same within jupyter lab, I get a fairly long set of errors, despite having run python -m ipykernel install --user --name=greenlandMapping several times, both with and without the greenlandMapping environment activated. Within jupyter lab, with the greenlandMapping environment active and sitting idle (I can, for example, successfully import dask, geopandas, rioxarray, etc.), when I try and import grimpfunc I get an error with the boxPicker submodule that stems from the internal call to import hvplot.xarray I'm copying this error below, and also attaching two screenshots, if they offer any more context.

AttributeError Traceback (most recent call last) Cell In [1], line 1 ----> 1 import grimpfunc

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/grimpfunc/init.py:4 1 all = ['boxPicker', 'cmrUrls', 'Flowlines', 'get_urls', 'GrIMPSubsetter', 2 'NASALogin', 'pointInspector'] ----> 4 from grimpfunc.boxPicker import boxPicker 5 from grimpfunc.cmrUrls import cmrUrls 6 from grimpfunc.cmr import get_urls

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/grimpfunc/boxPicker.py:11 9 import panel as pn 10 import holoviews as hv ---> 11 import hvplot.xarray 12 import grimpfunc as grimp 13 import rioxarray

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/xarray.py:61 57 xr.register_dataarray_accessor(interactive)(XArrayInteractive) 59 post_patch(extension, logo) ---> 61 patch()

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/xarray.py:59, in patch(name, interactive, extension, logo) 56 xr.register_dataset_accessor(interactive)(XArrayInteractive) 57 xr.register_dataarray_accessor(interactive)(XArrayInteractive) ---> 59 post_patch(extension, logo)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/init.py:166, in post_patch(extension, logo) 164 def post_patch(extension='bokeh', logo=False): 165 if extension and not getattr(_hv.extension, '_loaded', False): --> 166 hvplot_extension(extension, logo=logo)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/pyviz_comms/init.py:64, in extension.new(cls, *args, *kwargs) 62 except Exception: 63 pass ---> 64 return param.ParameterizedFunction.new(cls, args, **kwargs)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/param/parameterized.py:3654, in ParameterizedFunction.new(class, *args, **params) 3652 inst = class.instance() 3653 inst.param._setname(class.name) -> 3654 return inst.call(*args,**params)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/utilities.py:45, in hvplot_extension.call(self, *args, params) 40 def call(self, *args, *params): 41 # importing e.g. hvplot.pandas always loads the bokeh extension. 42 # so hvplot.extension('matplotlib', compatibility='bokeh') doesn't 43 # require the user or the code to explicitely load bokeh. 44 compatibility = params.pop('compatibility', None) ---> 45 super().call(args, params) 46 backend = _hv.Store.current_backend 47 if compatibility in ['matplotlib', 'plotly'] and backend != compatibility:

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/holoviews/ipython/init.py:149, in notebook_extension.call(self, *args, **params) 147 completer.completions_sorting_key = self.completions_sorting_key 148 if not p.allow_jedi_completion and hasattr(IPCompleter, 'use_jedi'): --> 149 ip.run_line_magic('config', 'IPCompleter.use_jedi = False') 151 resources = self._get_resources(args, params) 153 Store.display_formats = p.display_formats

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/IPython/core/interactiveshell.py:2364, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth) 2362 kwargs['local_ns'] = self.get_local_scope(stack_depth) 2363 with self.builtin_trap: -> 2364 result = fn(*args, **kwargs) 2365 return result

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/IPython/core/magics/config.py:163, in ConfigMagics.config(self, s) 159 from traitlets.config.loader import Config 160 # some IPython objects are Configurable, but do not yet have 161 # any configurable traits. Exclude them from the effects of 162 # this magic, as their presence is just noise: --> 163 configurables = sorted(set([ c for c in self.shell.configurables 164 if c.class.class_traits(config=True) 165 ]), key=lambda x: x.class.name) 166 classnames = [ c.class.name for c in configurables ] 168 line = s.strip()

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/IPython/core/magics/config.py:164, in (.0) 159 from traitlets.config.loader import Config 160 # some IPython objects are Configurable, but do not yet have 161 # any configurable traits. Exclude them from the effects of 162 # this magic, as their presence is just noise: 163 configurables = sorted(set([ c for c in self.shell.configurables --> 164 if c.class.class_traits(config=True) 165 ]), key=lambda x: x.class.name) 166 classnames = [ c.class.name for c in configurables ] 168 line = s.strip()

AttributeError: type object 'CommManager' has no attribute 'class_traits'

As I mentioned at the top, I'm up and running on other systems, so this isn't urgent, but I thought I'd mention as a potential issue in case this comes up for anyone else, or if this is actually easily addressed or meaningful to you. Without debugging the call to import hvplot.xarray, specifically within a jupyter lab kernel, I'm not sure this is something I can get to the bottom of without a lot of effort. Thanks again.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

fastice commented 1 year ago

Tim,

Hvplot was recently upgraded, so I expect that’s where the problem was. On my M1 Mac, it worked fine, but it was one version behind. I was able to reproduce the problem on a 2016 intel Mac Pro with a fresh environment.

I upgraded the environment to Python 10 and unpinned a couple of other programs, which all now seem to work. I have tested the new environment on both intel and M1 Macs and it seems to work fine (waiting for binder to load to finish that test).

BTW, in rebuilding the environments I ran the ipykernel command from the base env. a couple of times and it didn’t take, so my guess is you did run the command earlier but in the wrong environment.

So pull the new environment.yml file and conda env remove -n greenlandMapping then repeat the create procedure with the the new environment file.

It would be great if you could also do this on your linux machine too (an extra test of everything wouldn’t hurt).

Please let me know if you have any more problems.

Ian

Ian Joughin Polar Science Center Applied Physics Lab University of Washington 1013 NE 40th Street Seattle, WA 98105-6698

Data products: https://nsidc.org/grimp

Data Tools: https://github.com/fastice/GrIMPTools

Phone: 206-221-3177 Fax: 206-616-3142

On Nov 28, 2022, at 5:22 PM, Timothy Bartholomaus @.***> wrote:

Hi Ian,

I'm somewhat loath to submit another issue here after my first, but now that I've got everything up and running on a machine at home (macbook pro 2016 intel chip) and my linux machine at work (CentOS 7), I really think this might be an actual bug.

I've installed the same greenlandMapping environment on my 2021 M1 Pro macbook pro, and proceeded without issue. At a python console, I have no problem importing grimpfunc:

(greenlandMapping) @.*** ~ % python Python 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:55:37) [Clang 14.0.6 ] on darwin Type "help", "copyright", "credits" or "license" for more information.

import grimpfunc

However, when I do the same within jupyter lab, I get a fairly long set of errors, despite having run python -m ipykernel install --user --name=greenlandMapping several times, both with and without the greenlandMapping environment activated.

Within jupyter lab, with the greenlandMapping environment active and sitting idle (I can, for example, successfully import dask, geopandas, rioxarray, etc.), when I try and import grimpfunc I get an error with the boxPicker submodule that stems from the internal call to import hvplot.xarray

I'm copying this error below, and also attaching two screenshots, if they offer any more context.


AttributeError Traceback (most recent call last) Cell In [1], line 1 ----> 1 import grimpfunc

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/grimpfunc/init.py:4 1 all = ['boxPicker', 'cmrUrls', 'Flowlines', 'get_urls', 'GrIMPSubsetter', 2 'NASALogin', 'pointInspector'] ----> 4 from grimpfunc.boxPicker import boxPicker 5 from grimpfunc.cmrUrls import cmrUrls 6 from grimpfunc.cmr import get_urls

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/grimpfunc/boxPicker.py:11 9 import panel as pn 10 import holoviews as hv ---> 11 import hvplot.xarray 12 import grimpfunc as grimp 13 import rioxarray

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/xarray.py:61 57 xr.register_dataarray_accessor(interactive)(XArrayInteractive) 59 post_patch(extension, logo) ---> 61 patch()

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/xarray.py:59, in patch(name, interactive, extension, logo) 56 xr.register_dataset_accessor(interactive)(XArrayInteractive) 57 xr.register_dataarray_accessor(interactive)(XArrayInteractive) ---> 59 post_patch(extension, logo)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/init.py:166, in post_patch(extension, logo) 164 def post_patch(extension='bokeh', logo=False): 165 if extension and not getattr(_hv.extension, '_loaded', False): --> 166 hvplot_extension(extension, logo=logo)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/pyviz_comms/init.py:64, in extension.new(cls, *args, *kwargs) 62 except Exception: 63 pass ---> 64 return param.ParameterizedFunction.new(cls, args, **kwargs)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/param/parameterized.py:3654, in ParameterizedFunction.new(class, *args, **params) 3652 inst = class.instance() 3653 inst.param._setname(class.name) -> 3654 return inst.call(*args,**params)

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/hvplot/utilities.py:45, in hvplot_extension.call(self, *args, params) 40 def call(self, *args, *params): 41 # importing e.g. hvplot.pandas always loads the bokeh extension. 42 # so hvplot.extension('matplotlib', compatibility='bokeh') doesn't 43 # require the user or the code to explicitely load bokeh. 44 compatibility = params.pop('compatibility', None) ---> 45 super().call(args, params) 46 backend = _hv.Store.current_backend 47 if compatibility in ['matplotlib', 'plotly'] and backend != compatibility:

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/holoviews/ipython/init.py:149, in notebook_extension.call(self, *args, **params) 147 completer.completions_sorting_key = self.completions_sorting_key 148 if not p.allow_jedi_completion and hasattr(IPCompleter, 'use_jedi'): --> 149 ip.run_line_magic('config', 'IPCompleter.use_jedi = False') 151 resources = self._get_resources(args, params) 153 Store.display_formats = p.display_formats

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/IPython/core/interactiveshell.py:2364, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth) 2362 kwargs['local_ns'] = self.get_local_scope(stack_depth) 2363 with self.builtin_trap: -> 2364 result = fn(*args, **kwargs) 2365 return result

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/IPython/core/magics/config.py:163, in ConfigMagics.config(self, s) 159 from traitlets.config.loader import Config 160 # some IPython objects are Configurable, but do not yet have 161 # any configurable traits. Exclude them from the effects of 162 # this magic, as their presence is just noise: --> 163 configurables = sorted(set([ c for c in self.shell.configurables 164 if c.class.class_traits(config=True) 165 ]), key=lambda x: x.class.name) 166 classnames = [ c.class.name for c in configurables ] 168 line = s.strip()

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.9/site-packages/IPython/core/magics/config.py:164, in (.0) 159 from traitlets.config.loader import Config 160 # some IPython objects are Configurable, but do not yet have 161 # any configurable traits. Exclude them from the effects of 162 # this magic, as their presence is just noise: 163 configurables = sorted(set([ c for c in self.shell.configurables --> 164 if c.class.class_traits(config=True) 165 ]), key=lambda x: x.class.name) 166 classnames = [ c.class.name for c in configurables ] 168 line = s.strip()

AttributeError: type object 'CommManager' has no attribute 'class_traits' As I mentioned at the top, I'm up and running on other systems, so this isn't urgent, but I thought I'd mention as a potential issue in case this comes up for anyone else, or if this is actually easily addressed or meaningful to you. Without debugging the call to import hvplot.xarray, specifically within a jupyter lab kernel, I'm not sure this is something I can get to the bottom of without a lot of effort.

Thanks again.

https://urldefense.com/v3/__https://user-images.githubusercontent.com/24902730/204414064-a9a3d02d-c251-4dea-b2f5-51331afa2815.png__;!!K-Hz7m0Vt54!lt2OHJIhHErILy-qp8uyDVopdkGR_HBVh7Rp6sH4CZad9HXJjOw81huWzDlywKNGdnKJRz5UzYydAiiHe7oF$ https://urldefense.com/v3/__https://user-images.githubusercontent.com/24902730/204414792-b762ca04-1a7c-4998-b3a0-9736fe2b5fde.png__;!!K-Hz7m0Vt54!lt2OHJIhHErILy-qp8uyDVopdkGR_HBVh7Rp6sH4CZad9HXJjOw81huWzDlywKNGdnKJRz5UzYydAiPjNFIx$ — Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/fastice/GrIMPTools/issues/1__;!!K-Hz7m0Vt54!lt2OHJIhHErILy-qp8uyDVopdkGR_HBVh7Rp6sH4CZad9HXJjOw81huWzDlywKNGdnKJRz5UzYydApqa6NvF$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALICGTRIDRQIG53LHSJYD6LWKVLGZANCNFSM6AAAAAASN53B2A__;!!K-Hz7m0Vt54!lt2OHJIhHErILy-qp8uyDVopdkGR_HBVh7Rp6sH4CZad9HXJjOw81huWzDlywKNGdnKJRz5UzYydAjLypoIa$. You are receiving this because you are subscribed to this thread.

tbartholomaus commented 1 year ago

Thanks for jumping right back into this, Ian. Importing grimpfunc in both M1 and linux machines, via jupyter lab all works great. But it looks like unpinning some of those packages might have led to other problems for the mac- looks like a gdal problem.

In a cell from your workbook that had previously worked fine

myTerminusUrls = grimp.cmrUrls(mode='terminus')  # mode image restricts search to the image products
myTerminusUrls.initialSearch();

myTermini = {}
for url in myTerminusUrls.getURLS():
    year = os.path.basename(url).split('_')[1]  # Extract year from name
    myTermini[year] = gpd.read_file(f'/vsicurl/&url={url}')  # Add terminus to data frame
    print(f'/vsicurl/&url={url}')

I now get the following error on the gpd.read_file from url. But! only on mac. On my linux this runs just fine. Here's the complete error message:

---------------------------------------------------------------------------
CPLE_OpenFailedError                      Traceback (most recent call last)
File fiona/_shim.pyx:83, in fiona._shim.gdal_open_vector()

File fiona/_err.pyx:291, in fiona._err.exc_wrap_pointer()

CPLE_OpenFailedError: '/vsicurl/&url=https://n5eil01u.ecs.nsidc.org/DP4/MEASURES/NSIDC-0642.002/1999.10.13/termini_1999_2000_v02.0.shp' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

DriverError                               Traceback (most recent call last)
Cell In [18], line 4
      2 for url in myTerminusUrls.getURLS():
      3     year = os.path.basename(url).split('_')[1]  # Extract year from name
----> 4     myTermini[year] = gpd.read_file(f'/vsicurl/&url={url}')  # Add terminus to data frame
      5     print(f'/vsicurl/&url={url}')

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.10/site-packages/geopandas/io/file.py:259, in _read_file(filename, bbox, mask, rows, engine, **kwargs)
    256     path_or_bytes = filename
    258 if engine == "fiona":
--> 259     return _read_file_fiona(
    260         path_or_bytes, from_bytes, bbox=bbox, mask=mask, rows=rows, **kwargs
    261     )
    262 elif engine == "pyogrio":
    263     return _read_file_pyogrio(
    264         path_or_bytes, bbox=bbox, mask=mask, rows=rows, **kwargs
    265     )

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.10/site-packages/geopandas/io/file.py:303, in _read_file_fiona(path_or_bytes, from_bytes, bbox, mask, rows, where, **kwargs)
    300     reader = fiona.open
    302 with fiona_env():
--> 303     with reader(path_or_bytes, **kwargs) as features:
    304         crs = features.crs_wkt
    305         # attempt to get EPSG code

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.10/site-packages/fiona/env.py:408, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwargs)
    405 @wraps(f)
    406 def wrapper(*args, **kwargs):
    407     if local._env:
--> 408         return f(*args, **kwargs)
    409     else:
    410         if isinstance(args[0], str):

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.10/site-packages/fiona/__init__.py:264, in open(fp, mode, driver, schema, crs, encoding, layer, vfs, enabled_drivers, crs_wkt, **kwargs)
    261     path = parse_path(fp)
    263 if mode in ('a', 'r'):
--> 264     c = Collection(path, mode, driver=driver, encoding=encoding,
    265                    layer=layer, enabled_drivers=enabled_drivers, **kwargs)
    266 elif mode == 'w':
    267     if schema:
    268         # Make an ordered dict of schema properties.

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.10/site-packages/fiona/collection.py:162, in Collection.__init__(self, path, mode, driver, schema, crs, encoding, layer, vsi, archive, enabled_drivers, crs_wkt, ignore_fields, ignore_geometry, **kwargs)
    160 if self.mode == 'r':
    161     self.session = Session()
--> 162     self.session.start(self, **kwargs)
    163 elif self.mode in ('a', 'w'):
    164     self.session = WritingSession()

File fiona/ogrext.pyx:540, in fiona.ogrext.Session.start()

File fiona/_shim.pyx:90, in fiona._shim.gdal_open_vector()

DriverError: '/vsicurl/&url=https://n5eil01u.ecs.nsidc.org/DP4/MEASURES/NSIDC-0642.002/1999.10.13/termini_1999_2000_v02.0.shp' not recognized as a supported file format.

Again, this is only a problem on mac (2021 M1 MBP), not on linux, and it looks like, from a comment in the former version of environment.yml that this might be a known issue?

I'm copying below a section of conda list from "identical" environments on both machines.

Mac (2021 M1) Linux
``` fiona 1.8.22 py310h5a14c0a_0 conda-forge flit-core 3.8.0 pyhd8ed1ab_0 conda-forge folium 0.13.0 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.1 h5bb23bf_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.38.0 py310h90acd4f_1 conda-forge freetype 2.12.1 h3f81eb7_1 conda-forge freexl 1.0.6 hb7f2c08_1 conda-forge frozenlist 1.3.3 py310h90acd4f_0 conda-forge fsspec 2022.11.0 pyhd8ed1ab_0 conda-forge future 0.18.2 pyhd8ed1ab_6 conda-forge gdal 3.5.2 py310he22d2f2_3 conda-forge geopandas 0.12.1 pyhd8ed1ab_1 conda-forge geopandas-base 0.12.1 pyha770c72_1 conda-forge geos 3.11.0 hb486fe8_0 conda-forge geotiff 1.7.1 ha1a2aeb_3 conda-forge geoviews 1.9.5 pyhd8ed1ab_0 conda-forge geoviews-core 1.9.5 pyha770c72_0 conda-forge gettext 0.21.1 h8a4c099_0 conda-forge giflib 5.2.1 hbcb3906_2 conda-forge glib 2.74.1 hbc0c0cd_1 conda-forge glib-tools 2.74.1 hbc0c0cd_1 conda-forge grimpfunc 0.0.4 pypi_0 pypi grimpqgis 0.0.1 pypi_0 pypi ``` ``` fiona 1.8.22 pypi_0 pypi flit-core 3.8.0 pyhd8ed1ab_0 conda-forge folium 0.13.0 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.1 hc2a2eb6_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.38.0 pypi_0 pypi freetype 2.12.1 hca18f0e_0 conda-forge freexl 1.0.6 h166bdaf_1 conda-forge frozenlist 1.3.3 pypi_0 pypi fsspec 2022.11.0 pyhd8ed1ab_0 conda-forge future 0.18.2 pyhd8ed1ab_6 conda-forge gdal 3.5.2 pypi_0 pypi geopandas 0.12.1 pyhd8ed1ab_1 conda-forge geopandas-base 0.12.1 pyha770c72_1 conda-forge geos 3.11.0 h27087fc_0 conda-forge geotiff 1.7.1 h4fc65e6_3 conda-forge geoviews 1.9.5 pyhd8ed1ab_0 conda-forge geoviews-core 1.9.5 pyha770c72_0 conda-forge gettext 0.21.1 h27087fc_0 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge glib 2.74.1 h6239696_1 conda-forge glib-tools 2.74.1 h6239696_1 conda-forge grimpfunc 0.0.4 pypi_0 pypi grimpqgis 0.0.1 pypi_0 pypi ```
fastice commented 1 year ago

Not sure what’s going on. Often the not recognized format indicates there were too many attempts to access NSIDC (you are allowed 16, but they don’t hang up right away so it's easy to get booted out). Try again a bit later and make sure you are not running it simultaneously in more than one notebook or machine (i.e. only have one file downloading from NSIDC at once). Let me know how it goes. Ian

Ian Joughin Polar Science Center Applied Physics Lab University of Washington 1013 NE 40th Street Seattle, WA 98105-6698

Data products: https://nsidc.org/grimp

Data Tools: https://github.com/fastice/GrIMPTools

Phone: 206-221-3177 Fax: 206-616-3142

On Nov 29, 2022, at 2:01 PM, Timothy Bartholomaus @.***> wrote:

Thanks for jumping right back into this, Ian. Importing grimpfunc in both M1 and linux machines, via jupyter lab all works great. But it looks like unpinning some of those packages might have led to other problems for the mac- looks like a gdal problem.

In a cell from your workbook that had previously worked fine

myTerminusUrls = grimp.cmrUrls(mode='terminus') # mode image restricts search to the image products myTerminusUrls.initialSearch();

myTermini = {} for url in myTerminusUrls.getURLS(): year = os.path.basename(url).split('_')[1] # Extract year from name myTermini[year] = gpd.read_file(f'/vsicurl/&url={url}') # Add terminus to data frame print(f'/vsicurl/&url={url}') I now get the following error on the gpd.read_file from url. But! only on mac. On my linux this runs just fine. Here's the complete error message:


CPLE_OpenFailedError Traceback (most recent call last) File fiona/_shim.pyx:83, in fiona._shim.gdal_open_vector()

File fiona/_err.pyx:291, in fiona._err.exc_wrap_pointer()

CPLE_OpenFailedError: '/vsicurl/&url=https://n5eil01u.ecs.nsidc.org/DP4/MEASURES/NSIDC-0642.002/1999.10.13/termini_1999_2000_v02.0.shp https://urldefense.com/v3/__https://n5eil01u.ecs.nsidc.org/DP4/MEASURES/NSIDC-0642.002/1999.10.13/termini_1999_2000_v02.0.shp__;!!K-Hz7m0Vt54!jF4gseudGo0-4x7w4w6XL0Hn0ifSuw1QDEAXe6_3baqmwZhyUezhmAh9VEjL50_2zQzMmfOMiTHQlT_HQZEH$' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

DriverError Traceback (most recent call last) Cell In [18], line 4 2 for url in myTerminusUrls.getURLS(): 3 year = os.path.basename(url).split('_')[1] # Extract year from name ----> 4 myTermini[year] = gpd.read_file(f'/vsicurl/&url={url}') # Add terminus to data frame 5 print(f'/vsicurl/&url={url}')

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.10/site-packages/geopandas/io/file.py:259, in _read_file(filename, bbox, mask, rows, engine, kwargs) 256 path_or_bytes = filename 258 if engine == "fiona": --> 259 return _read_file_fiona( 260 path_or_bytes, from_bytes, bbox=bbox, mask=mask, rows=rows, kwargs 261 ) 262 elif engine == "pyogrio": 263 return _read_file_pyogrio( 264 path_or_bytes, bbox=bbox, mask=mask, rows=rows, **kwargs 265 )

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.10/site-packages/geopandas/io/file.py:303, in _read_file_fiona(path_or_bytes, from_bytes, bbox, mask, rows, where, kwargs) 300 reader = fiona.open 302 with fiona_env(): --> 303 with reader(path_or_bytes, kwargs) as features: 304 crs = features.crs_wkt 305 # attempt to get EPSG code

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.10/site-packages/fiona/env.py:408, in ensure_env_with_credentials..wrapper(*args, kwargs) 405 @wraps(f) 406 def wrapper(*args, *kwargs): 407 if local._env: --> 408 return f(args, kwargs) 409 else: 410 if isinstance(args[0], str):

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.10/site-packages/fiona/init.py:264, in open(fp, mode, driver, schema, crs, encoding, layer, vfs, enabled_drivers, crs_wkt, kwargs) 261 path = parse_path(fp) 263 if mode in ('a', 'r'): --> 264 c = Collection(path, mode, driver=driver, encoding=encoding, 265 layer=layer, enabled_drivers=enabled_drivers, kwargs) 266 elif mode == 'w': 267 if schema: 268 # Make an ordered dict of schema properties.

File ~/opt/anaconda3/envs/greenlandMapping/lib/python3.10/site-packages/fiona/collection.py:162, in Collection.init(self, path, mode, driver, schema, crs, encoding, layer, vsi, archive, enabled_drivers, crs_wkt, ignore_fields, ignore_geometry, kwargs) 160 if self.mode == 'r': 161 self.session = Session() --> 162 self.session.start(self, kwargs) 163 elif self.mode in ('a', 'w'): 164 self.session = WritingSession()

File fiona/ogrext.pyx:540, in fiona.ogrext.Session.start()

File fiona/_shim.pyx:90, in fiona._shim.gdal_open_vector()

DriverError: '/vsicurl/&url=https://n5eil01u.ecs.nsidc.org/DP4/MEASURES/NSIDC-0642.002/1999.10.13/termini_1999_2000_v02.0.shp https://urldefense.com/v3/__https://n5eil01u.ecs.nsidc.org/DP4/MEASURES/NSIDC-0642.002/1999.10.13/termini_1999_2000_v02.0.shp__;!!K-Hz7m0Vt54!jF4gseudGo0-4x7w4w6XL0Hn0ifSuw1QDEAXe6_3baqmwZhyUezhmAh9VEjL50_2zQzMmfOMiTHQlT_HQZEH$' not recognized as a supported file format. Again, this is only a problem on mac (2021 M1 MBP), not on linux, and it looks like, from a comment in the former version of environment.yml that this might be a known issue?

I'm copying below a section of conda list from "identical" environments on both machines.

Mac (2021 M1) Linux fiona 1.8.22 py310h5a14c0a_0 conda-forge flit-core 3.8.0 pyhd8ed1ab_0 conda-forge folium 0.13.0 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.1 h5bb23bf_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.38.0 py310h90acd4f_1 conda-forge freetype 2.12.1 h3f81eb7_1 conda-forge freexl 1.0.6 hb7f2c08_1 conda-forge frozenlist 1.3.3 py310h90acd4f_0 conda-forge fsspec 2022.11.0 pyhd8ed1ab_0 conda-forge future 0.18.2 pyhd8ed1ab_6 conda-forge gdal 3.5.2 py310he22d2f2_3 conda-forge geopandas 0.12.1 pyhd8ed1ab_1 conda-forge geopandas-base 0.12.1 pyha770c72_1 conda-forge geos 3.11.0 hb486fe8_0 conda-forge geotiff 1.7.1 ha1a2aeb_3 conda-forge geoviews 1.9.5 pyhd8ed1ab_0 conda-forge geoviews-core 1.9.5 pyha770c72_0 conda-forge gettext 0.21.1 h8a4c099_0 conda-forge giflib 5.2.1 hbcb3906_2 conda-forge glib 2.74.1 hbc0c0cd_1 conda-forge glib-tools 2.74.1 hbc0c0cd_1 conda-forge grimpfunc 0.0.4 pypi_0 pypi grimpqgis 0.0.1 pypi_0 pypi fiona 1.8.22 pypi_0 pypi flit-core 3.8.0 pyhd8ed1ab_0 conda-forge folium 0.13.0 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.1 hc2a2eb6_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.38.0 pypi_0 pypi freetype 2.12.1 hca18f0e_0 conda-forge freexl 1.0.6 h166bdaf_1 conda-forge frozenlist 1.3.3 pypi_0 pypi fsspec 2022.11.0 pyhd8ed1ab_0 conda-forge future 0.18.2 pyhd8ed1ab_6 conda-forge gdal 3.5.2 pypi_0 pypi geopandas 0.12.1 pyhd8ed1ab_1 conda-forge geopandas-base 0.12.1 pyha770c72_1 conda-forge geos 3.11.0 h27087fc_0 conda-forge geotiff 1.7.1 h4fc65e6_3 conda-forge geoviews 1.9.5 pyhd8ed1ab_0 conda-forge geoviews-core 1.9.5 pyha770c72_0 conda-forge gettext 0.21.1 h27087fc_0 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge glib 2.74.1 h6239696_1 conda-forge glib-tools 2.74.1 h6239696_1 conda-forge grimpfunc 0.0.4 pypi_0 pypi grimpqgis 0.0.1 pypi_0 pypi — Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/fastice/GrIMPTools/issues/1*issuecomment-1331376995__;Iw!!K-Hz7m0Vt54!jF4gseudGo0-4x7w4w6XL0Hn0ifSuw1QDEAXe6_3baqmwZhyUezhmAh9VEjL50_2zQzMmfOMiTHQlUi3glxd$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALICGTU44H2VHUC62SF5YBTWKZ4MTANCNFSM6AAAAAASN53B2A__;!!K-Hz7m0Vt54!jF4gseudGo0-4x7w4w6XL0Hn0ifSuw1QDEAXe6_3baqmwZhyUezhmAh9VEjL50_2zQzMmfOMiTHQlWiSDRVt$. You are receiving this because you commented.

tbartholomaus commented 1 year ago

I think your read on this must have been right. I tried yesterday afternoon, and again this morning, and it's working no problem. Kind of unfortunate that this is the way NSIDC declines access attempts, but it's good to know now. I think I'm in business!

Thanks again, Ian. Really, your vision of and support for community/open science is just great. Very much appreciated.