search

cal-adapt / climakitae

A Python toolkit for retrieving, visualizing, and performing scientific analyses with data from the Cal-Adapt Analytics Engine.
https://climakitae.readthedocs.io
BSD 3-Clause "New" or "Revised" License
21 stars 2 forks source link

Move Boundary data load out of init and to load method #279

Closed elehmer closed 1 year ago

elehmer commented 1 year ago

This PR moves Boundaries data loading to a load method that is then called on DataInterface init.

elehmer commented 1 year ago

Model uncertainty notebook works just fine; internal variability notebook running into an issue in cell 3, specifically at the selections.cached_area = 'CA' line

Unexpected exception formatting exception. Falling back to standard exception
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3378, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/tmp/ipykernel_1886/1238143973.py", line 10, in <module>
    selections.cached_area = 'CA'
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/parameterized.py", line 367, in _f
    instance_param.__set__(obj, val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/parameterized.py", line 369, in _f
    return f(self, obj, val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/parameterized.py", line 1197, in __set__
    self._validate(val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/__init__.py", line 1882, in _validate
    super(ListSelector, self)._validate(o)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/__init__.py", line 1263, in _validate
    self._ensure_value_is_in_objects(val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/__init__.py", line 1296, in _ensure_value_is_in_objects
    self.objects.append(val)
AttributeError: 'dict_keys' object has no attribute 'append'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 1997, in showtraceback
    stb = self.InteractiveTB.structured_traceback(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 1112, in structured_traceback
    return FormattedTB.structured_traceback(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 1006, in structured_traceback
    return VerboseTB.structured_traceback(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 859, in structured_traceback
    formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 812, in format_exception_as_a_whole
    frames.append(self.format_record(r))
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 730, in format_record
    result += ''.join(_format_traceback_lines(frame_info.lines, Colors, self.has_colors, lvals))
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 734, in lines
    pieces = self.included_pieces
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 677, in included_pieces
    scope_pieces = self.scope_pieces
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 617, in scope_pieces
    for piece in self.source.pieces
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 106, in pieces
    return list(self._clean_pieces())
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 130, in _clean_pieces
    raise AssertionError("Pieces mismatches: %s" % mismatches)
AssertionError: Pieces mismatches: [{680, 681}, {696, 695}, {708, 709}, {714, 715}]

Hmm, ok let me take a look. Thanks!

elehmer commented 1 year ago

Model uncertainty notebook works just fine; internal variability notebook running into an issue in cell 3, specifically at the selections.cached_area = 'CA' line

Unexpected exception formatting exception. Falling back to standard exception
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3378, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/tmp/ipykernel_1886/1238143973.py", line 10, in <module>
    selections.cached_area = 'CA'
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/parameterized.py", line 367, in _f
    instance_param.__set__(obj, val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/parameterized.py", line 369, in _f
    return f(self, obj, val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/parameterized.py", line 1197, in __set__
    self._validate(val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/__init__.py", line 1882, in _validate
    super(ListSelector, self)._validate(o)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/__init__.py", line 1263, in _validate
    self._ensure_value_is_in_objects(val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/__init__.py", line 1296, in _ensure_value_is_in_objects
    self.objects.append(val)
AttributeError: 'dict_keys' object has no attribute 'append'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 1997, in showtraceback
    stb = self.InteractiveTB.structured_traceback(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 1112, in structured_traceback
    return FormattedTB.structured_traceback(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 1006, in structured_traceback
    return VerboseTB.structured_traceback(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 859, in structured_traceback
    formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 812, in format_exception_as_a_whole
    frames.append(self.format_record(r))
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 730, in format_record
    result += ''.join(_format_traceback_lines(frame_info.lines, Colors, self.has_colors, lvals))
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 734, in lines
    pieces = self.included_pieces
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 677, in included_pieces
    scope_pieces = self.scope_pieces
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 617, in scope_pieces
    for piece in self.source.pieces
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 106, in pieces
    return list(self._clean_pieces())
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 130, in _clean_pieces
    raise AssertionError("Pieces mismatches: %s" % mismatches)
AssertionError: Pieces mismatches: [{680, 681}, {696, 695}, {708, 709}, {714, 715}]

Hmm, ok let me take a look. Thanks!

Looks like this error is in main as well? Can others confirm?

elehmer commented 1 year ago

Model uncertainty notebook works just fine; internal variability notebook running into an issue in cell 3, specifically at the selections.cached_area = 'CA' line

Unexpected exception formatting exception. Falling back to standard exception
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3378, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/tmp/ipykernel_1886/1238143973.py", line 10, in <module>
    selections.cached_area = 'CA'
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/parameterized.py", line 367, in _f
    instance_param.__set__(obj, val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/parameterized.py", line 369, in _f
    return f(self, obj, val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/parameterized.py", line 1197, in __set__
    self._validate(val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/__init__.py", line 1882, in _validate
    super(ListSelector, self)._validate(o)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/__init__.py", line 1263, in _validate
    self._ensure_value_is_in_objects(val)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/param/__init__.py", line 1296, in _ensure_value_is_in_objects
    self.objects.append(val)
AttributeError: 'dict_keys' object has no attribute 'append'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 1997, in showtraceback
    stb = self.InteractiveTB.structured_traceback(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 1112, in structured_traceback
    return FormattedTB.structured_traceback(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 1006, in structured_traceback
    return VerboseTB.structured_traceback(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 859, in structured_traceback
    formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 812, in format_exception_as_a_whole
    frames.append(self.format_record(r))
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/IPython/core/ultratb.py", line 730, in format_record
    result += ''.join(_format_traceback_lines(frame_info.lines, Colors, self.has_colors, lvals))
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 734, in lines
    pieces = self.included_pieces
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 677, in included_pieces
    scope_pieces = self.scope_pieces
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 617, in scope_pieces
    for piece in self.source.pieces
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/utils.py", line 145, in cached_property_wrapper
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 106, in pieces
    return list(self._clean_pieces())
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/stack_data/core.py", line 130, in _clean_pieces
    raise AssertionError("Pieces mismatches: %s" % mismatches)
AssertionError: Pieces mismatches: [{680, 681}, {696, 695}, {708, 709}, {714, 715}]

Hmm, ok let me take a look. Thanks!

Looks like this error is in main as well? Can others confirm?

Ahh! Multi-Select got merged and this now needs to be: `

selections.cached_area = 'CA'

Ahh! Multi-select got merged to main. This now needs to be: selections.cached_area = ['CA']

I will fix

claalmve commented 1 year ago

Addressed bugs in cae-notebooks PR fix here.

elehmer commented 1 year ago

@claalmve @vicford Are you guys seeing 16sec load times with main or this PR for loading climakitae?

vicford commented 1 year ago

@claalmve @vicford Are you guys seeing 16sec load times with main or this PR for loading climakitae?

Yes, about a 16 sec load time in both branches!

elehmer commented 1 year ago

@claalmve @vicford Are you guys seeing 16sec load times with main or this PR for loading climakitae?

Yes, about a 16 sec load time in both branches!

Load time should now be down to around 4s. I had to split Variable Descriptions CSV reading out of DataInterface as it is used in view which is a top level method and loaded when package is loaded.

What do you guys think of the code I've done?

elehmer commented 1 year ago

In order to get the 4s down we would need to remove load and view from top level methods. When these are loaded it has to load all the dependent packages in the python files.

elehmer commented 1 year ago

I think one way to speed up is to jam data_export, data_interface, data_load, data_view into one file as we load all 4 in the package init and there are duplicate imports. Is everyone ok with this?

elehmer commented 1 year ago

Going to merge as this is an improvement if not complete overhaul of this setup