search

cubed-dev / cubed

Bounded-memory serverless distributed N-dimensional array processing
https://cubed-dev.github.io/cubed/
Apache License 2.0
116 stars 14 forks source link

`TimelineVisualizationCallback()` usage [cubed - cubed/xarray] #544

Closed norlandrhagen closed 1 month ago

norlandrhagen commented 1 month ago

Related to this issue: https://github.com/cubed-dev/cubed/issues/543. I'm trying out some of the callbacks in timeline. Once again, might be getting the usage totally wrong. 


from cubed import Spec
import xarray as xr 
from cubed.diagnostics.timeline import TimelineVisualizationCallback

spec = Spec(work_dir='tmp', allowed_mem='2GB')
timeline_viz = TimelineVisualizationCallback()

ds = xr.tutorial.open_dataset('air_temperature', chunked_array_type='cubed',
     from_array_kwargs={'spec': spec},chunks={})

ds.compute(callbacks=[timeline_viz])

When I call this, I get a FileNotFoundError:

FileNotFoundError: [Errno 2] No such file or directory: '~/cubed/notebooks/history/compute-20240807T124600.864122/timeline.svg'

Full traceback:

```python FileNotFoundError Traceback (most recent call last) Cell In[23], [line 12](vscode-notebook-cell:?execution_count=23&line=12) [6](vscode-notebook-cell:?execution_count=23&line=6) timeline_viz = TimelineVisualizationCallback() [8](vscode-notebook-cell:?execution_count=23&line=8) ds = xr.tutorial.open_dataset('air_temperature', chunked_array_type='cubed', [9](vscode-notebook-cell:?execution_count=23&line=9) from_array_kwargs={'spec': spec},chunks={}) ---> [12](vscode-notebook-cell:?execution_count=23&line=12) ds.compute(callbacks=[timeline_viz]) File ~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:1038, in Dataset.compute(self, **kwargs) [1014](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:1014) """Manually trigger loading and/or computation of this dataset's data [1015](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:1015) from disk or a remote source into memory and return a new dataset. [1016](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:1016) Unlike load, the original dataset is left unaltered. (...) [1035](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:1035) dask.compute [1036](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:1036) """ [1037](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:1037) new = self.copy(deep=False) -> [1038](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:1038) return new.load(**kwargs) File ~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:865, in Dataset.load(self, **kwargs) [862](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:862) chunkmanager = get_chunked_array_type(*lazy_data.values()) [864](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:864) # evaluate all the chunked arrays simultaneously --> [865](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:865) evaluated_data: tuple[np.ndarray[Any, Any], ...] = chunkmanager.compute( [866](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:866) *lazy_data.values(), **kwargs [867](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:867) ) [869](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:869) for k, data in zip(lazy_data, evaluated_data): [870](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/xarray/core/dataset.py:870) self.variables[k].data = data File ~/micromamba/envs/cubed/lib/python3.11/site-packages/cubed_xarray/cubedmanager.py:65, in CubedManager.compute(self, *data, **kwargs) [62](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/cubed_xarray/cubedmanager.py:62) def compute(self, *data: "CubedArray", **kwargs) -> tuple[np.ndarray, ...]: [63](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/cubed_xarray/cubedmanager.py:63) from cubed import compute ---> [65](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/cubed_xarray/cubedmanager.py:65) return compute(*data, **kwargs) File ~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:282, in compute(executor, callbacks, optimize_graph, optimize_function, resume, *arrays, **kwargs) [279](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:279) executor = SingleThreadedExecutor() [281](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:281) _return_in_memory_array = kwargs.pop("_return_in_memory_array", True) --> [282](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:282) plan.execute( [283](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:283) executor=executor, [284](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:284) callbacks=callbacks, [285](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:285) optimize_graph=optimize_graph, [286](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:286) optimize_function=optimize_function, [287](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:287) resume=resume, [288](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:288) spec=spec, [289](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:289) **kwargs, [290](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:290) ) [292](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:292) if _return_in_memory_array: [293](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/array.py:293) return tuple(a._read_stored() for a in arrays) File ~/Documents/carbonplan/cubed/cubed/cubed/core/plan.py:276, in Plan.execute(self, executor, callbacks, optimize_graph, optimize_function, compile_function, resume, spec, **kwargs) [274](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/plan.py:274) if callbacks is not None: [275](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/plan.py:275) event = ComputeEndEvent(compute_id, dag) --> [276](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/plan.py:276) [callback.on_compute_end(event) for callback in callbacks] File ~/Documents/carbonplan/cubed/cubed/cubed/core/plan.py:276, in (.0) [274](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/plan.py:274) if callbacks is not None: [275](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/plan.py:275) event = ComputeEndEvent(compute_id, dag) --> [276](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/core/plan.py:276) [callback.on_compute_end(event) for callback in callbacks] File ~/Documents/carbonplan/cubed/cubed/cubed/diagnostics/timeline.py:33, in TimelineVisualizationCallback.on_compute_end(self, event) [31](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/diagnostics/timeline.py:31) dst = f"history/{event.compute_id}" [32](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/diagnostics/timeline.py:32) format = self.format ---> [33](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/diagnostics/timeline.py:33) create_timeline(self.stats, self.start_tstamp, end_tstamp, dst, format) File ~/Documents/carbonplan/cubed/cubed/cubed/diagnostics/timeline.py:101, in create_timeline(stats, start_tstamp, end_tstamp, dst, format) [98](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/diagnostics/timeline.py:98) dst = os.path.expanduser(dst) if "~" in dst else dst [99](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/diagnostics/timeline.py:99) dst = "{}/{}".format(os.path.realpath(dst), f"timeline.{format}") --> [101](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/Documents/carbonplan/cubed/cubed/cubed/diagnostics/timeline.py:101) fig.savefig(dst) File ~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/figure.py:3395, in Figure.savefig(self, fname, transparent, **kwargs) [3393](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/figure.py:3393) for ax in self.axes: [3394](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/figure.py:3394) _recursively_make_axes_transparent(stack, ax) -> [3395](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/figure.py:3395) self.canvas.print_figure(fname, **kwargs) File ~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2204, in FigureCanvasBase.print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, bbox_inches, pad_inches, bbox_extra_artists, backend, **kwargs) [2200](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2200) try: [2201](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2201) # _get_renderer may change the figure dpi (as vector formats [2202](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2202) # force the figure dpi to 72), so we need to set it again here. [2203](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2203) with cbook._setattr_cm(self.figure, dpi=dpi): -> [2204](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2204) result = print_method( [2205](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2205) filename, [2206](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2206) facecolor=facecolor, [2207](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2207) edgecolor=edgecolor, [2208](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2208) orientation=orientation, [2209](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2209) bbox_inches_restore=_bbox_inches_restore, [2210](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2210) **kwargs) [2211](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2211) finally: [2212](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2212) if bbox_inches and restore_bbox: File ~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2054, in FigureCanvasBase._switch_canvas_and_return_print_method..(*args, **kwargs) [2050](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2050) optional_kws = { # Passed by print_figure for other renderers. [2051](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2051) "dpi", "facecolor", "edgecolor", "orientation", [2052](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2052) "bbox_inches_restore"} [2053](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2053) skip = optional_kws - {*inspect.signature(meth).parameters} -> [2054](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2054) print_method = functools.wraps(meth)(lambda *args, **kwargs: meth( [2055](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2055) *args, **{k: v for k, v in kwargs.items() if k not in skip})) [2056](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/backend_bases.py:2056) else: # Let third-parties do as they see fit. ... --> [483](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/cbook.py:483) fh = open(fname, flag, encoding=encoding) [484](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/cbook.py:484) opened = True [485](https://file+.vscode-resource.vscode-cdn.net/Users/nrhagen/Documents/carbonplan/cubed/notebooks/~/micromamba/envs/cubed/lib/python3.11/site-packages/matplotlib/cbook.py:485) elif hasattr(fname, 'seek'): ```

It seems like in diagnostics.timeline.py L101 savefig is triggering the FileNotFoundError. Does this need some write/create permissions to save a svg?

tomwhite commented 1 month ago

Does this need some write/create permissions to save a svg?

Yes, it needs permissions to create the history/compute-{id} directories and to write the SVG file. Perhaps we need another os.makedirs before savefig?

norlandrhagen commented 1 month ago

https://github.com/cubed-dev/cubed/issues/538