fziegner / tobac_realcase_testing

1 stars 1 forks source link

Realcase Testing Crashes for the PR example #3

Open fsenf opened 2 months ago

fsenf commented 2 months ago

Hi @fziegner ,

I created a description for using realcase testing for PRs. This is included here: #2 .

Somehow, the command is not working in my case and throws the following error:

...
Traceback (most recent call last):
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/create_references.py", line 136, in <module>
    main()
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/create_references.py", line 132, in main
    create_reference_data(source_notebooks, args.sv, args.name)
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/create_references.py", line 123, in create_reference_data
    run_notebook(notebook, output_path)
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/create_references.py", line 97, in run_notebook
    ep.preprocess(nb, {"metadata": {"path": output}})
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/testing_PR386/realcase_testing/lib/python3.12/site-packages/nbconvert/preprocessors/execute.py", line 103, in preprocess
    self.preprocess_cell(cell, resources, index)
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/testing_PR386/realcase_testing/lib/python3.12/site-packages/nbconvert/preprocessors/execute.py", line 124, in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/testing_PR386/realcase_testing/lib/python3.12/site-packages/jupyter_core/utils/__init__.py", line 165, in wrapped
    return loop.run_until_complete(inner)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/testing_PR386/realcase_testing/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/testing_PR386/realcase_testing/lib/python3.12/site-packages/nbclient/client.py", line 1062, in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/testing_PR386/realcase_testing/lib/python3.12/site-packages/nbclient/client.py", line 918, in _check_raise_for_error
    raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
# Import tobac itself:
import tobac
print('using tobac version', str(tobac.__version__))
------------------

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[2], line 2
      1 # Import tobac itself:
----> 2 import tobac
      3 print('using tobac version', str(tobac.__version__))

ModuleNotFoundError: No module named 'tobac'

ERROR conda.cli.main_run:execute(124): `conda run python create_references.py --nb 2fe56c08928208bdd5416fc4f3ee560a3329b8f9 --sv ./testing_PR386 --name source_reference_data --nb_names All` failed. (See above for error)
Traceback (most recent call last):
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/./realcase_testing.py", line 323, in <module>
    main()
  File "/home/senf/TROPOS/proj/2022-06-tobac-devel/tobac_realcase_testing/./realcase_testing.py", line 263, in main
    subprocess.run(
  File "/home/senf/.conda/base/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mamba', 'run', '-p', './testing_PR386/realcase_testing', 'python', 'create_references.py', '--nb', '2fe56c08928208bdd5416fc4f3ee560a3329b8f9', '--sv', './testing_PR386', '--name', 'source_reference_data', '--nb_names', 'All']' returned non-zero exit status 1.

Can you somehow reproduce the error?

fziegner commented 2 months ago

Tobac is found in my case but instead I receive a different error while importing it:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[2], line 2
      1 # Import tobac itself:
----> 2 import tobac
      3 print('using tobac version', str(tobac.__version__))

File ~\Desktop\tobac_realcase_testing\testing_PR386\realcase_testing\Lib\site-packages\tobac\__init__.py:26
     15 from .segmentation import (
     16     segmentation_3D,
     17     segmentation_2D,
     18     watershedding_3D,
     19     watershedding_2D,
     20 )
     21 from .centerofgravity import (
     22     calculate_cog,
     23     calculate_cog_untracked,
     24     calculate_cog_domain,
     25 )
---> 26 from .plotting import (
     27     plot_tracks_mask_field,
     28     plot_tracks_mask_field_loop,
     29     plot_mask_cell_track_follow,
     30     plot_mask_cell_track_static,
     31     plot_mask_cell_track_static_timeseries,
     32 )
     33 from .plotting import (
     34     plot_lifetime_histogram,
     35     plot_lifetime_histogram_bar,
     36     plot_histogram_cellwise,
     37     plot_histogram_featurewise,
     38 )
     39 from .plotting import plot_mask_cell_track_3Dstatic, plot_mask_cell_track_2D3Dstatic

File ~\Desktop\tobac_realcase_testing\testing_PR386\realcase_testing\Lib\site-packages\tobac\plotting.py:20
     18 import warnings
     19 import logging
---> 20 from .analysis import lifetime_histogram
     21 from .analysis import histogram_cellwise, histogram_featurewise
     23 import numpy as np

File ~\Desktop\tobac_realcase_testing\testing_PR386\realcase_testing\Lib\site-packages\tobac\analysis\__init__.py:29
      1 """Provide tools to analyse and visualize the tracked objects.          
      2 This module provides a set of routines that enables performing analyses
      3 and deriving statistics for individual tracks, such as the time series
   (...)
     26 -----
     27 """
---> 29 from tobac.analysis.cell_analysis import *
     30 from tobac.analysis.feature_analysis import *
     31 from tobac.analysis.spatial import *

File ~\Desktop\tobac_realcase_testing\testing_PR386\realcase_testing\Lib\site-packages\tobac\analysis\cell_analysis.py:18
     16 from tobac.utils.mask import mask_cell, mask_cell_surface, mask_cube_cell
     17 from tobac.utils.general import get_bounding_box
---> 18 from tobac.analysis.spatial import (
     19     calculate_distance,
     20     calculate_velocity,
     21 )
     23 __all__ = (
     24     "cell_statistics_all",
     25     "cell_statistics",
   (...)
     30     "calculate_overlap",
     31 )
     34 def cell_statistics_all(
     35     input_cubes,
     36     track,
   (...)
     45     **kwargs,
     46 ):

File ~\Desktop\tobac_realcase_testing\testing_PR386\realcase_testing\Lib\site-packages\tobac\analysis\spatial.py:329
    324     area = dists1 * dists2
    326     return area
--> 329 @decorators.xarray_to_iris()
    330 def calculate_area(features, mask, method_area=None, vertical_coord=None):
    331     """Calculate the area of the segments for each feature.
    332
   (...)
    375
    376     """
    378     features["area"] = np.nan

TypeError: xarray_to_iris() missing 1 required positional argument: 'func'

ERROR conda.cli.main_run:execute(124): `conda run python create_references.py --nb 2fe56c08928208bdd5416fc4f3ee560a3329b8f9 --sv ./testing_PR386 --name target_reference_data --nb_names [Example_OLR_Tracking_model]` failed. (See above for error)
Traceback (most recent call last):
  File "C:\Users\Fabian\Desktop\tobac_realcase_testing\realcase_testing.py", line 323, in <module>
    main()
  File "C:\Users\Fabian\Desktop\tobac_realcase_testing\realcase_testing.py", line 289, in main
    subprocess.run(
  File "C:\Users\Fabian\miniforge3\envs\tobac_dev\lib\subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mamba', 'run', '-p', './testing_PR386\\realcase_testing', 'python', 'create_references.py', '--nb', '2fe56c08928208bdd5416fc4f3ee560a3329b8f9', '--sv', './testing_PR386', '--name', 'target_reference_data', '--nb_names', '[Example_OLR_Tracking_model]']' returned non-zero exit status 1.
fsenf commented 2 months ago

Hi @fziegner ,

I did another test on the supercomputer levante of the DKRZ.

Two issues appeared:

  1. The script tried to write into the system-wide conda /mamba installation
...
error    libmamba Error opening for writing "/sw/spack-levante/mambaforge-4.11.0-0-Linux-x86_64-sobz6z/pkgs/urls.txt": Read-only file system
...
  1. The script finally failed again with
...
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[3], line 2
      1 # Import tobac itself:
----> 2 import tobac
      3 import tobac.utils
      5 print("using tobac version", str(tobac.__version__))

ModuleNotFoundError: No module named 'tobac'

ERROR conda.cli.main_run:execute(49): `conda run python create_references.py --nb 2fe56c08928208bdd5416fc4f3ee560a3329b8f9 --sv ./testing_PR386 --name source_reference_data --nb_names All` failed. (See above for error)
Traceback (most recent call last):
  File "/scratch/b/b380352/tobac_realcase_testing/./realcase_testing.py", line 323, in <module>
    main()
  File "/scratch/b/b380352/tobac_realcase_testing/./realcase_testing.py", line 263, in main
    subprocess.run(
  File "/sw/spack-levante/mambaforge-4.11.0-0-Linux-x86_64-sobz6z/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mamba', 'run', '-p', './testing_PR386/realcase_testing', 'python', 'create_references.py', '--nb', '2fe56c08928208bdd5416fc4f3ee560a3329b8f9', '--sv', './testing_PR386', '--name', 'source_reference_data', '--nb_names', 'All']' returned non-zero exit status 1.

Do you have any idea? Any chance for independent testing on your side?

Cheers, Fabian.

fziegner commented 2 months ago

Sorry for the late response, I will be looking into it today.