search

isofit / isofit

Imaging Spectrometer Optimal FITting (ISOFIT) contains a set of routines and utilities for fitting surface, atmosphere and instrument models to imaging spectrometer data.
Apache License 2.0
82 stars 80 forks source link

More useful error messages if environment is not set up correctly or extra-files are missing #466

Open danschef opened 6 months ago

danschef commented 6 months ago

isofit 3.0 (current dev branch)

There are a couple of not very user-friendly exceptions that are raised in case the environment is not correctly configured or the ISOFIT extra-files are missing. I think these cases could easily be catched beforehand and a useful error message could be raised pointing to the solution.

FileNotFoundError if `isofit download data` was not executed -> data folder missing ```python test_isofit_enmap.py::Test_ISOFIT_EnMAP::test_run_isofit ======================== 1 failed, 3 warnings in 30.74s ======================== 2024-03-14 20:17:49,255 INFO worker.py:1724 -- Started a local Ray instance. Stopping ray. FAILED tests/test_isofit_enmap.py:63 (Test_ISOFIT_EnMAP.test_run_isofit) self = def test_run_isofit(self): # from geoarray import GeoArray # gA = GeoArray('/home/gfz-fe/scheffler/temp/EnPT/isofit_implementation/data_in/emp20220712t184754_rdn_sub.bil') # gA.show() from time import time t0 = time() > IsofitEnMAP()._apply_oe( # input_radiance='/home/gfz-fe/scheffler/temp/EnPT/isofit_implementation/data_in/emp20220712t184754_rdn_sub.bil', input_radiance='/home/gfz-fe/scheffler/temp/EnPT/isofit_implementation/data_in/ENMAP01-____L1X-DT000000XXXX_20220712T000000Z_00x_VXXXXXX_XXXXXXTXXXXXXZ.bil', input_loc='/home/gfz-fe/scheffler/temp/EnPT/isofit_implementation/data_in/emp20220712t184754_loc_sub.bil', input_obs='/home/gfz-fe/scheffler/temp/EnPT/isofit_implementation/data_in/emp20220712t184754_obs_sub.bil', working_directory='/home/gfz-fe/scheffler/temp/EnPT/isofit_implementation/data_out/', surface_path='/home/gfz-fe/scheffler/temp/EnPT/isofit_implementation/surface/surface_20221020_EnMAP.mat', wavelength_path='/home/gfz-fe/scheffler/temp/EnPT/isofit_implementation/sensor_new/enmap_wavelengths.txt', log_file='/home/gfz-fe/scheffler/temp/EnPT/isofit_implementation/data_out/isofit.log', presolve=True, emulator_base='/home/gfz-fe/scheffler/sRTMnet_v100/sRTMnet_v100', # FIXME why not /home/gfz-fe/scheffler/sRTMnet_v100/ n_cores=30 ) test_isofit_enmap.py:72: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../enpt/processors/atmospheric_correction/_isofit_enmap.py:108: in _apply_oe apply_oe(SimpleNamespace(**params)) ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/utils/apply_oe.py:590: in apply_oe retrieval_full.run() ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/core/isofit.py:182: in run res = list( ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/ray/util/actor_pool.py:170: in get_generator yield self.get_next_unordered() ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/ray/util/actor_pool.py:370: in get_next_unordered return ray.get(future) ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/ray/_private/auto_init_hook.py:22: in auto_init_wrapper return fn(*args, **kwargs) ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/ray/_private/client_mode_hook.py:103: in wrapper return func(*args, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ object_refs = [ObjectRef(8c4854248414f6339358be101ae77cb2af9663a30100000001000000)] @PublicAPI @client_mode_hook def get( object_refs: Union["ObjectRef[Any]", Sequence["ObjectRef[Any]"]], *, timeout: Optional[float] = None, ) -> Union[Any, List[Any]]: """Get a remote object or a list of remote objects from the object store. This method blocks until the object corresponding to the object ref is available in the local object store. If this object is not in the local object store, it will be shipped from an object store that has it (once the object has been created). If object_refs is a list, then the objects corresponding to each object in the list will be returned. Ordering for an input list of object refs is preserved for each object returned. That is, if an object ref to A precedes an object ref to B in the input list, then A will precede B in the returned list. This method will issue a warning if it's running inside async context, you can use ``await object_ref`` instead of ``ray.get(object_ref)``. For a list of object refs, you can use ``await asyncio.gather(*object_refs)``. Passing :class:`~ObjectRefGenerator` is not allowed. Related patterns and anti-patterns: - :doc:`/ray-core/patterns/ray-get-loop` - :doc:`/ray-core/patterns/unnecessary-ray-get` - :doc:`/ray-core/patterns/ray-get-submission-order` - :doc:`/ray-core/patterns/ray-get-too-many-objects` Args: object_refs: Object ref of the object to get or a list of object refs to get. timeout (Optional[float]): The maximum amount of time in seconds to wait before returning. Set this to None will block until the corresponding object becomes available. Setting ``timeout=0`` will return the object immediately if it's available, else raise GetTimeoutError in accordance with the above docstring. Returns: A Python object or a list of Python objects. Raises: GetTimeoutError: A GetTimeoutError is raised if a timeout is set and the get takes longer than timeout to return. Exception: An exception is raised if the task that created the object or that created one of the objects raised an exception. """ worker = global_worker worker.check_connected() if hasattr(worker, "core_worker") and worker.core_worker.current_actor_is_asyncio(): global blocking_get_inside_async_warned if not blocking_get_inside_async_warned: logger.warning( "Using blocking ray.get inside async actor. " "This blocks the event loop. Please use `await` " "on object ref with asyncio.gather if you want to " "yield execution to the event loop instead." ) blocking_get_inside_async_warned = True with profiling.profile("ray.get"): # TODO(sang): Should make ObjectRefGenerator # compatible to ray.get for dataset. if isinstance(object_refs, ObjectRefGenerator): return object_refs is_individual_id = isinstance(object_refs, ray.ObjectRef) if is_individual_id: object_refs = [object_refs] if not isinstance(object_refs, list): raise ValueError( f"Invalid type of object refs, {type(object_refs)}, is given. " "'object_refs' must either be an ObjectRef or a list of ObjectRefs. " ) # TODO(ujvl): Consider how to allow user to retrieve the ready objects. values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout) for i, value in enumerate(values): if isinstance(value, RayError): if isinstance(value, ray.exceptions.ObjectLostError): worker.core_worker.dump_object_store_memory_usage() if isinstance(value, RayTaskError): > raise value.as_instanceof_cause() E ray.exceptions.RayTaskError(FileNotFoundError): ray::Worker.run_set_of_spectra() (pid=1671034, ip=139.17.56.61, actor_id=9358be101ae77cb2af9663a301000000, repr=) E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/core/isofit.py", line 250, in run_set_of_spectra E input_data = self.io.get_components_at_index(row, col) E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/core/fileio.py", line 495, in get_components_at_index E geom = Geometry( E ^^^^^^^^^ E File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/core/geometry.py", line 61, in __init__ E self.earth_sun_distance_reference = np.loadtxt(self.earth_sun_distance_path) E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/numpy/lib/npyio.py", line 1373, in loadtxt E arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter, E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/numpy/lib/npyio.py", line 992, in _read E fh = np.lib._datasource.open(fname, 'rt', encoding=encoding) E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/numpy/lib/_datasource.py", line 193, in open E return ds.open(path, mode, encoding=encoding, newline=newline) E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/numpy/lib/_datasource.py", line 533, in open E raise FileNotFoundError(f"{path} not found.") E FileNotFoundError: /home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/data/earth_sun_distance.txt not found. ```
FileNotFoundError if `isofit download examples` was not executed (examples folder missing) ```python ../enpt/processors/atmospheric_correction/_isofit_enmap.py:100: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/utils/apply_oe.py:490: in apply_oe retrieval_h2o.run() ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/core/isofit.py:119: in run self.fm = fm = ForwardModel(self.config) ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/core/forward.py:80: in __init__ self.RT = RadiativeTransfer(self.full_config) ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/radiative_transfer.py:106: in __init__ rte = RTE[confRT.engine_name](**params) ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/radiative_transfer_engine.py:218: in __init__ self.runSimulations() ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/radiative_transfer_engine.py:416: in runSimulations pre = self.preSim() ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/sRTMnet.py:100: in preSim sim = SixSRT( ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/six_s.py:73: in __init__ super().__init__(engine_config, **kwargs) ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/radiative_transfer_engine.py:218: in __init__ self.runSimulations() ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/radiative_transfer_engine.py:442: in runSimulations post = self.postSim() ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/six_s.py:159: in postSim irr = np.loadtxt(self.engine_config.irradiance_file, comments="#") ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/numpy/lib/npyio.py:1373: in loadtxt arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter, ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/numpy/lib/npyio.py:992: in _read fh = np.lib._datasource.open(fname, 'rt', encoding=encoding) ../../../../mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/numpy/lib/_datasource.py:193: in open return ds.open(path, mode, encoding=encoding, newline=newline) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = path = '/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/examples/20151026_SantaMonica/data/prism_optimized_irr.dat' mode = 'rt', encoding = None, newline = None def open(self, path, mode='r', encoding=None, newline=None): """ Open and return file-like object. If `path` is an URL, it will be downloaded, stored in the `DataSource` directory and opened from there. Parameters ---------- path : str Local file path or URL to open. mode : {'r', 'w', 'a'}, optional Mode to open `path`. Mode 'r' for reading, 'w' for writing, 'a' to append. Available modes depend on the type of object specified by `path`. Default is 'r'. encoding : {None, str}, optional Open text file with given encoding. The default encoding will be what `io.open` uses. newline : {None, str}, optional Newline to use when reading text file. Returns ------- out : file object File object. """ # TODO: There is no support for opening a file for writing which # doesn't exist yet (creating a file). Should there be? # TODO: Add a ``subdir`` parameter for specifying the subdirectory # used to store URLs in self._destpath. if self._isurl(path) and self._iswritemode(mode): raise ValueError("URLs are not writeable") # NOTE: _findfile will fail on a new file opened for writing. found = self._findfile(path) if found: _fname, ext = self._splitzipext(found) if ext == 'bz2': mode.replace("+", "") return _file_openers[ext](found, mode=mode, encoding=encoding, newline=newline) else: > raise FileNotFoundError(f"{path} not found.") E FileNotFoundError: /home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/examples/20151026_SantaMonica/data/prism_optimized_irr.dat not found. ```
NameError if SIXS_DIR not set to /path/to/6sV2.1 ```python [...] :task_name:streamSimulation [2024-03-14 17:31:55,918 E 1580751 1580751] core_worker.cc:1749: Pushed Error with JobID: 01000000 of type: task with message: ray::streamSimulation() (pid=1580751, ip=139.17.56.61) File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/radiative_transfer_engine.py", line 624, in streamSimulation data = reader(point) ^^^^^^^^^^^^^ File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/six_s.py", line 151, in readSim return self.parse_file(file=file, wl=self.wl, wl_size=self.wl.size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/six_s.py", line 445, in parse_file Logger.error(f"Failed to parse any data for point: {point}") ^^^^^ NameError: name 'point' is not defined at time: 1.71043e+09 ```
FileNotFoundError if EMULATOR_PATH not set or apply_oe(emulator_base=None) ```python :task_name:streamSimulation [2024-03-14 13:50:38,732 E 1079104 1079104] core_worker.cc:1749: Pushed Error with JobID: 01000000 of type: task with message: ray::streamSimulation() (pid=1079104, ip=139.17.56.61) File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/radiative_transfer_engine.py", line 624, in streamSimulation data = reader(point) ^^^^^^^^^^^^^ File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/modtran.py", line 270, in readSim solzen = self.load_tp6(f"{file}.tp6") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/modtran.py", line 200, in load_tp6 with open(file, "r") as tp6: ^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpfm0mwnjo/lut_full/AERFRAC_2-0.6004_H2OSTR-1.7825_surface_elevation_km-1.0528.tp6' at time: 1.71042e+09 :task_name:streamSimulation :task_name:streamSimulation [2024-03-14 13:50:38,885 E 1079104 1079104] core_worker.cc:1749: Pushed Error with JobID: 01000000 of type: task with message: ray::streamSimulation() (pid=1079104, ip=139.17.56.61) File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/radiative_transfer_engine.py", line 624, in streamSimulation data = reader(point) ^^^^^^^^^^^^^ File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/modtran.py", line 270, in readSim solzen = self.load_tp6(f"{file}.tp6") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/isofit/radiative_transfer/modtran.py", line 200, in load_tp6 with open(file, "r") as tp6: ^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpfm0mwnjo/lut_full/AERFRAC_2-0.6004_H2OSTR-1.7825_surface_elevation_km-1.2602.tp6' at time: 1.71042e+09 ```
danschef commented 6 months ago

@unbohn

betsymfarris commented 5 months ago

Files are missing for running examples via Quick Start instructions in v3.0.0., e.g: cd examples/20171108_Pasadena ./run_examples_modtran.sh