search

pytroll / pytroll-collectors

Collector modules for Pytroll
GNU General Public License v3.0
3 stars 18 forks source link

If tle is missing the actual gatherer thread is crashing #30

Open TAlonglong opened 5 years ago

TAlonglong commented 5 years ago

If tle for some reason is missing the actual gatherer thread is crashing like this:

[DEBUG: 2019-02-13 04:12:02 : trollsched.satpass] Failed in PyOrbital: u"Found no TLE entry for 'NOAA 19'"
[DEBUG: 2019-02-13 04:12:02 : pyorbital.tlefile] Reading TLE from /data/pytroll/tle-in/tle_db/tle-latest-pytroll.txt
Exception in thread Thread-11:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/software/pytroll/lib/python2.7/site-packages/pytroll_collectors-0.7.0-py2.7.egg/pytroll_collectors/trigger.py", line 381, in run
    self.process(msg)
  File "/software/pytroll/lib/python2.7/site-packages/pytroll_collectors-0.7.0-py2.7.egg/pytroll_collectors/trigger.py", line 94, in add_file
    self._do(pathname)
  File "/software/pytroll/lib/python2.7/site-packages/pytroll_collectors-0.7.0-py2.7.egg/pytroll_collectors/trigger.py", line 89, in _do
    Trigger._do(self, mda)
  File "/software/pytroll/lib/python2.7/site-packages/pytroll_collectors-0.7.0-py2.7.egg/pytroll_collectors/trigger.py", line 64, in _do
    res = collector(metadata.copy())
  File "/software/pytroll/lib/python2.7/site-packages/pytroll_collectors-0.7.0-py2.7.egg/pytroll_collectors/region_collector.py", line 61, in __call__
    return self.collect(granule_metadata)
  File "/software/pytroll/lib/python2.7/site-packages/pytroll_collectors-0.7.0-py2.7.egg/pytroll_collectors/region_collector.py", line 169, in collect
    instrument=granule_metadata["sensor"])
  File "/software/pytroll/lib/python2.7/site-packages/pytroll_schedule-0.3.3-py2.7.egg/trollsched/satpass.py", line 184, in __init__
    self.orb = orbital.Orbital(NOAA20_NAME.get(satellite, satellite), line1=tle1, line2=tle2)
  File "/software/pytroll/lib/python2.7/site-packages/pyorbital-1.3.1-py2.7.egg/pyorbital/orbital.py", line 142, in __init__
    line1=line1, line2=line2)
  File "/software/pytroll/lib/python2.7/site-packages/pyorbital-1.3.1-py2.7.egg/pyorbital/tlefile.py", line 146, in read
    return Tle(platform, tle_file=tle_file, line1=line1, line2=line2)
  File "/software/pytroll/lib/python2.7/site-packages/pyorbital-1.3.1-py2.7.egg/pyorbital/tlefile.py", line 197, in __init__
    self._read_tle()
  File "/software/pytroll/lib/python2.7/site-packages/pyorbital-1.3.1-py2.7.egg/pyorbital/tlefile.py", line 280, in _read_tle
    raise KeyError("Found no TLE entry for '%s'" % self._platform)
KeyError: u"Found no TLE entry for 'NOAA 19'"

A possible try/excet with restart might avoid this. Or at least restart the tread.

TAlonglong commented 2 years ago

I got this again. Now a bit different traceback:

[DEBUG: 2022-03-26 17:04:04 : pyorbital.tlefile] Reading TLE from /data/pytroll/tle/tle_db/tle-latest-pytroll.txt
[DEBUG: 2022-03-26 17:04:04 : trollsched.satpass] Failed in PyOrbital: "Found no TLE entry for 'METOP-B'"
[DEBUG: 2022-03-26 17:04:04 : pyorbital.tlefile] Reading TLE from /data/pytroll/tle/tle_db/tle-latest-pytroll.txt
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/trollsched/satpass.py", line 180, in __init__
    self.orb = orbital.Orbital(satellite, line1=tle1, line2=tle2)
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pyorbital/orbital.py", line 156, in __init__
    self.tle = tlefile.read(satellite, tle_file=tle_file,
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pyorbital/tlefile.py", line 113, in read
    return Tle(platform, tle_file=tle_file, line1=line1, line2=line2)
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pyorbital/tlefile.py", line 159, in __init__
    self._read_tle()
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pyorbital/tlefile.py", line 200, in _read_tle
    raise KeyError("Found no TLE entry for '%s'" % self._platform)
KeyError: "Found no TLE entry for 'METOP-B'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/software/miniconda/envs/pytroll/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pytroll_collectors/triggers/_posttroll.py", line 68, in run
    self.process(msg)
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pytroll_collectors/triggers/_base.py", line 186, in add_file
    self._process_pathname(pathname)
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pytroll_collectors/triggers/_base.py", line 182, in _process_pathname
    Trigger._process_metadata(self, mda)
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pytroll_collectors/triggers/_base.py", line 85, in _process_metadata
    res = collector(metadata.copy())
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pytroll_collectors/region_collector.py", line 69, in __call__
    return self.collect(granule_metadata)
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pytroll_collectors/region_collector.py", line 105, in collect
    if _granule_covers_region(granule_metadata, self.region):
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pytroll_collectors/region_collector.py", line 306, in _granule_covers_region
    granule_pass = Pass(_get_platform_name(granule_metadata),
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/trollsched/satpass.py", line 183, in __init__
    self.orb = orbital.Orbital(
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pyorbital/orbital.py", line 156, in __init__
    self.tle = tlefile.read(satellite, tle_file=tle_file,
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pyorbital/tlefile.py", line 113, in read
    return Tle(platform, tle_file=tle_file, line1=line1, line2=line2)
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pyorbital/tlefile.py", line 159, in __init__
    self._read_tle()
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/pyorbital/tlefile.py", line 200, in _read_tle
    raise KeyError("Found no TLE entry for '%s'" % self._platform)
KeyError: "Found no TLE entry for 'METOP-B'"

So here is where this can be caught:

https://github.com/pytroll/pytroll-collectors/blob/main/pytroll_collectors/triggers/_posttroll.py#L58-L70

The problem is that this gatherer thread stops, but the rest is working. So its not restarted by eg supervisor. I think this is similar to what is commented in #79

Is it possible to have a try catch for IndexError around the process line? In my case I think this error happened due to a race condition in my processing setup. The TLE file was just updated at the same time as the collector tried to read it and then could not find the Metop-B TLE.

This can be problematic if the user does not provide the requested TLE if the exception is just passed. But it will partly solve my problem. I will loose the segment handled, but I think this is better than the gathering for this area is stopping.