search

pytroll / pytroll-collectors

Collector modules for Pytroll
GNU General Public License v3.0
3 stars 18 forks source link

gatherer fails with KeyError: 'format' when fallback to default format and no format in metadata #73

Closed gerritholl closed 3 years ago

gerritholl commented 3 years ago

When for some reason in gatherer.py there is a call to terminator() with publish_topic=None, and the metadata (containing a message sent via posttroll?) does not contain the format key, gatherer.py raises a KeyError: 'format'.

MCVE

The true minimum would be a Python script calling terminator() with certain arguments, but I think that's not the most useful way to reproduce it, because the bug may be upstream (why does terminator() get called this way). My setup:

[avhrrl0]
topic=/file/poes/avhrr
directory=/data/pytroll/IN/HRPT/
posttroll_port=0
publish_port=
event_names=IN_CLOSE_WRITE,IN_MOVED_TO
loglevel=DEBUG
stalker_log_config=/opt/pytroll/pytroll_inst/config/trollstalker_logging.ini
filepattern={path}AVHR_HRP_00_{platform_name}_{start_time:%Y%m%d%H%M%S}Z_{end_time:%Y%m%d%H%M%S}Z_N_O_{processing_time:%Y%m%d%H%M%S}Z
instruments=avhrr/3
alias_platform_name=M01:Metop-B|M02:Metop-A|M03:Metop-C|noaa18:NOAA-18|noaa19:NOAA-19
history=0
nameservers=localhost
[default]
regions = eurol
timeliness = 180
service =
topics = /file/poes/avhrr

[metopa-avhrr-hrpt]
regions = eurol
service =
pattern = /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_{start_time:%Y%m%d%H%M%S}Z_{end_time:%Y%m%d%H%M%S}Z_N_O_{processing_time:%Y%m%d%H%M%S}Z
sensor = avhrr
topics = /file/poes/avhrr
publish_topic = /collection/metopa/avhrr
timeliness = 180
duration = 300
platform_name = Metop-A
format = HRPT
type = binary
variant = DR
level = 0
orbit_type = polar
nameserver = localhost

Running gatherer.py -v gatherer.ini. In another terminal, move some matching files to the directory watched by trollstalker (mv AVHR_HRP_00_M02_20210114080* /tmp/foo/ && mv /tmp/foo/AVHR_HRP_00* .).

Expected output

I don't understand gatherer.py well enough yet to know what to expect, but I suspect it should publish a message and continue running.

Actual output

Output from gatherer.py (I added a print(metadata) in the terminator function for debugging purposes):

Output from `gatherer.py -v gatherer.ini` ``` [DEBUG: 2021-01-14 16:37:41 : gatherer] Using posttroll for default [DEBUG: 2021-01-14 16:37:41 : pytroll_collectors.trigger] Nameserver: localhost [DEBUG: 2021-01-14 16:37:41 : gatherer] Using posttroll for metopa-avhrr-hrpt [DEBUG: 2021-01-14 16:37:41 : pytroll_collectors.trigger] Nameserver: localhost [INFO: 2021-01-14 16:37:41 : posttroll.publisher] publisher started on port 33687 [INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding SUB hook tcp://localhost:16543 for topics ['pytroll://address'] [INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding address tcp://141.38.37.143:16543 with topics ['pytroll://file/poes/avhrr'] [INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding address tcp://141.38.37.143:32883 with topics ['pytroll://file/poes/avhrr'] [INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding SUB hook tcp://localhost:16543 for topics ['pytroll://address'] [INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding address tcp://141.38.37.143:16543 with topics ['pytroll://file/poes/avhrr'] [INFO: 2021-01-14 16:37:41 : posttroll.subscriber] Subscriber adding address tcp://141.38.37.143:32883 with topics ['pytroll://file/poes/avhrr'] [DEBUG: 2021-01-14 16:37:47 : pytroll_collectors.trigger] mda: {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'uid': 'AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02'} [DEBUG: 2021-01-14 16:37:47 : pytroll_collectors.region_collector] Adding area ID to metadata: eurol [INFO: 2021-01-14 16:37:47 : pytroll_collectors.region_collector] Platform name Metop-A and sensor ['avhrr/3']: Start and end times = 20210114 08:00:12 20210114 08:03:12 [DEBUG: 2021-01-14 16:37:47 : pytroll_collectors.trigger] mda: {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'uid': 'AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02'} [DEBUG: 2021-01-14 16:37:47 : pytroll_collectors.region_collector] Adding area ID to metadata: eurol [DEBUG: 2021-01-14 16:37:47 : pytroll_collectors.region_collector] Estimated granule duration to 0:03:00 [INFO: 2021-01-14 16:37:47 : pytroll_collectors.region_collector] Platform name Metop-A and sensor ['avhrr/3']: Start and end times = 20210114 08:00:12 20210114 08:03:12 [DEBUG: 2021-01-14 16:37:47 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:37:47 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:37:47 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:37:47 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:37:47 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:37:47 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:37:48 : trollsched.boundary] Instrument = avhrr/3 [INFO: 2021-01-14 16:37:49 : pytroll_collectors.region_collector] Added Metop-A (2021-01-14 08:00:12) granule to area eurol [DEBUG: 2021-01-14 16:37:49 : pytroll_collectors.region_collector] Predicting granules covering eurol [DEBUG: 2021-01-14 16:37:49 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:37:49 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:37:49 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:37:50 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:37:51 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:37:51 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:37:51 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:37:52 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:37:53 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:37:53 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:37:53 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:37:53 : trollsched.boundary] Instrument = avhrr/3 [INFO: 2021-01-14 16:37:53 : pytroll_collectors.region_collector] Added Metop-A (2021-01-14 08:00:12) granule to area eurol [DEBUG: 2021-01-14 16:37:53 : pytroll_collectors.region_collector] Predicting granules covering eurol [DEBUG: 2021-01-14 16:37:53 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:37:53 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:37:53 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:37:54 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:37:54 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:37:54 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:37:54 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:37:59 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:37:59 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:37:59 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:37:59 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:01 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:38:01 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:01 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:01 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:02 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:38:02 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:02 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:02 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:06 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:38:07 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:07 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:07 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:07 : trollsched.boundary] Instrument = avhrr/3 [INFO: 2021-01-14 16:38:08 : pytroll_collectors.region_collector] Planned granules for Euro 3.0km area - Europe: [datetime.datetime(2021, 1, 14, 7, 50, 12), datetime.datetime(2021, 1, 14, 7, 55, 12), datetime.datetime(2021, 1, 14, 8, 0, 12), datetime.datetime(2021, 1, 14, 8, 5, 12)] [INFO: 2021-01-14 16:38:08 : pytroll_collectors.region_collector] Planned timeout for Euro 3.0km area - Europe: 2021-01-14T11:10:12 [WARNING: 2021-01-14 16:38:08 : pytroll_collectors.trigger] Timeout detected, terminating collector [DEBUG: 2021-01-14 16:38:08 : pytroll_collectors.trigger] mda: {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'uid': 'AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02'} /opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() [DEBUG: 2021-01-14 16:38:08 : pytroll_collectors.trigger] Area: Area ID: eurol Description: Euro 3.0km area - Europe Projection: {'ellps': 'WGS84', 'lat_0': '90', 'lat_ts': '60', 'lon_0': '0', 'no_defs': 'None', 'proj': 'stere', 'type': 'crs', 'units': 'm', 'x_0': '0', 'y_0': '0'} Number of columns: 2560 Number of rows: 2048 Area extent: (-3780000.0, -7644000.0, 3900000.0, -1500000.0), timeout: 2021-01-14 11:10:12 [DEBUG: 2021-01-14 16:38:08 : pytroll_collectors.region_collector] Adding area ID to metadata: eurol [INFO: 2021-01-14 16:38:08 : gatherer] Composing topic. [INFO: 2021-01-14 16:38:08 : pytroll_collectors.region_collector] Platform name Metop-A and sensor ['avhrr/3']: Start and end times = 20210114 08:03:12 20210114 08:06:12 [DEBUG: 2021-01-14 16:38:08 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:08 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:08 : pyorbital.tlefile] Fetch TLE from the internet. [INFO: 2021-01-14 16:38:08 : gatherer] sending pytroll://collection/metopa/avhrr collection pytroll@oflks333.dwd.de 2021-01-14T16:38:08.450784 v1.01 application/json {"path": "", "platform_name": "Metop-A", "start_time": "2021-01-14T08:00:12", "end_time": "2021-01-14T08:03:12", "processing_time": "2021-01-14T08:00:12", "sensor": ["avhrr/3"], "orig_platform_name": "M02", "collection_area_id": "eurol", "collection": [{"start_time": "2021-01-14T08:00:12", "end_time": "2021-01-14T08:03:12", "uri": "/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z", "uid": "AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z"}]} [DEBUG: 2021-01-14 16:38:08 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:38:09 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:09 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:09 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:12 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:38:13 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:13 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:13 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:13 : trollsched.boundary] Instrument = avhrr/3 [INFO: 2021-01-14 16:38:14 : pytroll_collectors.region_collector] Added Metop-A (2021-01-14 08:03:12) granule to area eurol [DEBUG: 2021-01-14 16:38:14 : pytroll_collectors.region_collector] Predicting granules covering eurol [DEBUG: 2021-01-14 16:38:14 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:14 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:14 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:15 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:38:15 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:15 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:15 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:18 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:38:18 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:18 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:18 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:20 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:38:21 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:21 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:21 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:21 : trollsched.boundary] Instrument = avhrr/3 [INFO: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Planned granules for Euro 3.0km area - Europe: [datetime.datetime(2021, 1, 14, 7, 51, 12), datetime.datetime(2021, 1, 14, 7, 54, 12), datetime.datetime(2021, 1, 14, 7, 57, 12), datetime.datetime(2021, 1, 14, 8, 0, 12), datetime.datetime(2021, 1, 14, 8, 3, 12), datetime.datetime(2021, 1, 14, 8, 6, 12)] [INFO: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Planned timeout for Euro 3.0km area - Europe: 2021-01-14T11:09:12 [DEBUG: 2021-01-14 16:38:22 : pytroll_collectors.trigger] mda: {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'uid': 'AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02'} [DEBUG: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Adding area ID to metadata: eurol [INFO: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Added Metop-A (2021-01-14 08:03:12) granule to area eurol [DEBUG: 2021-01-14 16:38:22 : pytroll_collectors.trigger] mda: {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 6, 41), 'processing_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z', 'uid': 'AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02'} [DEBUG: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Adding area ID to metadata: eurol [INFO: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Added Metop-A (2021-01-14 08:06:12) granule to area eurol [INFO: 2021-01-14 16:38:22 : pytroll_collectors.region_collector] Adjusted timeout: 2021-01-14T11:00:12 [WARNING: 2021-01-14 16:38:22 : pytroll_collectors.trigger] Timeout detected, terminating collector /opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() [DEBUG: 2021-01-14 16:38:22 : pytroll_collectors.trigger] Area: Area ID: eurol Description: Euro 3.0km area - Europe Projection: {'ellps': 'WGS84', 'lat_0': '90', 'lat_ts': '60', 'lon_0': '0', 'no_defs': 'None', 'proj': 'stere', 'type': 'crs', 'units': 'm', 'x_0': '0', 'y_0': '0'} Number of columns: 2560 Number of rows: 2048 Area extent: (-3780000.0, -7644000.0, 3900000.0, -1500000.0), timeout: 2021-01-14 11:00:12 [INFO: 2021-01-14 16:38:22 : gatherer] Using default topic. Exception in thread Thread-2: Traceback (most recent call last): File "/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/site-packages/pytroll_collectors/trigger.py", line 141, in run self.terminator(next_timeout[0].finish(), File "/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/bin/gatherer.py", line 90, in terminator subject = "/".join(("", mda["format"], mda["data_processing_level"], KeyError: 'format' [CRITICAL: 2021-01-14 16:38:22 : gatherer] Something went wrong! [WARNING: 2021-01-14 16:38:22 : gatherer] Ending publication the gathering of granules... Setting timezone to UTC [{'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'uid': 'AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02', 'collection_area_id': 'eurol'}] /collection/metopa/avhrr [{'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 0, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'uid': 'AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02', 'collection_area_id': 'eurol'}, {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'processing_time': datetime.datetime(2021, 1, 14, 8, 3, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'uid': 'AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02', 'collection_area_id': 'eurol'}, {'path': '', 'platform_name': 'Metop-A', 'start_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'end_time': datetime.datetime(2021, 1, 14, 8, 6, 41), 'processing_time': datetime.datetime(2021, 1, 14, 8, 6, 12), 'uri': '/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z', 'uid': 'AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z', 'sensor': ['avhrr/3'], 'orig_platform_name': 'M02', 'collection_area_id': 'eurol'}] None [DEBUG: 2021-01-14 16:38:25 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:38:26 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:26 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:26 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:28 : trollsched.boundary] Instrument = avhrr/3 [DEBUG: 2021-01-14 16:38:29 : trollsched.satpass] kwargs: {'instrument': 'avhrr/3'} [DEBUG: 2021-01-14 16:38:29 : trollsched.satpass] instrument: avhrr/3 [DEBUG: 2021-01-14 16:38:29 : pyorbital.tlefile] Fetch TLE from the internet. [DEBUG: 2021-01-14 16:38:34 : trollsched.boundary] Instrument = avhrr/3 [INFO: 2021-01-14 16:38:35 : pytroll_collectors.region_collector] Planned granules for Euro 3.0km area - Europe: [datetime.datetime(2021, 1, 14, 7, 48, 12), datetime.datetime(2021, 1, 14, 7, 53, 12), datetime.datetime(2021, 1, 14, 7, 58, 12), datetime.datetime(2021, 1, 14, 8, 3, 12)] [INFO: 2021-01-14 16:38:35 : pytroll_collectors.region_collector] Planned timeout for Euro 3.0km area - Europe: 2021-01-14T11:08:12 [INFO: 2021-01-14 16:38:35 : pytroll_collectors.region_collector] Adjusted timeout: 2021-01-14T11:03:12 ```

At this point, gatherer.py quits. I'm not sure if it quits normally; it quits with exit code 0. The logging output from trollstalker in the meantime:

Logging output from trollstalker ``` [DEBUG: 2021-01-14 16:37:47,858: trollstalker] trigger: IN_CLOSE_WRITE [DEBUG: 2021-01-14 16:37:47,858: trollstalker] processing /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z [DEBUG: 2021-01-14 16:37:47,858: trollstalker] filter: {path}AVHR_HRP_00_{platform_name}_{start_time:%Y%m%d%H%M%S}Z_{end_time:%Y%m%d%H%M%S}Z_N_O_{processing_time:%Y%m%d%H%M%S}Z event: /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z [DEBUG: 2021-01-14 16:37:47,858: trollstalker] No origin_inotify_base_dir_skip_levels in self.custom_vars [DEBUG: 2021-01-14 16:37:47,859: trollstalker] Extracted: OrderedDict([('path', ''), ('platform_name', 'M02'), ('start_time', datetime.datetime(2021, 1, 14, 8, 0, 12)), ('end_time', datetime.datetime(2021, 1, 14, 8, 3, 12)), ('processing_time', datetime.datetime(2021, 1, 14, 8, 0, 12))]) [DEBUG: 2021-01-14 16:37:47,859: trollstalker] self.info['sensor']: ['avhrr/3'] [INFO: 2021-01-14 16:37:47,860: trollstalker] Publishing message pytroll://file/poes/avhrr file pytroll@oflks333.dwd.de 2021-01-14T16:37:47.859793 v1.01 application/json {"path": "", "platform_name": "Metop-A", "start_time": "2021-01-14T08:00:12", "end_t ime": "2021-01-14T08:03:12", "processing_time": "2021-01-14T08:00:12", "uri": "/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z", "uid": "AVHR_HRP_00_M02_20210114080012Z_20210114080312Z_N_O_20210114080012Z", "sensor": ["avhrr/3"], "orig_platform_name": "M02"} [DEBUG: 2021-01-14 16:37:47,943: trollstalker] trigger: IN_CLOSE_WRITE [DEBUG: 2021-01-14 16:37:47,943: trollstalker] processing /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z [DEBUG: 2021-01-14 16:37:47,943: trollstalker] filter: {path}AVHR_HRP_00_{platform_name}_{start_time:%Y%m%d%H%M%S}Z_{end_time:%Y%m%d%H%M%S}Z_N_O_{processing_time:%Y%m%d%H%M%S}Z event: /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z [DEBUG: 2021-01-14 16:37:47,944: trollstalker] No origin_inotify_base_dir_skip_levels in self.custom_vars [DEBUG: 2021-01-14 16:37:47,944: trollstalker] Extracted: OrderedDict([('path', ''), ('platform_name', 'M02'), ('start_time', datetime.datetime(2021, 1, 14, 8, 3, 12)), ('end_time', datetime.datetime(2021, 1, 14, 8, 6, 12)), ('processing_time', datetime.datetime(2021, 1, 14, 8, 3, 12))]) [DEBUG: 2021-01-14 16:37:47,944: trollstalker] self.info['sensor']: ['avhrr/3'] [INFO: 2021-01-14 16:37:47,945: trollstalker] Publishing message pytroll://file/poes/avhrr file pytroll@oflks333.dwd.de 2021-01-14T16:37:47.945067 v1.01 application/json {"path": "", "platform_name": "Metop-A", "start_time": "2021-01-14T08:03:12", "end_time": "2021-01-14T08:06:12", "processing_time": "2021-01-14T08:03:12", "uri": "/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z", "uid": "AVHR_HRP_00_M02_20210114080312Z_20210114080612Z_N_O_20210114080312Z", "sensor": ["avhrr/3"], "orig_platform_name": "M02"} [DEBUG: 2021-01-14 16:37:47,965: trollstalker] trigger: IN_CLOSE_WRITE [DEBUG: 2021-01-14 16:37:47,965: trollstalker] processing /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z [DEBUG: 2021-01-14 16:37:47,965: trollstalker] filter: {path}AVHR_HRP_00_{platform_name}_{start_time:%Y%m%d%H%M%S}Z_{end_time:%Y%m%d%H%M%S}Z_N_O_{processing_time:%Y%m%d%H%M%S}Z event: /data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z [DEBUG: 2021-01-14 16:37:47,965: trollstalker] No origin_inotify_base_dir_skip_levels in self.custom_vars [DEBUG: 2021-01-14 16:37:47,966: trollstalker] Extracted: OrderedDict([('path', ''), ('platform_name', 'M02'), ('start_time', datetime.datetime(2021, 1, 14, 8, 6, 12)), ('end_time', datetime.datetime(2021, 1, 14, 8, 6, 41)), ('processing_time', datetime.datetime(2021, 1, 14, 8, 6, 12))]) [DEBUG: 2021-01-14 16:37:47,966: trollstalker] self.info['sensor']: ['avhrr/3'] [INFO: 2021-01-14 16:37:47,966: trollstalker] Publishing message pytroll://file/poes/avhrr file pytroll@oflks333.dwd.de 2021-01-14T16:37:47.966510 v1.01 application/json {"path": "", "platform_name": "Metop-A", "start_time": "2021-01-14T08:06:12", "end_time": "2021-01-14T08:06:41", "processing_time": "2021-01-14T08:06:12", "uri": "/data/pytroll/IN/HRPT/AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z", "uid": "AVHR_HRP_00_M02_20210114080612Z_20210114080641Z_N_O_20210114080612Z", "sensor": ["avhrr/3"], "orig_platform_name": "M02"} ```

Additional diagnosis

When I add to gatherer.ini a line to the [default] section publish_topic = /collection/metopa/avhrr (duplicating the line under [metopa-avhrr-hrpt], I don't get this exception, and it does not appear to quit at all.

gerritholl commented 3 years ago

Github gave me an Error 404 when I submitted this, but apparently it was published anyway. Duplicate of #74