search

pytroll / pytroll-collectors

Collector modules for Pytroll
GNU General Public License v3.0
3 stars 18 forks source link

gatherer mixing satellite granules if timeout isn't reached #69

Open mraspaud opened 3 years ago

mraspaud commented 3 years ago

When gatherer is waiting for granules from eg npp, and noaa 20 granules are comming in, they are collected together with the npp granules.

pnuu commented 3 years ago

My solution has been to use different topics for each satellite: /ears/viirs/noaa-20/segment and /ears/viirs/suomi-npp/segment. Works nicely.

mraspaud commented 3 years ago

yes, but it feels like a workaround, doesn't it?

pnuu commented 3 years ago

Maybe a little. This is the way I'd plan my chains in any case. Now all the messages are more explicit having different topics for different data.

gerritholl commented 3 years ago

I think this is related so I'm adding it here rather than as a new issue. I have a Metop-B granule from 2021-02-05 07:04:03 which is added to an Antarctica region and a Metop-C granule from 2021-02-05 07:10:03 — exactly six minutes later — over the Arctica region. My granule duration is three minutes. For subsequent granules at 07:13:03, 07:16:03 etc., rather than adding the Metop-B granule to Antarctica and Metop-C to Arctica, it's adding either Metop-B or Metop-C to both Arctica and Antarctica, and the other to neither. Extract from my logfiles:

[DEBUG: 2021-02-05 07:30:45 : pytroll_collectors.region_collector] Granule /data/pytroll/IN/EMC_AVR/AVHR_xxx_1B_M01_20210205070403Z_20210205070703Z_N_O_20210205072550Z is overlapping region spantarktis3km by fraction 0.00396
[INFO: 2021-02-05 07:30:45 : pytroll_collectors.region_collector] Added Metop-B (2021-02-05 07:04:03) granule to area spantarktis3km
[DEBUG: 2021-02-05 07:30:45 : pytroll_collectors.region_collector] Predicting granules covering spantarktis3km
[INFO: 2021-02-05 07:31:24 : pytroll_collectors.region_collector] Planned granules for spantarktis3km: [datetime.datetime(2021, 2, 5, 7, 4, 3), datetime.datetime(2021, 2, 5, 7, 7, 3), datetime.datetime(2021, 2, 5, 7, 10, 3), datetime.datetime(2021, 2, 5, 7, 13, 3), datetime.datetime(2021, 2, 5, 7, 16, 3), datetime.datetime(2021, 2, 5, 7, 19, 3), datetime.datetime(2021, 2, 5, 7, 22, 3), datetime.datetime(2021, 2, 5, 7, 25, 3)]
[DEBUG: 2021-02-05 07:51:54 : pytroll_collectors.region_collector] Granule /data/pytroll/IN/EMC_AVR/AVHR_xxx_1B_M03_20210205071003Z_20210205071303Z_N_O_20210205074715Z is overlapping region nparktis1km by fraction 0.00067
[INFO: 2021-02-05 07:51:54 : pytroll_collectors.region_collector] Added Metop-C (2021-02-05 07:10:03) granule to area nparktis1km
[DEBUG: 2021-02-05 07:51:54 : pytroll_collectors.region_collector] Predicting granules covering nparktis1km
[[INFO: 2021-02-05 07:52:16 : pytroll_collectors.region_collector] Planned granules for nparktis1km: [datetime.datetime(2021, 2, 5, 7, 10, 3), datetime.datetime(2021, 2, 5, 7, 13, 3), datetime.datetime(2021, 2, 5, 7, 16, 3), datetime.datetime(2021, 2, 5, 7, 19, 3), datetime.datetime(2021, 2, 5, 7, 22, 3), datetime.datetime(2021, 2, 5, 7, 25, 3)]
[INFO: 2021-02-05 07:52:31 : pytroll_collectors.region_collector] Added Metop-C (2021-02-05 07:16:03) granule to area nparktis1km
[INFO: 2021-02-05 07:52:31 : pytroll_collectors.region_collector] Added Metop-C (2021-02-05 07:16:03) granule to area spantarktis3km
[INFO: 2021-02-05 07:52:34 : pytroll_collectors.region_collector] Added Metop-C (2021-02-05 07:19:03) granule to area nparktis1km
[INFO: 2021-02-05 07:52:34 : pytroll_collectors.region_collector] Added Metop-C (2021-02-05 07:19:03) granule to area spantarktis3km
[INFO: 2021-02-05 08:18:16 : pytroll_collectors.region_collector] Added Metop-B (2021-02-05 07:22:03) granule to area nparktis1km
[INFO: 2021-02-05 08:18:16 : pytroll_collectors.region_collector] Added Metop-B (2021-02-05 07:22:03) granule to area spantarktis3km

To diagnose this more easily I used additional logging from #77.