Pass platform-dependent arguments to writer

gerritholl commented 3 years ago

Feature Request

Is your feature request related to a problem? Please describe.

NinJoTIFF needs the satellite ID in the header. The NinJoTIFF writer needs the satellite ID as a keyword argument. The trollflow2 save_datasets plugin needs to pass the satellite ID to the NinJoTIFF writer. The user needs to write this satellite ID in the trollflow2 configuration file. This is difficult if a single trollflow2 instance is used to process multiple satellites.

Describe the solution you'd like

I would like that in trollflow2.yaml, there is a way for the value of at arbitrary keyword argument passed to a plugin to change depending on the satellite platform. Specifically, I need the value of sat_id passed to the writer ninjotiff_nostretch to depend on the platform_name.

Some ways I can think of how this might be implemented.

Passing a callable:

product_list:
  ...
  areas:
    nqceur1km:
      areaname: nqceur1km
      products:
        '1':
          ...
          formats:
            - &ninjotiff
              format: tif
              writer: ninjotiff_nostretch
              sat_id: !!python/name:trollflow2.plugins.get_sat_id
              chan_id: ...

trollflow2 would replace the value of sat_id by the result of the callable, which it calls using the same arguments it would call the writer.

Configuring a translation table with a magic keyword:

product_list:
  ...
  keyword_translator:
    sat_id:
      platform_id:
        Metop-A: 7800014
        Metop-B: 7800015
      ...
  min_coverage: 15.0
  areas:
    nqceur1km:
      areaname: nqceur1km
      products: &chans_and_overview
        '1':
          ...
          formats:
            - &ninjotiff
              format: tif
              writer: ninjotiff_nostretch
              sat_id: !!python/name:trollflow2.sentinel
              chan_id: ...

when encountering the sentinel, trollflow2 translates the sat_id keyword.

Some other way:

There may be other ways I'm not thinking of.

Describe any changes to existing user workflow

This should be backward compatible.

Additional context

Other solutions, without changing anything in trollflow2:

Adapt pyninjotiff and do the translation there. This would require pyninjotiff to know, based on the platform name which is available from the scene metadata, what ID should be written. This adds a complication to pyninjotiff.
Adapt NinJo, so it doesn't need those IDs. In principle, all information NinJo should need is available from either the filename or the header. Although there is work going on for NinJo to understand GeoTIFF, it will be awhile until all our users use a version that can. We cannot wait for this.
Implement an additional processing step that adds the IDs. This is complicated, because by then the files would have been already written.
Use different trollflow2 instances for different satellites, with different configuration files. This would involve nearly identical configuration files. YAML does not natively support includes, so this would become a maintenance burden with nearly identical files spread around. In our case, we process the five currently operational satellites carrying AVHRR. This burden could be reduced if trollflow2 implemented its own include directive.

gerritholl commented 3 years ago

NB, current configuration with sat_id appears confusing. I'm clarifying with NinJo developers what this actually means.

gerritholl commented 3 years ago

Apparently, sat_id is not a satellite ID but can refer to a region. For now, I'm generating different satellites for the same region with the same satellite ID. Remaining repetitiveness I'm resolving with heavy use of YAML anchors. Therefore, implementing the feature requested in this issue is probably a low priority for now.

nedelceo commented 2 years ago

Hi @gerritholl, I am trying to pass start_time to decorate image with text. I tried to use your example above that is showing passing a collable. But I am getting this error:

2022-07-18 08:26:48,628 DEBG 'trollflow2_ears_viirs' stdout output:
  File "/opt/conda/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.9/site-packages/trollflow2/launcher.py", line 331, in queue_logged_process
    process(msg, prod_list, produced_files)
  File "/opt/conda/lib/python3.9/site-packages/trollflow2/launcher.py", line 375, in process
    cwrk.pop('fun')(job, **cwrk)
  File "/opt/conda/lib/python3.9/site-packages/trollflow2/plugins/__init__.py", line 283, in save_datasets
    obj = save_dataset(scns, fmat, fmat_config, renames, compute=eager_writing)
  File "/opt/conda/lib/python3.9/site-packages/trollflow2/plugins/__init__.py", line 231, in save_dataset
    obj = scns[fmat['area']].save_dataset(dsid,
  File "/opt/conda/lib/python3.9/site-packages/satpy/scene.py", line 1084, in save_dataset
    return writer.save_dataset(self[dataset_id],
  File "/opt/conda/lib/python3.9/site-packages/satpy/writers/__init__.py", line 812, in save_dataset
    img = get_enhanced_image(dataset.squeeze(), enhance=self.enhancer, overlay=overlay,
  File "/opt/conda/lib/python3.9/site-packages/satpy/writers/__init__.py", line 457, in get_enhanced_image
    img = add_decorate(img, fill_value=fill_value, **decorate)
  File "/opt/conda/lib/python3.9/site-packages/satpy/writers/__init__.py", line 393, in add_decorate
    img = add_text(img, dc, img_orig, text=dec['text'])
  File "/opt/conda/lib/python3.9/site-packages/satpy/writers/__init__.py", line 286, in add_text
    dc.add_text(**text)
  File "/opt/conda/lib/python3.9/site-packages/pydecorate/decorator_agg.py", line 45, in add_text
    self._add_text(txt, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/pydecorate/decorator_base.py", line 255, in _add_text
    txt_nl = txt.split("\n")
AttributeError: 'function' object has no attribute 'split'

Here is par of my PL:

        M07_sun_corrected:
          productname: M07_sun_corrected
          formats:
            - format: png
              writer: simple_image
              decorate:  
                decorate: 
                  - text:
                      txt: !!python/name:trollflow2.plugins._get_start_time
                      align: 
                        top_bottom: top
                        left_right: right
                      font_size : 72
                      height: 100
                      bg_opacity: 127
                      bg: black
                      line: white

I' ve so far tried few versions of _get_start_time function in Trollflow2.plugins

def _get_start_time(job):
    scn_mda = _get_scene_metadata(job)
    return scn_mda['start_time']

def _get_start_time(start_time):
    return start_time

def _get_start_time(job):
    return 'text'

But I am still getting the same error. It seems that I am not getting the return value of my function.

Can you please show example of how would trollflow2.plugins.get_sat_id that you mentioned look like?

mraspaud commented 2 years ago

@nedelceo would you mind creating a new issue for this? I feel it belongs in it's own discussion thread :)

nedelceo commented 2 years ago

I am sorry I forget about this problem for a while. I created #161

pytroll / trollflow2

Pass platform-dependent arguments to writer #107

Feature Request