Add a Command Line Interface

[mraspaud]

This PR adds a command line interface (CLI) to trollflow2.

This allows for easier launching in environments that might not have posttroll messaging enabled or for batch reprocessing.

• [x] Tests added
• [x] Tests passed
• [x] Passes flake8 trollflow2
• [ ] Fully documented
• [ ] Add your name to AUTHORS.md if not there already

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 96.53465% with 7 lines in your changes missing coverage. Please review.

Project coverage is 95.93%. Comparing base (4aebf81) to head (e3c255e). Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
trollflow2/cli.py	91.48%	4 Missing :warning:
trollflow2/tests/test_cli.py	97.77%	2 Missing :warning:
trollflow2/launcher.py	97.22%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #184 +/- ## ======================================== Coverage 95.93% 95.93% ======================================== Files 12 14 +2 Lines 3098 3246 +148 ======================================== + Hits 2972 3114 +142 - Misses 126 132 +6 ``` | [Flag](https://app.codecov.io/gh/pytroll/trollflow2/pull/184/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pytroll) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/pytroll/trollflow2/pull/184/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pytroll) | `95.93% <96.53%> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pytroll#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[gerritholl]

This looks very interesting! Do you have an example of how to use this? To process old data, where we no longer have logfiles to recover a posstroll message from, we'd have to construct the JSON manually to pass this on the command line?

[mraspaud]

example usage:

satpy_cli -p pl.yaml ~/data/satellite/Meteosat-11/seviri/lvl1.5/2018/02/28/HRIT/* -m '{"start_time": "201802281500", "platform_name": "MSG4"}'

[gerritholl]

I tried

$ satpy_cli -p /home/gholl/checkouts/pytroll-dwd-config/geo-fci/trollflow2-fci-fdss-germany-classic.yaml *FDHSI*BODY---*0075_*.nc -m '{"start_time": "202306271220", "platform_name": "MTI1"}'

but this fails with

[ERROR: 2023-08-30 16:46:28 : trollflow2.launcher] Process crashed
Traceback (most recent call last):
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/launcher.py", line 396, in process_files
    process_jobs(config["workers"], jobs, produced_files)
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/launcher.py", line 440, in process_jobs
    cwrk.pop('fun')(job, **cwrk)
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/plugins/__init__.py", line 360, in save_datasets
    late_saver = save_dataset(scns, fmat, fmat_config, renames, compute=eager_writing)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/plugins/__init__.py", line 254, in save_dataset
    with prepared_filename(fmat, renames) as filename:
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/plugins/__init__.py", line 208, in prepared_filename
    directory, orig_filename = _prepare_filename_and_directory(fmat)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/plugins/__init__.py", line 182, in _prepare_filename_and_directory
    filename = os.path.join(directory, compose(fname_pattern, fmat))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollsift/parser.py", line 491, in compose
    return _strict_compose(fmt=fmt, keyvals=keyvals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollsift/parser.py", line 688, in _strict_compose
    return formatter.format(fmt, **keyvals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/string.py", line 190, in format
    return self.vformat(format_string, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/string.py", line 194, in vformat
    result, _ = self._vformat(format_string, args, kwargs, used_args, 2)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/string.py", line 247, in _vformat
    result.append(self.format_field(obj, format_spec))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/string.py", line 264, in format_field
    return format(value, format_spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Invalid format specifier '%Y%m%d_%H%M' for object of type 'str'

if I process the same files with

satpy_launcher.py -m /tmp/message /home/gholl/checkouts/pytroll-dwd-config/geo-fci/trollflow2-fci-fdss-germany-classic.yaml

then everything seems fine.

[mraspaud]

could you show us trollflow2-fci-fdss-germany-classic.yaml ?

[gerritholl]

product_list:
  output_dir:
    /data/pytroll/OUTBOXES/generic/
  reader:
    - fci_l1c_nc
  resampler: nearest
  areas:
    eurol:
      areaname: eurol
      products:
        'dwd_wv73':
          productname: 'dwd_wv73'
          formats:
            - writer: geotiff
              fname_pattern: "{platform_name}-{productname}-{areaname}-{start_time:%Y%m%d_%H%M}-{end_time:%H%M}-eager-devel-geotiff.tif"
              fill_value: 0

workers:
  - fun: !!python/name:trollflow2.plugins.create_scene
  - fun: !!python/name:trollflow2.plugins.load_composites
  - fun: !!python/name:trollflow2.plugins.resample
  - fun: !!python/name:trollflow2.plugins.save_datasets

And I know why it's happening too: normally start_time is a datetime, but by passing "202306271220" it's getting a string instead, which of course can't handle the format specification.

[gerritholl]

How would I encode a datetime object in JSON?

[mraspaud]

If you just use "{platform_name}-{productname}-{areaname}-{start_time}-{end_time}-eager-devel-geotiff.tif", it'll work...

[mraspaud]

Ok, so there is not default encoding for datetimes in json. However, in posttroll we do have a codec for datetimes that we could use here. I'll have a look.

[mraspaud]

@gerritholl @pnuu This is now ready I think