"ValueError: No reader named: olci_nc" while "olci_nc" reader was installed and exists

[sgcderek]

Describe the bug I am trying to process a Sentinel-3 OLCI .nc file with satpy. I have installed the olci_nc reader using "pip3 install satpy[olci_nc]" and I have confirmed that "olci_nc.py" indeed exists in the "readers" folder. When called upon, however, satpy returns an error saying that it can not find this reader.

To Reproduce Install the olci_nc reader, attempt to load a sentinel-3 olci nc file.

from satpy.utils import debug_on; debug_on()
from satpy.scene import Scene
scn = Scene(reader="olci_nc", filenames=["/home/derek/Documents/satimages/Oa08_radiance.nc"])

Expected behavior The scene is loaded without any error.

Actual results

/home/derek/.local/lib/python3.8/site-packages/xarray/backends/cfgrib_.py:27: UserWarning: Failed to load cfgrib - most likely there is a problem accessing the ecCodes library. Try `import cfgrib` to get the full error message
  warnings.warn(
/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import fnmatch, glob, traceback, errno, sys, atexit, locale, imp, stat
Traceback (most recent call last):
  File "satpy (copy).py3", line 5, in <module>
    scn = Scene(reader="olci_nc", filenames=["/home/derek/Documents/satimages/Oa08_radiance.nc"])
  File "/home/derek/.local/lib/python3.8/site-packages/satpy/scene.py", line 108, in __init__
    self._readers = self._create_reader_instances(filenames=filenames,
  File "/home/derek/.local/lib/python3.8/site-packages/satpy/scene.py", line 157, in _create_reader_instances
    return load_readers(filenames=filenames,
  File "/home/derek/.local/lib/python3.8/site-packages/satpy/readers/__init__.py", line 475, in load_readers
    for idx, reader_configs in enumerate(configs_for_reader(reader)):
  File "/home/derek/.local/lib/python3.8/site-packages/satpy/readers/__init__.py", line 274, in configs_for_reader
    raise ValueError("No reader named: {}".format(reader_name))
ValueError: No reader named: olci_nc

Environment Info:

• OS: Linux Mint
• Python Version: 3.8.10
• Satpy Version: 0.29.0
• PyResample Version: 1.21.0
• Readers and writers dependencies (when relevant): olci_nc

[pnuu]

The reader is named olci_l1b (or olci_l2 for the Level 2 files). There is also no extras for the reader, so just having XArray, Dask and Numpy installed should bring all the required dependencies. That is pip3 install satpy should be enough to install everything necessary.

For reading the data, you'll need to keep the full directory structure (ending with `.SEN3) intact as Satpy parses metadata from the directory name.

[sgcderek]

Thank you very much for the answer, I completely misunderstood the way OLCI data is loaded into the scene. However now I've ran into another issue that I can't quite understand. For testing purposes I have a simple script that loops between available composites after loading the OLCI data set and saves each one into an image file. But for some reason it just stops reloading the composite after it goes through a few bands. Here is the code snippet in question;

        import glob
        import os
        from satpy.scene import Scene
        import sys

        reader = "olci_l1b"
        file = "S3A_OL_1_EFR____20211102T152209_20211102T152509_20211102T171022_0179_078_125_2520_LN1_O_NR_002.SEN3"
        composites = ['Oa01', 'Oa02', 'Oa03', 'Oa04', 'Oa05', 'Oa06', 'Oa07', 'Oa08', 'Oa09', 'Oa10', 'Oa11', 'Oa12', 'Oa13', 'Oa14', 'Oa15', 'Oa16', 'Oa17', 'Oa18', 'Oa19', 'Oa20', 'Oa21']
        if reader == "olci_l1b":
            scn = Scene(reader=reader, filenames=glob.glob(file + "/*"))
        else:
            scn = Scene(reader=reader, filenames=[file])
        print(scn.available_dataset_names())
        for composite in composites:
            if os.path.isfile(file+"__"+composite+".png"):
                print("Composite " + composite + " exists, skipping")

            else:
                print("Loading composite " + composite)
                scn.load([composite])
                print("Saving composite " + composite)
                scn.save_datasets(filename=file+"__"+composite+".png")

This works fine with other readers (EPS AVHRR and MSG SEVIRI for example). It prints out the available composites correctly and when saving the first and second one it seems to work fine, but once it gets to band 3 the saved image is just a duplicate of band 2 and gets further duplicated over all the remaining bands. What's worse is that the exact same code worked just yesterday, I even dug up an old version of the script file that I'm 100% sure worked correctly and saved all the bands but running it again today makes it not work. I am pretty clueless at this point. My python/satpy/pyresample versions haven't changed since my initial issue. Any help on this would be greatly appreciated, I hope it's just something obvious that I am overlooking, but as I said the same code worked essentially just a few hours ago and now started behaving strangely without any changes to it or the source directory.

[simonrp84]

Can you try including:

from satpy.utils import debug_on
debug_on()

At the top of your script, and let us know what output that produces - it might show up what the problem is.

[djhoese]

I don't have an answer necessarily for your exact issue, but I have some ideas that may help.

There are couple things going on here that could either be done faster or in some cases your code is duplicating or overriding work. The first and most important is that you are calling scn.save_datasets on the same Scene every time. Scene.save_datasets will save every dataset that you've requested using Scene.load and that is currently contained in the Scene. So every iteration of your loop you are saving all of the previously loaded datasets. First Oa01, then the second iteration it saves Oa01 and Oa02, and so on.

Second, it is likely faster if instead of iterating over all composites, you compute which composites don't exist and then call Scene.load once with that whole list.

needed_composites = [comp_name for comp_name in composites if not os.path.isfile(file + "__" + comp_name + ".png")]
scn.load(needed_composites)
scn.save_datasets(filename=file + "__{name}.png")

Note the last line which uses Satpy's ability to format filenames for you based on metadata in the DataArrays being saved.