search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.07k stars 295 forks source link

Trying to add VIIRS L2 data product to viirs_l2 reader and getting errors #2865

Open agp-earth opened 3 months ago

agp-earth commented 3 months ago

Describe the bug I am trying to add the "VIIRS/NPP Land Surface Temperature and Emissivity 6-Min L2 Swath 750m v002" or "VNP21" product from NASA to the viirs_l2 reader in the yaml file. This file comes in nc format like the other examples. I copy the same format as the pre-existing yaml file and just add the new product details. When I try to run scene with the file I get:

ValueError: No supported files found

To Reproduce What I added to the yaml file:

file_types:
  lst_l2_viirs:
    file_reader: !!python/name:satpy.readers.viirs_l2.VIIRSL2FileHandler
    file_patterns:
      - 'V{spacecraft_name:2s}21.A{start_time:%Y%j.%H%M}.{collection_number:03d}.{production_time:%Y%j%H%M%S}.nc'
      # filename: VNP21.A2018030.0512.002.2024122190830.nc

datasets:
  lst_lon:
    name: lst_lon
    resolution: 750
    file_type: [lst_l2_viirs]
    file_key: geolocation_data/longitude
    units: degrees
    standard_name: longitude
  lst_lat:
    name: lst_lat
    resolution: 750
    file_type: [lst_l2_viirs]
    file_key: geolocation_data/latitude
    units: degrees
    standard_name: latitude

##################################
# Datasets in file lst_l2_viirs
##################################
  Land_Surface_Temperature:
    name: Land_Surface_Temperature
    long_name: VIIRS NPP Land Surface Temperature Emissivity
    units: "K"
    coordinates: [lst_lon, lst_lat]
    file_key: geophysical_data/Land_Surface_Temperature
    file_type: lst_l2_viirs
    standard_name: land_surface_temperature

And then in a separate python file trying to run the code:

scn = Scene(filenames=["VNP21.A2018030.0512.002.2024122190830.nc"], reader='viirs_l2')

Actual results I get "ValueError: No supported files found"

I am probably missing something pretty straightforward here but any advice or suggestions would be super appreciated! Thanks!

djhoese commented 3 months ago

Try adding debug logging and see if it gives any more clues:

from satpy.utils import debug_on
debug_on()

# your code

By skimming your YAML you've pasted it looks correct. And you added to the YAML file in Satpy, right? That is, not a custom file of your own (that's fine, but I want to make sure I understand).

agp-earth commented 3 months ago

Thank you! Yes, I just added this to the YAML file in Satpy. I get the following message with the debug:

[DEBUG: 2024-08-01 09:24:46 : satpy.readers] while constructing a Python object
cannot find 'VIIRSL2FileHandler' in the module 'satpy.readers.viirs_l2'

Although , at the beginning of my viirs_l2.py file I have this defined (as I made sure my file looked just like the file on github):

class VIIRSL2FileHandler(NetCDF4FileHandler):
    """NetCDF File Handler for VIIRS L2 Products."""
    def _parse_datetime(self, datestr):
        """Parse datetime."""
        return dt.datetime.strptime(datestr, "%Y-%m-%dT%H:%M:%S.000Z")
djhoese commented 3 months ago

Could you describe your environment/setup? You modified the YAML in your satpy installation, but did you also modify the python code in your satpy installation? Or did you make your own python modules? How was satpy installed?

This might not be the issue, as I'd expect another error above the one you got in your log, but do you have the netcdf4 python library installed in your python environment?

agp-earth commented 3 months ago

I believe I had an older install so I just made a new conda environment with the command: conda create -c conda-forge -n my_satpy_env python satpy

I updated the files again to match what I had before and this helped a little! Now I am getting a new error: KeyError: '/attr/time_coverage_start'

And my debug says this:

[DEBUG: 2024-08-01 10:30:13 : satpy.readers.yaml_reader] Reading ('/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/etc/readers/viirs_l2.yaml',)
[DEBUG: 2024-08-01 10:30:13 : satpy.readers.yaml_reader] Assigning to viirs_l2: ['/Users/name/Desktop/VNP21.A2018032.0612.002.2022262211952.nc']
djhoese commented 3 months ago

So it looks like its working but your files don't have the global attribute time_coverage_start. Is there a different global attribute or something that contains observation start time information?

agp-earth commented 3 months ago

Yes! If I open the file in xarray there is a StartTime attribute. Should I change the attribute it searches for in the viirs_l2.py file?

    @property
    def start_time(self):
        """Get start time."""
        return self._parse_datetime(self["/attr/time_coverage_start"])
djhoese commented 3 months ago

If it has the same date/time format then you could probably change it to:

    @property
    def start_time(self):
        """Get start time."""
        time_str = self["/attr/time_coverage_start"] if "/attr/time_coverage_start" in self else self["/attr/StartTime"]
        return self._parse_datetime(time_str)

You'd have to do something similar for end time probably. You could also just subclass the VIIRS L2 handler and customize the start_time and end_time properties. You'd have to point your YAML file_type to this subclass too.

agp-earth commented 3 months ago

Okay! I believe they are the same string format ('StartTime': '2018-02-01 06:12:00.000') and I've updated start and end times in the python file:

    @property
    def start_time(self):
        """Get start time."""
        time_str = self["/attr/time_coverage_start"] if "/attr/time_coverage_start" in self else self["/attr/StartTime"]
        return self._parse_datetime(time_str)

    @property
    def start_time(self):
        """Get start time."""
        time_str = self["/attr/time_coverage_end"] if "/attr/time_coverage_end" in self else self["/attr/EndTime"]
        return self._parse_datetime(time_str)

But I am still getting the same error: KeyError: '/attr/time_coverage_start'

djhoese commented 3 months ago

Do you have the full traceback for that error? Are you sure the python code you're modifying is the same python code that is being run?

agp-earth commented 3 months ago

I was updating the correct files but it didn't seem to be running the updates so I closed out of VS code and opened it again and it is now reading the updated files!

I am now getting: ValueError: No dataset could be loaded. Either missing requirements (such as Epilog, Prolog) or none of the provided files match the filter parameters.

I believe this is because the nc file I am loading in has its variables stored within a group:

import xarray as xr
import netCDF4

nc = netCDF4.Dataset("/Users/name/Desktop/VNP21.A2018032.0612.002.2022262211952.nc")
print('GROUPS: ', nc.groups)

Which gives:

GROUPS:  {'VIIRS_Swath_LSTE': <class 'netCDF4._netCDF4.Group'>
group [/VIIRS_Swath_LSTE](https://file+.vscode-resource.vscode-cdn.net/VIIRS_Swath_LSTE):
    dimensions(sizes): 
    variables(dimensions): 
    groups: Data Fields, Geolocation Fields, Swath Attributes}

Where the temperature data (LST) is in a variable within Data Fields and the lat/lon are coordinates within the Geolocation Fields.

ds = xr.open_dataset("/Users/name/Desktop/VNP21.A2018032.0612.002.2022262211952.nc", group = 'VIIRS_Swath_LSTE/Data Fields')
print(ds)
<xarray.Dataset> Size: 1GB
Dimensions:      (number_of_lines: 3248, number_of_pixels: 3200)
Dimensions without coordinates: number_of_lines, number_of_pixels
Data variables: (12/13)
    Emis_14      (number_of_lines, number_of_pixels) float64 83MB ...
    Emis_14_err  (number_of_lines, number_of_pixels) float64 83MB ...
    Emis_15      (number_of_lines, number_of_pixels) float64 83MB ...
    Emis_15_err  (number_of_lines, number_of_pixels) float64 83MB ...
    Emis_16      (number_of_lines, number_of_pixels) float64 83MB ...
    Emis_16_err  (number_of_lines, number_of_pixels) float64 83MB ...
    ...           ...
    LST          (number_of_lines, number_of_pixels) float64 83MB ...
    LST_err      (number_of_lines, number_of_pixels) float64 83MB ...
    PWV          (number_of_lines, number_of_pixels) float64 83MB ...
    QC           (number_of_lines, number_of_pixels) float64 83MB ...
    View_angle   (number_of_lines, number_of_pixels) float64 83MB ...
    oceanpix     (number_of_lines, number_of_pixels) float64 83MB ...

I think this is where this file type might differ from the other L2 files in the YAML file. Do you think there is a work around for this issue? Sorry for all of the simple questions and thank you again for your help!

djhoese commented 3 months ago

If you update the file_key in your YAML entries to match the grouping in the file then you might be OK. So file_key: "VIIRS_Swath_LSTE/Data Fields/LST". I'd also suggest, just for consistency with other readers and products, to name your products to match the variable name in the file. So "LST" in this case instead of "Land_Surface_Temperature". You'll also need to update the lon/lat file_keys to match the file grouping.

agp-earth commented 3 months ago

Awesome, thanks! I have updated the YAML file:

datasets:
  lst_lon:
    name: lst_lon
    resolution: 750
    file_type: [lst_l2_viirs]
    file_key: VIIRS_Swath_LSTE/Geolocation Fields/longitude
    units: degrees
    standard_name: longitude
  lst_lat:
    name: lst_lat
    resolution: 750
    file_type: [lst_l2_viirs]
    file_key: VIIRS_Swath_LSTE/Geolocation Fields/latitude
    units: degrees
    standard_name: latitude

##################################
# Datasets in file lst_l2_viirs
##################################
  Land_Surface_Temperature:
    name: LST
    long_name: VIIRS NPP Land Surface Temperature Emissivity
    units: "K"
    coordinates: [lst_lon, lst_lat]
    file_key: VIIRS_Swath_LSTE/Data Fields/LST
    file_type: lst_l2_viirs
    standard_name: land_surface_temperature

And I am able to read in the file! But, I believe that it is still not reading in the variables correctly. When I try to load the LST layer I get the following error: scn.load(['LST'])

[WARNING: 2024-08-01 12:09:58 : satpy.readers.yaml_reader] Failed to load DataID(name='lst_lat', resolution=750, modifiers=()) from <VIIRSL2FileHandler: '/Users/name/Desktop/VNP21.A2018032.0612.002.2022262211952.nc'>
Traceback (most recent call last):
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 712, in _load_dataset
    projectable = fh.get_dataset(dsid, ds_info)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/viirs_l2.py", line 153, in get_dataset
    metadata = self.get_metadata(ds_id, ds_info)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/viirs_l2.py", line 113, in get_metadata
    "sensor": self.sensor_name,
              ^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/viirs_l2.py", line 78, in sensor_name
    return self["/attr/instrument"].lower()
           ~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/netcdf_utils.py", line 273, in __getitem__
    val = self.file_content[key]
          ~~~~~~~~~~~~~~~~~^^^^^
KeyError: '/attr/instrument'
[ERROR: 2024-08-01 12:09:58 : satpy.readers.yaml_reader] Could not load dataset 'DataID(name='lst_lat', resolution=750, modifiers=())': "Could not load DataID(name='lst_lat', resolution=750, modifiers=()) from any provided files"
Traceback (most recent call last):
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 836, in _load_dataset_with_area
    ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 736, in _load_dataset_data
    proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 721, in _load_dataset
    raise KeyError(
KeyError: "Could not load DataID(name='lst_lat', resolution=750, modifiers=()) from any provided files"
[WARNING: 2024-08-01 12:09:58 : satpy.readers.yaml_reader] Failed to load DataID(name='lst_lon', resolution=750, modifiers=()) from <VIIRSL2FileHandler: '/Users/name/Desktop/VNP21.A2018032.0612.002.2022262211952.nc'>
Traceback (most recent call last):
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 712, in _load_dataset
    projectable = fh.get_dataset(dsid, ds_info)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/viirs_l2.py", line 153, in get_dataset
    metadata = self.get_metadata(ds_id, ds_info)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/viirs_l2.py", line 113, in get_metadata
    "sensor": self.sensor_name,
              ^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/viirs_l2.py", line 78, in sensor_name
    return self["/attr/instrument"].lower()
           ~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/netcdf_utils.py", line 273, in __getitem__
    val = self.file_content[key]
          ~~~~~~~~~~~~~~~~~^^^^^
KeyError: '/attr/instrument'
[ERROR: 2024-08-01 12:09:58 : satpy.readers.yaml_reader] Could not load dataset 'DataID(name='lst_lon', resolution=750, modifiers=())': "Could not load DataID(name='lst_lon', resolution=750, modifiers=()) from any provided files"
Traceback (most recent call last):
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 836, in _load_dataset_with_area
    ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 736, in _load_dataset_data
    proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 721, in _load_dataset
    raise KeyError(
KeyError: "Could not load DataID(name='lst_lon', resolution=750, modifiers=()) from any provided files"
[WARNING: 2024-08-01 12:09:58 : satpy.readers.yaml_reader] Failed to load DataID(name='LST', modifiers=()) from <VIIRSL2FileHandler: '/Users/name/Desktop/VNP21.A2018032.0612.002.2022262211952.nc'>
Traceback (most recent call last):
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 712, in _load_dataset
    projectable = fh.get_dataset(dsid, ds_info)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/viirs_l2.py", line 153, in get_dataset
    metadata = self.get_metadata(ds_id, ds_info)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/viirs_l2.py", line 113, in get_metadata
    "sensor": self.sensor_name,
              ^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/viirs_l2.py", line 78, in sensor_name
    return self["/attr/instrument"].lower()
           ~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/netcdf_utils.py", line 273, in __getitem__
    val = self.file_content[key]
          ~~~~~~~~~~~~~~~~~^^^^^
KeyError: '/attr/instrument'
[ERROR: 2024-08-01 12:09:58 : satpy.readers.yaml_reader] Could not load dataset 'DataID(name='LST', modifiers=())': "Could not load DataID(name='LST', modifiers=()) from any provided files"
Traceback (most recent call last):
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 836, in _load_dataset_with_area
    ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 736, in _load_dataset_data
    proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/name/miniconda3/envs/my_satpy_env/lib/python3.12/site-packages/satpy/readers/yaml_reader.py", line 721, in _load_dataset
    raise KeyError(
KeyError: "Could not load DataID(name='LST', modifiers=()) from any provided files"
[WARNING: 2024-08-01 12:09:58 : satpy.scene] The following datasets were not created and may require resampling to be generated: DataID(name='LST', modifiers=())
djhoese commented 3 months ago

Yeah it seems like your files are just too different. It may be best to create a subclass (or whole new class) of the VIIRS L2 file handler and update all attribute accesses to the names in your files. That error you posted says your files don't have /attr/instrument (the global "instrument" attribute).

agp-earth commented 3 months ago

Sounds good, I can do that! Thanks so much for your help!!