"Don't know how to open the following files" ERROR in MTG-I1 LI data.

[GeorgeMJ23]

In the Real Time Lightning Data Visualisation with Python article of Medium I'm trying to run the code in there, but it gives me an error related to satpy.

The code takes MTG-I1 LI data for some period of time from EUMETSAT in .nc form and downloads them and then draws them on a map. But something is wrong and satpy doesn't recognize the data.

The code is: (you must have some EUMETSAT credentials(keys) for this to work) (I did not give all the code, but only all the code up to the point it gave the error, as the remaining code is irrelevant.)

import eumdac
import datetime
import shutil
import requests

# Insert your personal key and secret into the single quotes
consumer_key = 'some key'
consumer_secret = 'some secret key'

credentials = (consumer_key, consumer_secret)

token = eumdac.AccessToken(credentials)
print(f"This token '{token}' expires {token.expiration}")

datastore = eumdac.DataStore(token)

import os
import zipfile
from io import BytesIO
from concurrent.futures import ThreadPoolExecutor, as_completed

# Assume datastore is already defined and connected
searchs = datastore.opensearch("""pi=EO:EUM:DAT:0691&dtstart=2024-07-30T22:00:00&dtend=2024-07-31T22:00:00""")

# Create the data directory if it doesn't exist
os.makedirs("point_data", exist_ok=True)

def download_and_extract(product):
    try:
        # Create a temporary BytesIO object to hold the downloaded zip file
        with product.open() as fsrc:
            with BytesIO(fsrc.read()) as bio:
                with zipfile.ZipFile(bio, 'r') as zip_ref:
                    # Check if all the files to be extracted already exist
                    all_files_exist = all(os.path.exists(os.path.join("point_data", name)) for name in zip_ref.namelist())
                    if all_files_exist:
                        print(f"All files already exist, skipping download for product: {product}")
                        return
                    # Extract all the contents into the data directory
                    zip_ref.extractall("point_data")
                    print(f'Extracted product finished.')
    except Exception as e:
        print(f"Failed to process product: {e}")

# Use ThreadPoolExecutor to handle concurrency
with ThreadPoolExecutor(max_workers=5) as executor:
    futures = {executor.submit(download_and_extract, product): product for product in searchs}
    for future in as_completed(futures):
        product = futures[future]
        try:
            future.result()
        except Exception as e:
            print(f"Exception occurred while processing product: {e}")
print("All products processed.")

from glob import glob
import os

import cartopy.crs as ccrs
import cartopy.feature as cfeature
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from satpy import Scene, MultiScene
import matplotlib.patheffects as path_effects

scn = Scene(filenames=glob("point_data/*001.nc"), reader="li_l2_nc")
scn.load(scn.available_dataset_names())

ds = scn.to_xarray_dataset()
df = ds.to_dataframe()

# Get longitude and latitude from metadata
df["lon"], df["lat"] = ds["number_of_events"].attrs["area"].get_lonlats()

# Drop columns that we will not need
df = df.drop(columns=["flash_filter_confidence"])

# Drop null values (if any)
df.dropna(inplace=True)

# Convert flash_time to datetime
df["flash_time"] = pd.to_datetime(df["flash_time"])
df = df[["flash_time", "lon", "lat"]]
print(df.head())

And now it returns the error in terminal:

Don't know how to open the following files: {'point_data\W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+LI-2-LFL--FD--CHK-BODY--ARC-NC4E_C_EUMT_20240731091020_L2PF_OPE_20240731090007_20240731091007_NT_0 055_0001.nc', 'point_data\W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+LI-2-LFL--FD--CHK-BODY--ARC-NC4E_C_EUMT_20240731160022_L2PF_OPE_20240731155007_20240731160007_NT_0096_0001.nc' etc... listing all the other files in this folder that are valid .nc files I can open with other programs.

With Visual Studio Code GUI giving the error:

No supported files found
  File "C:\Users\GeorgeMJ23\Documents\TestSAT24LI.py", line 76, in <module>
    scn = Scene(filenames=glob("point_data/*001.nc"), reader="li_l2_nc")

What is wrong here?

[gerritholl]

You might miss a dependency. Try running

from satpy.utils import check_satpy
check_satpy()

and/or rerun the whole script while adding

from satpy.utils import debug_on
debug_on()

somewhere near the top

[ameraner]

Hi @GeorgeMJ23 I just tried with the indicated files e.g. scn = Scene(filenames=["W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+LI-2-LFL--FD--CHK-BODY--ARC-NC4E_C_EUMT_20240731091020_L2PF_OPE_20240731090007_20240731091007_N__T_0055_0001.nc"], reader="li_l2_nc") and it worked for me, so I also suspect an issue with missing dependencies, let us know if you find out more with Gerrit's suggestions :)

[ameraner]

Hi @GeorgeMJ23 , I'm closing this issue as it is likely related to your conda env/missing dependencies. Feel free to reopen if you have further issues!