search

SM24-Industrial-Software-Dev / ML-forecasting-NOx-levels

0 stars 0 forks source link

Data Cleaning from Sentinel 5-P data #11

Open rameshnatarajanus opened 5 days ago

rameshnatarajanus commented 5 days ago

Implement a data cleaning module for Sentinel 5-P data

  1. Negative mean concentrations to be thresholded (should this filtered at the pixel level or after mean statistics are obtained using reduceRegions?)
  2. What is the impact of changing CLOUD_MASK_FRACTION on the image quality and on the number of valid daily time series data points
  3. Issue identified by Aaron below: 
    There still seems to be an issue when i span 2019 until 2021. It seems that sometimes the cloud masking results in a non-image object somehow, since when I add the following piece of code into the maskClouds(image) function:
if isinstance(masked_image, ee.Image):#rest of code
else: #retun the original image and print("Masking resulted in a non-image object. Returning original image")
it sometimes fails. Not sure why this is happening, but it seems to be only 9 problematic images in the entire dataset.  
Also when i set the end date year to 2022 it still fails and I cant get it working even with the above code.
  4. Issue identified by Aaron: Why is there no data past 2021?