search

samapriya / awesome-gee-community-datasets

Community Datasets added by users and made available for use at large
https://gee-community-catalog.org/
Creative Commons Attribution 4.0 International
756 stars 120 forks source link

HISTARFM database #115

Closed EIzquierdo closed 1 year ago

EIzquierdo commented 1 year ago

The HISTARFM database is a high spatial resolution monthly reflectance temporal series corrected from cloud data gaps. The dataset was created at 30 meters resolution through the fusion of the Landsat and Moderate Resolution Imaging Spectroradiometer (MODIS) temporal series. The method involves using two estimators that work together to eliminate random noise and minimize the bias of Landsat spectral reflectances. The first estimator is an optimal interpolator that generates Landsat reflectance estimates using Landsat historical data and fused MODIS and Landsat reflectances from the nearest overpasses. The fusion process employs a linear regression model at a pixel level. The second estimator is a Kalman filter that corrects any bias in the reflectance produced by the first estimator. HISTARFM provides improved reflectance values and a unique and useful side product, the reflectance uncertainties, which is helpful for realistic error calculation (e.g., computing error bars of Vegetation Indexes or biophysical variables). For a more detailed explanation of the HISTARFM algorithm, please refer to the Moreno-Martinez et al. 2020 manuscript.

Example of a mosaic of the HISTARFM data [right bottom], the red band uncertainty [left bottom], and the derivate leaf area index [top] over a large area in continental East Asia for a given year (2021).

Earth Engine Snippet if dataset already in GEE

Different versions and study areas are already processes:

  1. The CONUS database contains 154 images stored as assets. It corresponds with version 2, and temporal coverage ranges from January 2009 to October 2021. Each image in the ImageCollection covers the full CONUS, and each has the properties 'version', 'month', and 'year'. This information is also present in their file names. For example, the image called Gap_Filled_Landsat_CONUS_month_10_2009_v2 is an October 2009 image for the CONUS area. The CONUS database is available in this asset, and the images are loaded in Earth Engine as follows: 
    
var histarfm_conus = ee.ImageCollection("projects/KalmanGFwork/GFLandsat_V1")

2. The European, main part of east Asia and Somalia databases are currently being generated with **version 5** of the algorithm. Version 5 contains 26,916 images. Europe contains nine years from 2013 to 2021, east Asia contains three years from 2019 to 2021, and Somalia contains five years from 2010 to 2014. All the study areas are divided into tiles stored in the Google Cloud Plattform as Cloud Optimized Geotiffs. The name of the image includes the month, year, specific study area, and tile. As an example, the image called GF_2018_10_EUROPA_1 represents the image in October 2018 over the first tile in Europe. Version 5 of the database is available [here](https://code.earthengine.google.com/?asset=projects/ee-gapfilled/histarfm_v5), and the images can be loaded in Earth Engine using the following code:

```js

var histarfm_ic = ee.ImageCollection("projects/ee-gapfilled/histarfm_v5")

For more information about how to work with HISTARFM and some examples of how to improve your research and applications with the HISTARFM database, visit the tutorial here.

Extra Info: The HISTARFM algorithm is described in the following paper:

The HISTARFM database was used in the following papers:

Required

License Information: Copyright (c) 2023 Emma Izquierdo-Verdiguier and Alvaro Moreno-Martinez

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Curated by: Álvaro Moreno-Martínez, Emma Izquierdo-Verdiguier, Jordi Muñoz-Marí and Nicolas Clinton.

Keywords: Land reflectance images, gap-filled temporal series, vegetation.

Last updated: 06-March-2023.

samapriya commented 1 year ago

Okay so I am part of the google group now and while I can access var histarfm_conus at projects/KalmanGFwork/GFLandsat_V1 I cannot access projects/ee-gapfilled/histarfm_v5 is there a separate google group for this? I am confused why they would behave differently otherwise

samapriya commented 1 year ago

I also there are some additional collections like projects/ee-emma/assets/GF_Landsat_Europe_C2 and projects/ee-emma/assets/GF_Landsat_China_C2_v1 are these what were combined to create projects/ee-gapfilled/histarfm_v5?

EIzquierdo commented 1 year ago

Okay so I am part of the google group now and while I can access var histarfm_conus at projects/KalmanGFwork/GFLandsat_V1 I cannot access projects/ee-gapfilled/histarfm_v5 is there a separate google group for this? I am confused why they would behave differently otherwise

Ups! Sorry, I forgot to share it. Now, it is shared. This one, projects/KalmanGFwork/GFLandsat_V1, contains the CONUS version 2. The images are not in the cloud, they are in the assets. Once we will update to version 5 or later, we will include in projects/ee-gapfilled/histarfm_v5.

EIzquierdo commented 1 year ago

I also there are some additional collections like projects/ee-emma/assets/GF_Landsat_Europe_C2 and projects/ee-emma/assets/GF_Landsat_China_C2_v1 are these what were combined to create projects/ee-gapfilled/histarfm_v5?

Yes, you are right. All the images in the cloud were in one project (ee-emma) in different IC. To avoid problems with the users, I generated a new project (ee-gapfilled) and only one image collection (histarfm_v5).

samapriya commented 1 year ago

Just had discussion correct path is 

var collection = ee.ImageCollection('projects/ee-kalman-gap-filled/assets/histarfm_v5')
samapriya commented 1 year ago

This has finally been completed https://gee-community-catalog.org/projects/histarfm/

Thank you for contributing :)

053L commented 10 months ago

The author, how are you, I can not access https://code.earthengine.google.com/?asset=projects/KalmanGFwork/GFLandsat_V1 now, is it no longer shared?

EIzquierdo commented 10 months ago

I just checked, and everything works well. Can you provide the code?

samapriya commented 10 months ago

Since this is a closed issue I am creating a new discussion section to discuss this further if you would like

@EIzquierdo and @053L you can find the discussion here

053L commented 10 months ago

I just checked, and everything works well. Can you provide the code?

I accessed the dataset and was Unable to load details about asset "projects/ee-gapfilled/histarfm_v5".This dataset does not cover my research area, can you provide the source code of HISTARFM algorithm? thank you!