search

parevalo / gee-ccdc-tools

Tools and Earth Engine apps to interact with the outputs from the CCDC algorithm
MIT License
72 stars 32 forks source link

Julian and Unix times are not matching #6

Closed shahriar49 closed 2 years ago

shahriar49 commented 2 years ago

I used the CCDC API to model a stack of Landsat 7 and Landsat 8 images (sorted by system:start_time property) and generate the model coefficients and also get the Start and End time for each model. I select the model output date format to be Julian because I wanted to regenerate the model predicted data and compare it to the original Landsat band values. Surprisingly, I get Julian dates for model start and stop far from Landsat timestamps

For example, I fed CCDC with a stack of Landsat imagery from 2010-01-01 till 2020-01-01. I first ran CCDC by setting the date format to Unix timestamp, and got the first model Start time at 1271254052686 and last model End time at 1571926543636. These values match the above range, and their equivalent Julian date is 733877 to 737357 (I used your converter function to obtain these values). However, when I set the CCDC output date format to Julian date, these values are reported as 734242 and 737722, almost 1 year after the correct ones.

Could you please help me to locate the problem?

parevalo commented 2 years ago

Shahriar, thanks for bringing our attention to this. Unfortunately I won't be able to look into this any time soon, but here's a couple of comments until then:

  1. Consider using fractional time, which is easier to verify and use on the fly, and also easy to convert to and from unix time using GEE's built-in date functions (i.e. ee.Date.millis() and ee.Date.getFraction(), for a start). All our CCDC runs are in fractional years.
  2. Keep in mind that the start of a segment won't always match the start of your time series. First of all, there might not be images or clear data for the very first day of the year for some locations. Second, depending on the data density and how stable the time series are at the beginning, CCDC might not start fitting the segment right way.
shahriar49 commented 2 years ago

Thanks for your attention and reply Paulo. I followed your suggestion to set the fractional time in CCDC input parameters and now the reported start, stop, and break times match better with image dates that I get from Landsat. Then I use the fractional time to plot the observations and regenerated fitted curves (to drop the outliers out of the range of 3xrmse from fitted value) and I use the formula#3 in Zhe Zhu's original 2014 paper. But I should set time period (T) to 1 instead of 365 in formula#3 and I am a little unsure if I am doing it right. The original paper used Julian days (so T was 365) and I was thinking that the GEE's CCDC function calculates the model coefficients based on Julian days. So I am confused.

I also tried comparing model output for the same input stack and looked for example at the tBreak in the CCDC output. While milliseconds and fractional time were in good match, Julian dates were about 1-year off. It looks like a very big issue in implementation. Had anybody else noticed it?

I understand that you are busy so feel free to return back to me whenever you can.

parevalo commented 2 years ago

I finally got to look into this, and I believe it's fixed now. We simply had the wrong number of Julian days until epoch and the offset was exactly a year, which is what you reported.