lsat_general_prep() year turning to NAs

[Julie-Pascale]

Hi !

I am currently cleaning the landsat data that I exported with the package. When I look at the file that I exported, I can see dates in the "DATE_ACQUIRED" column. However, after runing it in lsat_general_prep(), every row of the column "year" and "doy" turned into NAs.

Do you have any ideas on how to fix this issue ?
Thanks !

[jakobjassmann]

I am sorry to hear, this is certainly unusual. I just did a test run to make sure nothing has changed on the GEE side of things, but everything works fine for me. Could you provide us with a bit more information and a reproducible example? Specifically, could you share your time-series exports (CSV file) with us and send us the code that you use to load the CSV file?

[Julie-Pascale]

Hi Jakob, Here is the CSV containing the time-series that I exported: lsatTS_export_Dempster.csv

Here is the code that I am using: lsat_demp<-read.csv("D:/ShrubHub/NDVI/lsatTS_export_Dempster.csv") lsat_demp<-lsat_general_prep(lsat_demp) lsat_demp <- lsat_clean_data(lsat_demp, geom.max = 15, cloud.max = 80, sza.max = 60, filter.cfmask.snow = T, filter.cfmask.water = T, filter.jrc.water = T)

It is only after the "lsat_general_prep()" that the date turns into NAs

[jakobjassmann]

You will have to read in the CSV as a data.table object from the data.table package using data.table::fread() instead of a data.frame object using base::read.csv(). You can read about that in the manuscript on p.7, https://github.com/logan-berner/lsatTS#3-clean-and-cross-calibrate-Landsat-data and in the help files ?lsat_general_prep.

If I use read.csv(), I get the same behaviour that you observe, but it works fine if I use fread() and the file you provided. Does this solve your problem? 🙂

[Julie-Pascale]

Yes it is working fine now ! I am sorry I don't know how I missed this info.

Thanks for your help, again :)

[jakobjassmann]

No worries, Julie! Easy to skim over that.

@logan-berner Perhaps you could make the function break if it is not given a data.table object - just to be safe?

# Check if dt is a data.table - if not stop
if(class(dt)[1] != "data.table") stop("dt is not a data.table object!\nHave you supplied a data.frame or tibble instead?") 

[logan-berner]

@Julie-Pascale thanks for bringing this to our attention! @jakobjassmann, the first step of the function actually type casts the input as a data.table: dt <- data.table::data.table(dt)

It looks like the problem stems from differences in how data.table::fread() and base::read.csv() treat the date column. While read.csv() treats the column as a character string, fread() instead treats it as an IDate data structure, which is evidently a data structure from data.table. Anyways, I fixed the problem by adding this line of code when parsing the date: dt[, date.acquired := data.table::as.IDate(date.acquired)] # incase file read using read.csv

The whole function now appears to properly work whether the file is initially read with read.csv() or fread(). The update has been pushed to the repo.