Extract 1/3 Hz data

[dlebauer]

@dlebauer commented on Thu Jun 01 2017

code to extract 1/3 hz spectral data; need to change date

library(dplyr)
library(readr)
load_loggerfiles <- function(date){
  path <- file.path(“/data/terraref/sites/ua-mac/raw_data/EnvironmentLogger”, date)
  files <- dir(path, full.names = TRUE)
  metdata <- lapply(files, jsonlite::fromJSON)  
  return(metdata)
}

loggerdata <- load_loggerfiles(date)
extract_downwelling_irradiance <- function(loggerdata){

  timestamp <- unlist(lapply(metdata, function(x){
    lubridate::ymd_hms(x$environment_sensor_readings$timestamp)
  }))

  wavelengths <- metdata[[1]]$environment_sensor_readings$spectrometer$wavelength[[1]]

  spectra <- do.call(‘rbind’, lapply(metdata, function(x){
    do.call(‘rbind’, x$environment_sensor_readings$spectrometer$spectrum)
    }
  ))

  return(list(spectra = spectra, timestamp = timestamp, wavelength = wavelength)
}
metfiles <- load_loggerfiles(date = ‘2017-04-15’)
spectra <- get_downwelling_irradiance(metfiles)

image(x = spectra$timestamp, y = spectra$wavelengths, z = spectra$spectra)

[lwang111]

When I run this

path <- file.path('/data/terraref/sites/ua-mac/raw_data/EnvironmentLogger', date)

I got an error

Error in file.path("/data/terraref/sites/ua-mac/raw_data/EnvironmentLogger",  : 
  cannot coerce type 'closure' to vector of type 'character'

[dlebauer]

try starting with

date <- '2015-04-15'

[dlebauer]

one issue was that R didn't recognize the “ that you copy-pasted. It will only recognize " and '.

This should work:

library(dplyr)
library(readr)
date = '2017-04-15'
load_loggerdata <- function(date){
  path <- file.path("/data/terraref/sites/ua-mac/raw_data/EnvironmentLogger", date)
  files <- dir(path, full.names = TRUE)
  loggerdata <- lapply(files, jsonlite::fromJSON) 
  timestamp <- combine(sapply(loggerdata, function(x){
    t <- x$environment_sensor_readings$timestamp
    lubridate::ymd_hms(t)
  }))
  return(list(data = loggerdata, timestamp = timestamp))
}

extract_downwelling_irradiance <- function(logdata){

  wavelengths <- logdata$data[[1]]$environment_sensor_readings$spectrometer$wavelength[[1]]

  spectra <- do.call('rbind', lapply(logdata$data, function(x){
    do.call('rbind', x$environment_sensor_readings$spectrometer$spectrum)
    }
  ))
  # image(x = timestamp, y = wavelengths, z = spectra)
  return(list(spectra = spectra, wavelengths = wavelengths, timestamp = logdata$timestamp))
}

extract_logger_met <- function(logdata){

  met <- do.call('rbind', lapply(logdata$data, function(x){
    tmp_met <- x$environment_sensor_readings
    data.frame(par = tmp_met$`sensor par`$value,
               co2 = tmp_met$`sensor co2`$value,
               sundir = tmp_met$weather_station$sunDirection$value,
               pressure = tmp_met$weather_station$airPressure$value,
               brightness = tmp_met$weather_station$brightness$value,
               rh = tmp_met$weather_station$relHumidity$value,
               temp = tmp_met$weather_station$temperature$value,
               wind_dir = tmp_met$weather_station$windDirection$value,
               wind_speed = tmp_met$weather_station$windVelocity$value)

  })) 
  return(met)
}

env_log_data <- load_loggerdata(date = '2017-04-15')
env_log_spectra <- extract_downwelling_irradiance(env_log_data)
env_log_met <- extract_logger_met(env_log_data)

[dileepdev7]

This worked for me, but I'm unsure what this data set is telling us since it looks quite different from the initial data sets we started with. Any ideas @dlebauer or @lwang111 ?

[dlebauer]

It is different because:

• it is available for every 3s interval over the past year
• it only covers VNIR range
• the goal would be to use these data to predict the full VNIR + SWIR range

[Ogaitnos]

Since this only covers VNIR range, we cannot (should not maybe is a better word) use this to train a machine or model to predict the behavior in the SWIR range. So we could apply a model (which we don't have yet) to this data afterward, but at this point, we really need more SWIR data. Is this correct? @dlebauer

[lwang111]

Totally agree. We need data to train the program. I found a folder 'data/terraref/sites/ua-mac/raw_data/SWIR' . Is that the data we can use?

[dlebauer]

This is how to extract the calibrated spectra from Level 1 / netCDF data

library(ncdf4)
library(udunits2)
library(lubridate)

for(date in c('2016-06-21', '2016-09-21', '2016-12-21', '2017-03-21', '2017-05-21')){

  directory <- file.path("/data/terraref/sites/ua-mac/Level_1/EnvironmentLogger", date)
  files <- dir(directory, full.names = TRUE)
  spectra_list <- lapply(files, function(x){
    metnc <- nc_open(x)
    spc <- ncvar_get(metnc, 'flx_spc_dwn')
    datetime <- ymd("1970-01-01") + 
      seconds(ud.convert(ncvar_get(metnc, 'time'), 'day', 's'))
    wvl <- ncvar_get(metnc, 'wvl_lgr')
    time <- hour(datetime) + 
      minute(datetime)/60 + 
      second(datetime)/3600
    return(list(spc = spc, wvl = wvl, date = ymd(strftime(datetime, '%Y%m%d')), datetime = datetime))

  })

  spectra_df <- do.call('cbind',(lapply(spectra_list, '[[', 'spc') ))
  dim(spectra_df)

  time <- do.call('c',lapply(spectra_list,'[[','datetime'))
  wavelengths <- spectra_list[[1]]$wvl
  save(spectra_df, time, wavelengths, file = file.path('data', paste0("spectra",date,".Rdata")))
  idx <- 1+0:700*24
  i <- 1:length(hr)[!is.na(hr)]
  library(lubridate)
  hr <- hour(time) + minute(time)/60 + second(time)/3600
  png(filename = paste0('data/spectra',date,'.png'))
  image(x = wavelengths, y = as.numeric(hr[idx]), spectra_df[,idx],
        ylab = 'hour of day', 
        xlab = 'wavelength (nm)',
        col = cm.colors(n=100),zlim = c(-1,2.1),
        main = paste0('diurnal solar spectral radiation\n',date))
  dev.off()

}  

[dlebauer]

@lwang111 for training you can use the small text files that were captured with a spectroradiometer. You could use the data in terraref/sites/ua-mac/Level_1/hyperspectral_swir/2017-04-15/, which are scans of the different grey / white targets with the SWIR camera for validation. However, this will require some work to identify the targets and subset.

[lwang111]

I run the code to read from folder 'Level_1' which gives me

Error in length(hr) : object 'hr' not found

Also I fixed one 'bug': we cannot save any file in '/data' so I changed it to 'data_'