Open dlebauer opened 7 years ago
When I run this
path <- file.path('/data/terraref/sites/ua-mac/raw_data/EnvironmentLogger', date)
I got an error
Error in file.path("/data/terraref/sites/ua-mac/raw_data/EnvironmentLogger", :
cannot coerce type 'closure' to vector of type 'character'
try starting with
date <- '2015-04-15'
one issue was that R didn't recognize the “
that you copy-pasted. It will only recognize "
and '
.
This should work:
library(dplyr)
library(readr)
date = '2017-04-15'
load_loggerdata <- function(date){
path <- file.path("/data/terraref/sites/ua-mac/raw_data/EnvironmentLogger", date)
files <- dir(path, full.names = TRUE)
loggerdata <- lapply(files, jsonlite::fromJSON)
timestamp <- combine(sapply(loggerdata, function(x){
t <- x$environment_sensor_readings$timestamp
lubridate::ymd_hms(t)
}))
return(list(data = loggerdata, timestamp = timestamp))
}
extract_downwelling_irradiance <- function(logdata){
wavelengths <- logdata$data[[1]]$environment_sensor_readings$spectrometer$wavelength[[1]]
spectra <- do.call('rbind', lapply(logdata$data, function(x){
do.call('rbind', x$environment_sensor_readings$spectrometer$spectrum)
}
))
# image(x = timestamp, y = wavelengths, z = spectra)
return(list(spectra = spectra, wavelengths = wavelengths, timestamp = logdata$timestamp))
}
extract_logger_met <- function(logdata){
met <- do.call('rbind', lapply(logdata$data, function(x){
tmp_met <- x$environment_sensor_readings
data.frame(par = tmp_met$`sensor par`$value,
co2 = tmp_met$`sensor co2`$value,
sundir = tmp_met$weather_station$sunDirection$value,
pressure = tmp_met$weather_station$airPressure$value,
brightness = tmp_met$weather_station$brightness$value,
rh = tmp_met$weather_station$relHumidity$value,
temp = tmp_met$weather_station$temperature$value,
wind_dir = tmp_met$weather_station$windDirection$value,
wind_speed = tmp_met$weather_station$windVelocity$value)
}))
return(met)
}
env_log_data <- load_loggerdata(date = '2017-04-15')
env_log_spectra <- extract_downwelling_irradiance(env_log_data)
env_log_met <- extract_logger_met(env_log_data)
This worked for me, but I'm unsure what this data set is telling us since it looks quite different from the initial data sets we started with. Any ideas @dlebauer or @lwang111 ?
It is different because:
Since this only covers VNIR range, we cannot (should not maybe is a better word) use this to train a machine or model to predict the behavior in the SWIR range. So we could apply a model (which we don't have yet) to this data afterward, but at this point, we really need more SWIR data. Is this correct? @dlebauer
Totally agree. We need data to train the program. I found a folder 'data/terraref/sites/ua-mac/raw_data/SWIR' . Is that the data we can use?
This is how to extract the calibrated spectra from Level 1 / netCDF data
library(ncdf4)
library(udunits2)
library(lubridate)
for(date in c('2016-06-21', '2016-09-21', '2016-12-21', '2017-03-21', '2017-05-21')){
directory <- file.path("/data/terraref/sites/ua-mac/Level_1/EnvironmentLogger", date)
files <- dir(directory, full.names = TRUE)
spectra_list <- lapply(files, function(x){
metnc <- nc_open(x)
spc <- ncvar_get(metnc, 'flx_spc_dwn')
datetime <- ymd("1970-01-01") +
seconds(ud.convert(ncvar_get(metnc, 'time'), 'day', 's'))
wvl <- ncvar_get(metnc, 'wvl_lgr')
time <- hour(datetime) +
minute(datetime)/60 +
second(datetime)/3600
return(list(spc = spc, wvl = wvl, date = ymd(strftime(datetime, '%Y%m%d')), datetime = datetime))
})
spectra_df <- do.call('cbind',(lapply(spectra_list, '[[', 'spc') ))
dim(spectra_df)
time <- do.call('c',lapply(spectra_list,'[[','datetime'))
wavelengths <- spectra_list[[1]]$wvl
save(spectra_df, time, wavelengths, file = file.path('data', paste0("spectra",date,".Rdata")))
idx <- 1+0:700*24
i <- 1:length(hr)[!is.na(hr)]
library(lubridate)
hr <- hour(time) + minute(time)/60 + second(time)/3600
png(filename = paste0('data/spectra',date,'.png'))
image(x = wavelengths, y = as.numeric(hr[idx]), spectra_df[,idx],
ylab = 'hour of day',
xlab = 'wavelength (nm)',
col = cm.colors(n=100),zlim = c(-1,2.1),
main = paste0('diurnal solar spectral radiation\n',date))
dev.off()
}
@lwang111 for training you can use the small text files that were captured with a spectroradiometer. You could use the data in terraref/sites/ua-mac/Level_1/hyperspectral_swir/2017-04-15/, which are scans of the different grey / white targets with the SWIR camera for validation. However, this will require some work to identify the targets and subset.
I run the code to read from folder 'Level_1' which gives me
Error in length(hr) : object 'hr' not found
Also I fixed one 'bug': we cannot save any file in '/data' so I changed it to 'data_'
@dlebauer commented on Thu Jun 01 2017
code to extract 1/3 hz spectral data; need to change date