load-sensor-data.R doesn't work for hpoulos

[hpoulos]

Now that I've recloned and forked the skyisland-climate repo and started over from scratch with git, I am having trouble getting the load-sensor-data.R script to run. I attempted to just run this from randomforest.R source("./microclimate-topo-PCA.R") PCAs <- loadPCAData() and it did not work. So then I went back to the load-sensor-data.R script and ran the following:

### main script ###
source("./iButton.R")

# get time of last iButton merge
if(! file.exists(BUILD_TIMESTAMP)) stop("Merged iButton data not found. Run build-merged-ibutton.py")
build_time <- scan(BUILD_TIMESTAMP, what="character", quiet=TRUE)
build_time <- ymd_hms(paste(build_time, collapse=" "))

alltemps <- get_data(TEMP_DATA, build_time, read_all_sensors) # retrieve the timeseries data

#at the step below, I got the following error after all the sensors were read in:
[1] "reading sensor SP603"
[1] "merging time series"
[1] "merging time series"
Error in mget(sensors) : invalid first argument
Called from: mget(sensors)

I think that this traces back to when Dylan sent me the .rds files because the script always worked for me after that, but not after I cloned the repo and started working on my local repo again. I just downloaded the .rds files you sent months ago to my results\temp data folder and now the script works.

[dschwilk]

Ok, so short term solution is for me to send you updated rds files.

I'll send those in a few minutes

On 10/13/2015 09:19 AM, hpoulos wrote:

Now that I've recloned and forked the skyisland-climate repo and started over from scratch with git, I am having trouble getting the load-sensor-data.R script to run. I attempted to just run this from randomforest.R source("./microclimate-topo-PCA.R") PCAs <- loadPCAData() and it did not work. So then I went back to the load-sensor-data.R script and ran the following:

  main script

source("./iButton.R")

get time of last iButton merge

if(! file.exists(BUILD_TIMESTAMP)) stop("Merged iButton data not found. Run build-merged-ibutton.py") build_time <- scan(BUILD_TIMESTAMP, what="character", quiet=TRUE) build_time <- ymd_hms(paste(build_time, collapse=" "))

alltemps <- get_data(TEMP_DATA, build_time, read_all_sensors) # retrieve the timeseries data

at the step below, I got the following error after all the sensors were read in:

[1] "reading sensor SP603" [1] "merging time series" [1] "merging time series" Error in mget(sensors) : invalid first argument Called from: mget(sensors)

I think that this traces back to when Dylan sent me the .rds files because the script always worked for me after that, but not after I cloned the repo and started working on my local repo again. I just downloaded the .rds files you sent months ago to my results\temp data folder and now the script works.

— Reply to this email directly or view it on GitHub https://github.com/schwilklab/skyisland-climate/issues/18.

[dschwilk]

Formatting hint for issues: use code blocks in markdown for showing code. I'll edit the issue above as an example

[dschwilk]

You should run build-merged-ibutton.py first. You must call this on its own or use the build-sensor-data.sh shell script. If this is running without error we can move on to the R code. But if that has errors, then you are not getting the time series built from the raw ibutton data. I notice that I have the R code warn you if the timestamp file does not exist at all, but not if it is out of date. Check that there are data for all three ranges in ./microclimate/merged-ibutton.

A few notes on this error and bug for helping to discover the cause:

1. SP603 should not be the last sensor. That should be "reading sensor SR523-GROUND currently.
2. The doubled info line, "merging time series" is weird. There should be one such notice after each mtn range. You are getting two notices after the GM sensors are read.

So my first guess is that somehow all of the CM iButton data is missing. is build_merged_ibutton.py running without error? Does the mergied-ibutton data exist?

[hpoulos]

I did run the build-merged-ibutton.py first, but I still go the error. I've just gotten the randomforest.R script to work and then I'll get back to this issue.

[dschwilk]

Ok, well the rds files should work for now. But weird! I've set up an appt to test out on a clean clone on a windows box.

It would be good to get the full console output and then the full traceback after the error (traceback())

Also, we can try just figuring out why no CM sensors seemed to be read by adding a print statement in iButton.R Around line 70:

   read.sensor.data <- function(sfiles){
    sensors=basename(sfiles)
    sensors = strip.extension(sensors)
    print(sensors) # add this so we can see what files it found. 
                   # Should be list of files in 
                   # microclimate/merged-ibutton/GM/T
    for (i in seq(along=sfiles)) {

[dschwilk]

We cannot reproduce this error with a fresh clone of the repo on windows 10 (using rstudio).

@lindbeem2791 kindly cloned a fresh copy of the repo on his windows laptop and ran all the steps with no errors. All iButton data reads fine. So there must be something specific to @hpoulos's setup. I'll leave this open for now since we do need this to work for Helen. But I'm at a loss as to what is going on.

[hpoulos]

Using python versions associated with ArcGIS can cause Python to hang. Installing latest version of Phython and running script fixes this bug.