missing 2019 temperature data

[wwieder]

Looks like there's no air temperature data from Jan through mid-March 2019. I'm assuming these are missing in the Tvan observations, can they be filled with measurements from the Saddle met station?

[wwieder]

Looks like the saddle met data have not yet been uploaded for 2019 with data product 57 https://portal.edirepository.org/nis/metadataviewer?packageid=knb-lter-nwt.57.3 @scelmendorf, I wonder is this is an equally good alternative? https://portal.edirepository.org/nis/metadataviewer?packageid=knb-lter-nwt.405.4.

Alternatively, will 57 be updated?

[scelmendorf]

@wwieder that was absolutely my fault, somehow I had checked off the list that both the hourly and daily had been updated but had only actually pushed submit on the daily. Fixed now - on to 57.4!

[wwieder]

Thanks @scelmendorf ! I wonder if you can also help with this error I'm now getting after running saddle_met_data_fp <- download_EDI(edi_id = saddle_met_data, dest_dir = paste0(DirOut,"/saddle_met_data/"), getNewData = TRUE)

Error in read_xml.raw(raw, encoding = encoding, base_url = base_url, as_html = as_html, : Opening and ending tag mismatch: link line 7 and head [76]

here's the code https://github.com/hhollandmoritz/NWT_CLM/blob/02a009e0067686f9595e654d70fbd2df473ff023/tvan_supplemental_cleaning.R#L733

[scelmendorf]

@wwieder I asked Hannah about this in the last meeting and thought she had updated. But anyhow, yes, known issue that EDI depracated the LTER ldap sometime in September. To fix see 1 and 2 (below). I'm also including my notes on a few other areas if you want to consider additional modifications. (1) EDI depracated the LTER data portal and merged it into EDI a month or so ago, update line 403 myurl<-paste0("https://portal.lternet.edu/nis/metadataviewer?packageid=", to myurl<-paste0("https://portal.edirepository.org/nis/metadataviewer?packageid=", (2) myeml<-xml2::read_xml(paste0("https://portal.lternet.edu/nis/metadataviewer?packageid=", update line 407 to myeml<-xml2::read_xml(paste0("https://portal.edirepository.org/nis/metadataviewer?packageid=",

OTHER things I discovered working through this: (3) FWIW (and I checked in with EDI and they confirmed this has not moved, timeline to moved unclear) "versions = readLines(paste0('https://pasta.lternet.edu/package/eml/knb-lter-nwt/', edi_id), warn = FALSE)" needs to stay as lternet.edu, so DO NOT replace that one. Pasta web services still working at the lternet domain, but the portal.lter has moved.

(4) I would set up the data_dir and output_dir_base in if{} statements so that the code do not require commenting out for different users, rather you can just add in your own as there's not really any harm (?) in having it set up to run on multiple computers. if (dir.exists('/home/hannah/Downloads/TVan_in/')){ data_dir <- '/home/hannah/Downloads/TVan_in/' output_dir_base <- '/home/hannah/Downloads/Tvan_out/' } if (dir.exists('D:/TVan_in/')){ data_dir <- 'D:/TVan_in/' output_dir_base <- 'D:/TVan_out/' }

(5) I am unclear on why the header is hard_coded lin lines ~1950 on. IMO this is risky as if two columns get switched or a variable gets added, it will break the code. I would just read in the header from the file and or do a 'rename' in the code to map old names to new if you need to rather than hard coding names to column order.

(6) to make the code work on windows (which probably is not your problem), I had to add at line 248 a line to monkey with the default windows line endings. But I have not tested it on unix/mac to figure out if it breaks things there. If we want to make it more system-agnostic it would be helpful to test this.

[wwieder]

Thanks Sarah, Now I can get the data, but the following few lines also throw an error I'm not clear how to address?

colclasses <- gsub("Date", "character", saddle_met_data_fp$colclasses) sadd_met <- read.csv(file = saddle_met_data_fp$csv,

• colClasses = colclasses) Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec, : scan() expected 'a real', got '"NaN"'

On Mon, Nov 2, 2020 at 8:37 AM Sarah Elmendorf notifications@github.com wrote:

@wwieder https://github.com/wwieder I asked Hannah about this in the last meeting and thought she had updated. But anyhow, yes, known issue that EDI depracated the LTER ldap sometime in September. To fix see 1 and 2 (below). I'm also including my notes on a few other areas if you want to consider additional modifications. (1) EDI depracated the LTER data portal and merged it into EDI a month or so ago, update line 403 myurl<-paste0("https://portal.lternet.edu/nis/metadataviewer?packageid=", to myurl<-paste0(" https://portal.edirepository.org/nis/metadataviewer?packageid=", (2) myeml<-xml2::read_xml(paste0(" https://portal.lternet.edu/nis/metadataviewer?packageid=", update line 407 to myeml<-xml2::read_xml(paste0(" https://portal.edirepository.org/nis/metadataviewer?packageid=",

OTHER things I discovered working through this: (3) FWIW (and I checked in with EDI and they confirmed this has not moved, timeline to moved unclear) "versions = readLines(paste0(' https://pasta.lternet.edu/package/eml/knb-lter-nwt/', edi_id), warn = FALSE)" needs to stay as lternet.edu, so DO NOT replace that one. Pasta web services still working at the lternet domain, but the portal.lter has moved.

(4) I would set up the data_dir and output_dir_base in if{} statements so that the code do not require commenting out for different users, rather you can just add in your own as there's not really any harm (?) in having it set up to run on multiple computers. if (dir.exists('/home/hannah/Downloads/TVan_in/')){ data_dir <- '/home/hannah/Downloads/TVan_in/' output_dir_base <- '/home/hannah/Downloads/Tvan_out/' } if (dir.exists('D:/TVan_in/')){ data_dir <- 'D:/TVan_in/' output_dir_base <- 'D:/TVan_out/' }

(5) I am unclear on why the header is hard_coded lin lines ~1950 on. IMO this is risky as if two columns get switched or a variable gets added, it will break the code. I would just read in the header from the file and or do a 'rename' in the code to map old names to new if you need to rather than hard coding names to column order.

(6) to make the code work on windows (which probably is not your problem), I had to add at line 248 a line to monkey with the default windows line endings. But I have not tested it on unix/mac to figure out if it breaks things there. If we want to make it more system-agnostic it would be helpful to test this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hhollandmoritz/NWT_CLM/issues/17#issuecomment-720547313, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5IWJFT5MOLPS2B4MEJCZ3SN3G2DANCNFSM4S6VDDXA .

[scelmendorf]

guessing you need a na.strings='NaN' in there?

[wwieder]

that didn't seem to help, but getting rid of colClasses = colclasses seems to work,

On Mon, Nov 2, 2020 at 9:31 AM Will Wieder wwieder@ucar.edu wrote:

Thanks Sarah, Now I can get the data, but the following few lines also throw an error I'm not clear how to address?

colclasses <- gsub("Date", "character", saddle_met_data_fp$colclasses) sadd_met <- read.csv(file = saddle_met_data_fp$csv,

• colClasses = colclasses) Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec, : scan() expected 'a real', got '"NaN"'

On Mon, Nov 2, 2020 at 8:37 AM Sarah Elmendorf notifications@github.com wrote:

@wwieder https://github.com/wwieder I asked Hannah about this in the last meeting and thought she had updated. But anyhow, yes, known issue that EDI depracated the LTER ldap sometime in September. To fix see 1 and 2 (below). I'm also including my notes on a few other areas if you want to consider additional modifications. (1) EDI depracated the LTER data portal and merged it into EDI a month or so ago, update line 403 myurl<-paste0("https://portal.lternet.edu/nis/metadataviewer?packageid=", to myurl<-paste0(" https://portal.edirepository.org/nis/metadataviewer?packageid=", (2) myeml<-xml2::read_xml(paste0(" https://portal.lternet.edu/nis/metadataviewer?packageid=", update line 407 to myeml<-xml2::read_xml(paste0(" https://portal.edirepository.org/nis/metadataviewer?packageid=",

OTHER things I discovered working through this: (3) FWIW (and I checked in with EDI and they confirmed this has not moved, timeline to moved unclear) "versions = readLines(paste0(' https://pasta.lternet.edu/package/eml/knb-lter-nwt/', edi_id), warn = FALSE)" needs to stay as lternet.edu, so DO NOT replace that one. Pasta web services still working at the lternet domain, but the portal.lter has moved.

(4) I would set up the data_dir and output_dir_base in if{} statements so that the code do not require commenting out for different users, rather you can just add in your own as there's not really any harm (?) in having it set up to run on multiple computers. if (dir.exists('/home/hannah/Downloads/TVan_in/')){ data_dir <- '/home/hannah/Downloads/TVan_in/' output_dir_base <- '/home/hannah/Downloads/Tvan_out/' } if (dir.exists('D:/TVan_in/')){ data_dir <- 'D:/TVan_in/' output_dir_base <- 'D:/TVan_out/' }

(5) I am unclear on why the header is hard_coded lin lines ~1950 on. IMO this is risky as if two columns get switched or a variable gets added, it will break the code. I would just read in the header from the file and or do a 'rename' in the code to map old names to new if you need to rather than hard coding names to column order.

(6) to make the code work on windows (which probably is not your problem), I had to add at line 248 a line to monkey with the default windows line endings. But I have not tested it on unix/mac to figure out if it breaks things there. If we want to make it more system-agnostic it would be helpful to test this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hhollandmoritz/NWT_CLM/issues/17#issuecomment-720547313, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5IWJFT5MOLPS2B4MEJCZ3SN3G2DANCNFSM4S6VDDXA .

[scelmendorf]

Sounds right? I am not sure the rationale for hardcoding the colclasses and if it breaks something further down. I think if you do a quick plot of the temp data it will be pretty immediately obvious if something read in wrong.

[wwieder]

Ah! after all that there's no air temperature data for the first part of 2019 from the saddle either?! Is this right? it seems odd that the everything fritzed out Jan 1 2019?

[scelmendorf]

@wwieder banana crisis resolved. I am remembering now - Jen cut over from the one sensor to triplicate hmp sensors last year. There I think(?) was some time in 2019 when there was overlap if you want to check one against the other for cross-calibration. But briefly, I think what you want to be using is the mean of the unflagged 3 airtemp_hmpX_avg, where X is 1, 2, or 3.