Thermistor chain data for Keweenaw North & South locations

[kknee]

From Colin:

When the power finally comes back up in northeast Ann Arbor (and we can turn on our servers again - we've been without power at MTRI for 24 hours now!), you should be able to find the 2016 thermistor chain XML data for the Keweenaw North & South sites at this location in GLOS-friendly format to our public ftp: ftp://ftp.mtri.org/pub/DMAC/Keweenaw_Thermistor/

[gcutrell]

Best guess is the xml files go in mtri's "MTRI-NKW" and "MTRI-SKW" in michigan.glin before being parsed and placed in the Glos_obs database? @kknee does that sound right?

[kknee]

@gcutrell actually we're using netCDF and THREDDS for these - the previous ones are here: http://tds.glos.us/thredds/mtri/water_temp_in_situ.html

[gcutrell]

@kknee Thanks. So the two xml files should be placed in their respective thredds directories within "MTRI-WT"? You then have a script to update the current .nc file with the new xml data? I'm assuming this is a manual process since we would only receive their data once a year? Just curious about the process to better understand what is happening.

[kknee]

@gcutrell - Guan did the conversion from XML to nc and places the nc files in the TDS directories. I'm not sure where his scrip is. If you'd like us to help with the conversion please just coordinate with @tslawecki to task us. Thanks!

[kkoch]

From Colin > Which reminds me, I think I have a missing year of Keweenaw thermistor chain data to send to you. 2015 perhaps? I'll dig it up. Amanda worked on prepping it for sharing.

[kkoch]

@gcutrell (from Colin) > When we shared the 2016 Keweenaw North & South thermistor chain data, I noticed we had never shared the 2015 data with the GLOS Portal: those data are now available at: ftp://ftp.mtri.org/pub/DMAC/Keweenaw_Thermistor/

Can we got those integrated into the Portal as well?

[gcutrell]

@kkoch Now that I understand better how the parser and database works I don't think wehave any data from Keweenaw since 2013. I believe the issue is the station is name "North Keeweenaw Waterway" in the 2015 xml files (probably in the 2016 as well) but the parser is expecting "MTRI-NKW" or "MTRI-SKW". So the parser is reading in the xml files but doing nothing since there is no matching station ID. It would be a simple fix in the xml files if Colin has the 2014 and 2016 xml files that he can provide.

[gcutrell]

@kkoch I still have the 2016 data so we just need the 2014.

[kkoch]

@ColinNBrooks > punt! (see Greg's comments above)

[ColinNBrooks]

OK, I'll send on the 2014 XML files too. I hadn't realized the issue with the station name.

[gcutrell]

@ColinNBrooks Friendly reminder about the 2014 data. I need to put the data into GLOS's database chronological so I can't put in 2015 or 2016 until 2014 is read in.

[gcutrell]

@ColinNBrooks Just crossed my mind that Amanda hasn't sent the 2014 data. I can't find it in my email but I believe you said she was going to send it.

[ColinNBrooks]

Note from Amanda:

Hi Colin & Greg, I re-posted the 2014 data at ftp://ftp.mtri.org/pub/DMAC/Keweenaw_Thermistor/

[gcutrell]

@ColinNBrooks @kkoch After half a year 2014, 2015, and 2016 is now in glos's db. Just want to confirm the south thermistor stopped on 7/22/16? Couple formatting issues with the xml files but I can easily fix going forward.

[ColinNBrooks]

I checked the original data as sent to me, and the south thermistor does indeed stop on 7/22/16. After that, there's only the north thermistor "late season" data. So I think we're good. Thank you.

[kkoch]

@gcutrell still not seeing current data in portal. Is there still something that needs to be done?

[gcutrell]

@kkoch All the data is in the database. Maybe it is showing that because there is no recent data?

[ColinNBrooks]

I tried using the "Custom Range" function to look at data from June & July 2016 when there should be data in there for the Keweenaw North thermistor chain, and I couldn't get anything to come up there either. But I'm not sure I was using the data Custom Range function properly.

[kknee]

@ColinNBrooks @gcutrell - using the custom date range won't work for this data because the obs cache only stores several days of recent data and will be empty for these datasets.

I believe the data really needs to go into the .nc files here: http://tds.glos.us/thredds/mtri/water_temp_in_situ.html and the portal needs to be pointed to those instead of the obs cache so that we have access to the full archive of data

[gcutrell]

@kknee Would the buoy catalog work? http://tds.glos.us/thredds/catalog/buoys_standard/MTRI-SKW/catalog.html & http://tds.glos.us/thredds/catalog/buoys_standard/MTRI-NKW/catalog.html

[kknee]

@gcutrell we'll take a look and come back to you. If yes, any reason not to clean up the older folder of data? Looks like you have duplicated it in the new files?

[gcutrell]

@kknee Glbuoys currently needs the data in http://tds.glos.us/thredds/catalog/buoys/catalog.html because it is using the variables "short name" and the new buoy datasets contain the variables long name. My plan is to point Glbuoys to the new datasets this winter after some code changes.

[Bobfrat]

These are on dev now. Titles are MTRI-SKW and MTRI-NKW.

[ColinNBrooks]

I see the data layers in there. The first value of the thermistor string look strange - 6 C vs. 21.5 at all others for the MTRI-NKW data of 10/15/16, and 1 C vs. Nan from 1-5m and then 22.6 C at 6m. I would need to check the original data to see where the issue is.

Looks like I need to provide an updated version of the layer description, or we can re-use the existing ones that show up under the "Keweenaw Waterway Thermistor Chain Data - North" (and "- South"), which are:

Keweenaw Waterway Thermistor Chain Data - North Michigan Technological University maintains thermistor sensor chains deployed each year at the north and south entrances to the Keweenaw Waterway in Lake Superior. This station is North Entry to Keweenaw Waterway, 47.210397 N, -88.617426 W. Each thermistor chain includes six sensors spaced at 1-m intervals that collect water temperature data at depths of 1-6 m below the surface. The sensor chains were designed by Michigan Tech Research Institute (MTRI) using Thermocron iButtons (model DS1921Z), temperature-sensitive resistors made of a semi-conductive material. Each iButton is enclosed in a 16 mm thick stainless steel can that is able to withstand the harsh conditions of Lake Superior. Following retrieval of the chains at the end of the season, temperature data are transferred from the iButton to a PC using a download wizard. Data are delivered to GLOS annually at the end of the buoy deployment season as a set of text files, one file per sensor. Each file consists of header data describing sampling parameters such as sample rate (60 minutes), temperature scale, mission start time, and total number of samples collected, followed by a data log of dates, times and temperatures. The design, deployment and monitoring of the Keweenaw Waterway thermistor chains is a collaboration between the Michigan Tech University Great Lakes Research Center, Michigan Tech Research Institute and Great Lakes Observing System. External links

Keweenaw Waterway Thermistor Chain Data - South Michigan Technological University maintains thermistor sensor chains deployed each year at the north and south entrances to the Keweenaw Waterway in Lake Superior. This station is South Entry to Keweenaw Waterway, 46.977896 N, 88.433310 W. Each thermistor chain includes six sensors spaced at 1-m intervals that collect water temperature data at depths of 1-6 m below the surface. The sensor chains were designed by Michigan Tech Research Institute (MTRI) using Thermocron iButtons (model DS1921Z), temperature-sensitive resistors made of a semi-conductive material. Each iButton is enclosed in a 16 mm thick stainless steel can that is able to withstand the harsh conditions of Lake Superior. Following retrieval of the chains at the end of the season, temperature data are transferred from the iButton to a PC using a download wizard. Data are delivered to GLOS annually at the end of the buoy deployment season as a set of text files, one file per sensor. Each file consists of header data describing sampling parameters such as sample rate (60 minutes), temperature scale, mission start time, and total number of samples collected, followed by a data log of dates, times and temperatures. The design, deployment and monitoring of the Keweenaw Waterway thermistor chains is a collaboration between the Michigan Tech University Great Lakes Research Center, Michigan Tech Research Institute and Great Lakes Observing System.

[kknee]

@ColinNBrooks we noted the same thing regarding the data. The nc files look okay, so @cheryldmorse will check the DB too.

Earlier in the year the values look okay

[cheryldmorse]

@kknee, @ColinNBrooks - The values being reported match what is in the obs database.

[ColinNBrooks]

Thanks. I tried looking at the data this morning and now I don't see any data being reported for either Keweenaw thermistor chain - shouldn't it report the most recent values, even though those are from last year? I thought that's what was happening last week when I was looking at the odd values through the mapping part of the portal.

I checked the original text files that were imported into the XML we shared and I don't see the 6C for the shallowest level for NKW and 1C values in the SKW data. I'll check the XML file we sent and see I can figure out what's going on.

[kknee]

@ColinNBrooks make sure you review the data on dev (http://dev.oceansmap.com/myglos/#), the new data has not been pushed to production yet

@gcutrell can you work with Colin on the odd values in the data that we are seeing?

[gcutrell]

@ColinNBrooks I see what has happened but not sure why the error occurred since your xml file looks okay. For some reason the parser placed a "null" temp string depth with a value of 6 in the db for NKW. I should be able to just delete the records that have a depth of "null" and everything should be correct. I'll let y'all know when it is done.

[cheryldmorse]

@gcutrell - The netCDF files will need to be regenerated after the database is updated

[ColinNBrooks]

Looking at the dev portal, the first number for MTRI-SKW and MTRI-NKW is not a temperature value, but the number of records at that particular time. For example, for MTRI-SKW on July 22, 2016 06:31, there is one value (at 6m with 22.6 deg C). For MTRI-NKW, there are six values for Oct. 15. 2016 at 15:09 from 1m to 6m deep all at 21.5 deg C. Can the first value be renamed from "Thermistor String" to something like "Thermistor String observations" at changed to a plain integer, not a deg. C one?

Also, can the descriptions for MTRI-SKW & MTRI-NKW be borrowed from the Keweenaw Waterway Thermistor Chain Data - South & - North (and they should be renamed to same as the old entries as well; we shouldn't go live with the names MTRI-SKW & MTRI-NKW.).

[gcutrell]

@cheryldmorse Sorry for delay but I just removed all the bad rows from the temp strings. The data should look okay now after recreating the files.

[cheryldmorse]

@gcutrell The NC files have been regenerated