barometric pressure in variable baro not plotted

[emontgomery-usgs]

@kwilcox I didn't even know these from Dauphin had barometric pressure: 9651hwl-a.nc, 9691hwl-a.nc, 9731hwl-a.nc, 9741hwl-a.nc, 9751hwl-a.nc, 9761hwl-a.nc, and this one from Chandeleur: 9461hwl-a.nc . The variable name is Baro, with long name Barometric pressure, but there's no epic_code, so has the same problems as the other 2. These are complicated by being buried in an overwash zone at the beach, but the variable called "baro" should map to atmospheric pressure, instead of not mapping to anything.

Could another line be added to collect.py to handle this case? Thanks!

The current CF version is at: http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/CF-1.6/DAUPHIN/catalog.html, http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/CF-1.6/CHANDELEUR/catalog.html

[kwilcox]

From the Dauphin files you listed I only see one with Air Pressure, and it has the epic_code set correctly: http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/Data/DAUPHIN/9651hwlb-a.nc.html

Are we seeing the same thing?

[emontgomery-usgs]

I just made corrections to those files. Previously 9651hwlb was 9651hwl and had 'Baro' and 'P_1', leading to confusion. While I was in there I changed Baro to BP_915, so I think now if you re-harvest these DAUPHIN and CHANDELEUR, they should be fixed.

Also please try to reharvest from PV_SHELF07- there's 8451met file that has BP-915, that doesn't plot in the portal. I think it's probably fixed by the mapping you fixed in the previous ticket.

Let me know when you re-harvest and I'll check then close the ticket. Thanks!

On Fri, Mar 23, 2018 at 1:58 PM, Kyle Wilcox notifications@github.com wrote:

From the Dauphin files you listed I only see one with Air Pressure, and it has the epic_code set correctly: http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/Data/ DAUPHIN/9651hwlb-a.nc.html

Are we seeing the same thing?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/277#issuecomment-375751238, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHnp5PUixWBg-N-8LHfcWR_lq0-qIkks5thTfDgaJpZM4S3gnQ .

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/operations/stg/

[kwilcox]

double BP_915(time, lon, lat, depth) ;
                BP_915:long_name = "Barometric Pressure" ;
                BP_915:units = "Psi" ;
                BP_915:height_depth_units = "m" ;
                BP_915:initial_instrument_height = 4. ;
                BP_915:minimum = 14.4108 ;
                BP_915:maximum = 15.0622 ;
                BP_915:serial_number = "10125471" ;
                BP_915:valid_range = 0., 20. ;
                BP_915:_FillValue = 1.e+35 ;
                BP_915:epic_code = "915" ;

epic_code 915 should be in mbar but this variable is in PSI and it is not being converted correctly. I don't think these should be labeled as BP_915 unless you convert the variable in the raw files.

[emontgomery-usgs]

Hmm, maybe that's why they were called "Baro" initially. Can't there be an over-ride if the units attribute doesn't match the EPIC definition of units? Isn't some form of automatic conversion that happens in CF, as long as it knows what the original units are?

On Mon, Mar 26, 2018 at 9:50 AM, Kyle Wilcox notifications@github.com wrote:

double BP_915(time, lon, lat, depth) ; BP_915:long_name = "Barometric Pressure" ; BP_915:units = "Psi" ; BP_915:height_depth_units = "m" ; BP_915:initial_instrument_height = 4. ; BP_915:minimum = 14.4108 ; BP_915:maximum = 15.0622 ; BP_915:serial_number = "10125471" ; BP_915:valid_range = 0., 20. ; BP_915:_FillValue = 1.e+35 ; BP_915:epic_code = "915" ;

epic_code 915 should be in mbar but this variable is in PSI and it is not being converted correctly. I don't think these should be labeled as BP_915 unless you convert the variable in the raw files.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/277#issuecomment-376172566, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHniioehUClIFeeurY-XsLZSia8rSpks5tiPIfgaJpZM4S3gnQ .

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/operations/stg/

[kwilcox]

If an epic_code is found it overrides everything else. If there is no epic_code for a variable it needs to be left off and standard_name and units attributes need to be present. Adding some logic to check to make sure the units attribute is correct for the epic_code is possible but would cause issues with raw netcdf that have incorrect units attributes.

The Baro variables seem to have been working fine: https://cmgdata.usgsportals.net/#metadata/56432/station (yes, that should be air_temperature and the next re-process should fix that). There has been an override for these for awhile: https://github.com/USGS-CMG/usgs-cmg-portal/blob/master/woods_hole_obs_data/collect.py#L160.

[emontgomery-usgs]

So if it's called 'Baro", it shouldn't have an epic_code, correct? I can rename the current BP_915 in Dauphin and Chadeleur, but don't want unintended consequences. The existing data seems to plot the Barometric pressure as milibars correctly., but incorrectly has a seawater pressure variable and calls air temperature water temperature.

a re-harvest will be needed to correct these once I have Baro as you want.

[kwilcox]

You are correct: Baro and no epic_code.

[emontgomery-usgs]

new files are up in Data/CHANDELEUR_13 and /DAUPHIN

[kwilcox]

I just pushed changes that should take care of all of the different representations of Air Pressure regardless of what the epic_code or units is set to. The most important thing is that epic_code: 915 is set and they have a units attribute that is convertible to Pa (the CF unit).

Variables with the name Baro, BP_915 and BPR_915 are set to epic_code: 915 as well as anything with long_name: barometric pressure (case insensitive).

This change required a new library cf_units to handle the unit conversions, you'll need to conda install -c conda-forge --file requirements.txt in the woods_hole_obs_data directory.

[emontgomery-usgs]

Thanks! I'm not clear about the case of air pressure variables called Baro, where units are PSI; and there's no epic_code att- is that still covered in the way it works now?

I just looked on the portal, and it appears that 965 has not been replaced/reharvested correctly. The file now only contains these data variables: 'Baro', and 'T_21' (air temperature and pressure, no sea_water pressure). As the site map suggests, the sensor was on a house, in air. The source file for this site is: http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/Data/DAUPHIN/9651hwlb-a.nc.html ,

On Tue, Mar 27, 2018 at 9:43 AM, Kyle Wilcox notifications@github.com wrote:

I just pushed changes that should take care of all of the different representations of Air Pressure regardless of what the epic_code or units is set to. The most important thing is that epic_code: 915 is set and they have a units attribute that is convertible to Pa (the CF unit).

Variables with the name Baro, BP_915 and BPR_915 are set to epic_code: 915 as well as anything with long_name: barometric pressure (case insensitive).

This change required a new library cf_units to handle the unit conversions, you'll need to conda install -c conda-forge --file requirements.txt in the woods_hole_obs_data directory.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/277#issuecomment-376529824, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHnhoaX1qgEiRnJJclOSjEhP_3gtYzks5tikH2gaJpZM4S3gnQ .

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/operations/stg/

[kwilcox]

Thanks! I'm not clear about the case of air pressure variables called Baro, where units are PSI; and there's no epic_code att- is that still covered in the way it works now?

Yes, the variable name Baro is enough to match it.

I just looked on the portal, and it appears that 965 has not been replaced/reharvested correctly.

I haven't re-harvested anything yet. Can you update the CF files on your end? We have been trying to automate the process and my end should pick up changes when you make them.

[emontgomery-usgs]

Already updated (at ~15:30 yesterday), so if it was supposed to pick up the change, it didn't.

Here's the URL to the CF version there now that has Baro and T_21: http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/CF-1.6/DAUPHIN/9651hwlb-a.nc.html

On Tue, Mar 27, 2018 at 10:01 AM, Kyle Wilcox notifications@github.com wrote:

Thanks! I'm not clear about the case of air pressure variables called Baro, where units are PSI; and there's no epic_code att- is that still covered in the way it works now?

Yes, the variable name Baro is enough to match it.

I just looked on the portal, and it appears that 965 has not been replaced/reharvested correctly.

I haven't re-harvested anything yet. Can you update the CF files on your end? We have been trying to automate the process and my end should pick up changes when you make them.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/277#issuecomment-376536179, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHnmt3yXHC4PUkcE0f-rtQeDUqoU_Sks5tikY2gaJpZM4S3gnQ .

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/operations/stg/

[kwilcox]

I need the CF files generated from the changes I pushed 2 hours ago. The older code was converting some pressure data to units outside of the allowable range of pressure so they were not showing up.

[kwilcox]

In the original issue you mentioned the following files should have Air Pressure:

DAUPHIN 9651hwl-a.nc - (actually 9651hwlb-a.nc - with a b). This does have Baro and is now correct in the portal: https://cmgdata.usgsportals.net/#metadata/56432/station

The following don't have a Baro variable. Air Pressure appears to be P_1? This is still be interpreted as sea_water_pressure since it has epic_code: 1 and I'll need to make additional changes unless you can rename these variables in the raw dataset to Baro. 9691hwl-a.nc 9731hwl-a.nc 9741hwl-a.nc 9751hwl-a.nc 9761hwl-a.nc

CHANDELEUR_13 9461hwl-a.nc. This does have Baro and is now correct in the portal: https://cmgdata.usgsportals.net/#metadata/56665/station

Let me know how you would like to proceed with the P_1 variables. I've got the code written already to handle the special case (any P_1 variable in a file named with hwl-a) but wanted to make sure this was better than renaming the variables inside of the files.

[emontgomery-usgs]

For Dauphin, there should only be 1 file with in air data (hwlb), 9651hwlb-a.nc ('b' in the name indicates baro). the other *hwl were buried in sand near the low tide line , so should have seawater variables. There should not be a 9651hwl-a.nc found now.

For Chandeleur, 9461hwlb-a.nc, 9471hwlb-a.nc and 9511hwlb.nc are sequential parts of a timeseries in air from the same location- they should have air temp and pres only. The map will look strange because these were mounted on a tower offshore, so it will look like they're from the water. There was other data from the tower collected under water, but these are not converted to CF so won't be in the portal. 9691hwl-a.nc was buried, and in the latest CF version does NOT have Baro see http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/CF-1.6/DAUPHIN/9691hwl-a.nc.html .

The rb files should all have seawater variables, since they were buried.

Hope this helps!

On Tue, Mar 27, 2018 at 12:50 PM, Kyle Wilcox notifications@github.com wrote:

In the original issue you mentioned the following files should have Air Pressure:

DAUPHIN 9651hwl-a.nc http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/Data/DAUPHIN/9651hwlb-a.nc.html

• (actually 9651hwlb-a.nc - with a b). This does have Baro and is now correct in the portal: https://cmgdata.usgsportals. net/#metadata/56432/station

The following don't have a Baro variable. Air Pressure appears to be P_1? This is still be interpreted as sea_water_pressure since it has epic_code: 1 and I'll need to make additional changes unless you can rename these variables in the raw dataset to Baro. 9691hwl-a.nc http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/Data/DAUPHIN/9691hwl-a.nc.html 9731hwl-a.nc http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/Data/DAUPHIN/9731hwl-a.nc.html 9741hwl-a.nc http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/Data/DAUPHIN/9741hwl-a.nc.html 9751hwl-a.nc http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/Data/DAUPHIN/9751hwl-a.nc.html 9761hwl-a.nc http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/Data/DAUPHIN/9761hwl-a.nc.html

CHANDELEUR_13 9461hwl-a.nc. This does have Baro and is now correct in the portal: https://cmgdata.usgsportals.net/#metadata/56665/station

Let me know how you would like to proceed with the P_1 variables. I've got the code written already to handle the special case (any P_1 variable in a file named with hwl-a) but wanted to make sure this was better than renaming the variables inside of the files.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/277#issuecomment-376595929, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHnmH_q6WDbjBeovjOxMBrFcKjg-N4ks5tim3WgaJpZM4S3gnQ .

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/operations/stg/

[emontgomery-usgs]

@kwilcox, are you getting CF data from silt? Rich had me change the active repo here http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/CF-1.6/DAUPHIN/catalog.html (the new part is silt/usgs/Projects). The old location with emontgomery in the URL is NOT up to date anymore. This screen grab from a few minutes ago suggests the most recent harvest was from the old URL (since seawater temp and press crept back in) Thanks!

[kwilcox]

Yup, that is what happened. The automated process went through pointed at the old location. Fixing...

[kwilcox]

DAUPHIN - 965 is all set: https://cmgdata.usgsportals.net/#metadata/56432/station

I'll have to manually go around and fix the others... let me know if you locate any with these duplicate datasets (old and new server).

[emontgomery-usgs]

I'll check- Are the station ID's or the experiment and platform easier for you to use?

For future reference, the new TDS is not linked to the old path, so we should avoid this issue in future. Thanks!

On Fri, Mar 30, 2018 at 10:16 AM, Kyle Wilcox notifications@github.com wrote:

DAUPHIN - 965 is all set: https://cmgdata.usgsportals. net/#metadata/56432/station

I'll have to manually go around and fix the others... let me know if you locate any with these duplicate datasets (old and new server).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/277#issuecomment-377533526, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHnjguZizjJfddOH7gC78DeUxCxjOQks5tjj47gaJpZM4S3gnQ .

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/operations/stg/

[kwilcox]

I just fixed CHANDELEUR_13 - 946. Experiment and platform are easiest for me to locate them!

[emontgomery-usgs]

Thanks, but... For CHANDELEUR_13, 946, 947, and 951 were all at the same site (location), so we have issue #261 in play. However, moving the time-slider to the dates appropriate for 947 and 951 doesn't display anything. Were they actually harvested?

On Fri, Mar 30, 2018 at 11:17 AM, Montgomery, Ellyn emontgomery@usgs.gov wrote:

I'll check- Are the station ID's or the experiment and platform easier for you to use?

For future reference, the new TDS is not linked to the old path, so we should avoid this issue in future. Thanks!

On Fri, Mar 30, 2018 at 10:16 AM, Kyle Wilcox notifications@github.com wrote:

DAUPHIN - 965 is all set: https://cmgdata.usgsportals.ne t/#metadata/56432/station

I'll have to manually go around and fix the others... let me know if you locate any with these duplicate datasets (old and new server).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/277#issuecomment-377533526, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHnjguZizjJfddOH7gC78DeUxCxjOQks5tjj47gaJpZM4S3gnQ .

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/ operations/stg/

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/operations/stg/

[emontgomery-usgs]

When 947 and 951 get displayed, we can close the issue. The rest I know of now are correct. Thanks!

On Fri, Mar 30, 2018 at 11:22 AM, Montgomery, Ellyn emontgomery@usgs.gov wrote:

Thanks, but... For CHANDELEUR_13, 946, 947, and 951 were all at the same site (location), so we have issue #261 in play. However, moving the time-slider to the dates appropriate for 947 and 951 doesn't display anything. Were they actually harvested?

On Fri, Mar 30, 2018 at 11:17 AM, Montgomery, Ellyn emontgomery@usgs.gov wrote:

I'll check- Are the station ID's or the experiment and platform easier for you to use?

For future reference, the new TDS is not linked to the old path, so we should avoid this issue in future. Thanks!

On Fri, Mar 30, 2018 at 10:16 AM, Kyle Wilcox notifications@github.com wrote:

DAUPHIN - 965 is all set: https://cmgdata.usgsportals.ne t/#metadata/56432/station

I'll have to manually go around and fix the others... let me know if you locate any with these duplicate datasets (old and new server).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/277#issuecomment-377533526, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHnjguZizjJfddOH7gC78DeUxCxjOQks5tjj47gaJpZM4S3gnQ .

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/ operations/stg/

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/ operations/stg/

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 548-8700 x2356 https://stellwagen.er.usgs.gov; https://woodshole.er.usgs.gov/operations/stg/

[kwilcox]

947 and 951 are displayed now!