search

dankelley / oce

R package for oceanographic processing
http://dankelley.github.io/oce/
GNU General Public License v3.0
143 stars 42 forks source link

read.netcdf does not read time variable from UHDAS netCDF #2259

Closed marecotec closed 2 hours ago

marecotec commented 2 hours ago

Hi all,

Thanks in advance for checking out my issue.

(1) a sentence or two describing the issue I am working with ship-acquired ADCP data from a UHDAS system, and then I processed the data in CODAS. Finally, I exported a netCDF of the ADCP data using the CODAS python script adcp_nc.py so I could bring this data to R.

I used read.netcdf() from oce with the expectation that I can work with the data and cut, plot and query it as I would a normal ADP object (maybe with some finessing), but for some reason, when I run summary on the imported netCDF, my time variable is missing and hence I can't query it as expected through R. If I read the file using ncdf4, then I can access the time variable, which I can also see in NASA Panoply.

I searched extensively and wasn't able to find any direction to figure out my issue, maybe I am too close and I've missed something obvious. I have emailed the data to Dan and Clark as indicated in the notes as it hasn't been made open access yet.

(2) a pared-down code example

library(oce)
library(ncdf4)

d <- read.netcdf("wh300.nc")
summary(d)
d@data$time # I get NULL

c <- nc_open("wh300.nc")
c$dim$time$vals # I get [1] 272.9971 272.9985 272.9998 273.0012 273.0026 273.0040

(3) the output from sessionInfo(). R version 4.4.2 (2024-10-31) Platform: aarch64-apple-darwin20 Running under: macOS Sequoia 15.1

dankelley commented 2 hours ago

The R code

library(oce)
f <- "~/Downloads/wh300.nc"
d <- read.netcdf(f)
cm <- colormap(zlim=quantile(d[["u"]], c(0.01, 0.99), na.rm=TRUE))
png("nc.png")
imagep(d[["time"]], d[["depth"]][,1], t(d[["u"]]), colormap=cm)

yields as below, with the "develop" branch, latest commit.

Does this seem reasonable? (Also, my summary(d) shows time.) Are you using the version on CRAN or the version in the "develop" branch?

PS. I won't see email again until tomorrow (Halifax is 1 hour east of NYC).

nc

marecotec commented 2 hours ago

Ah, thanks Dan, I am on CRAN, oce version 1.8-3, I will move onto the develop version next, here's my summary(d) from 1.8-3:

> summary(d)
* Data Overview

                               Min.      Mean   Max.    Dim.    NAs
    lon                     -89.089   -87.465 -86.86    3961      0
    lat                      29.395    29.797 30.351    3961      0
    depth                      9.08     78.08 147.08 70x3961      0
    u                       -3.6754 -0.059641  3.738 70x3961 192949
    v                       -2.4752 -0.083724 2.4604 70x3961 192949
    amp                          41    71.276    231 70x3961      0
    pg                            0    30.811    100 70x3961      0
    pflag                         0    3.7513      6 70x3961      0
    heading                 -179.89   0.79049 179.95    3961      0
    tr_temp                  27.084    28.984 29.754    3961      0
    num_pings                    75    75.001     76    3961      0
    uship                   -5.4196   0.01776 5.4258    3961      1
    vship                   -5.0293 -0.025475 4.7256    3961      1
    index_config_start            0      1950   3900      14      0
    ensemble_seconds            120       120    120      14      0
    num_depth_bins               70        70     70      14      0
    transducer_depth              3         3      3      14      0
    depth_bin_length              2         2      2      14      0
    pulse_length               1.99    1.9993      2      14      0
    blank_length                  4         4      4      14      0
    ping_interval          0.028284   0.13969 1.5797      14      0
    transducer_orientation   -21.39    -21.39 -21.39      14      0

* Processing Log

    - 2024-11-14 23:02:15 UTC: `Create oce object`
    - 2024-11-14 23:02:15 UTC: `read.netcdf("wh300.nc")`

Plot looks exactly as expected.

Thanks as ever, Andy

dankelley commented 2 hours ago

Sounds good. I did some work on netcdf format a while ago, because more and more data are becoming available in that format.

I have found that some files have unusual names for the data entries, so I also added a new function called renamer() that can help with that. It is designed to have dictionaries defined in files, not in code, so that users can customize things easily, and also can contribute entries back to the oce project (if the entries are 'official' in some sense, and not just something used in a single working group).

I don't exactly know how to set up to build from github, but the wiki for oce has some instructions written by @richardsc and I imagine they are up-to-date for the various likely platforms. We don't update oce often on cran because that is frowned upon there.

marecotec commented 2 hours ago

Awesome, I am back in play and making progress. Agree on netCDF, its becoming a huge format now. Using the develop branch, my summary() output contains a lot more information, and time is one of the variables! I'll have a look at developing a dictionary to help support the UHDAS/CODAS data pipeline into oce as I work through this dataset. I did see some examples as I was searching through Github. As ever, I greatly appreciate the work that you and Clark do on this package. 

> d <- read.netcdf("wh300.nc")
Warning messages:
1: In read.netcdf("wh300.nc") :
  "depth" unit "meter" is not recognized; please report an issue
2: In read.netcdf("wh300.nc") :
  "u" unit "meter second-1" is not recognized; please report an issue
3: In read.netcdf("wh300.nc") :
  "v" unit "meter second-1" is not recognized; please report an issue
4: In read.netcdf("wh300.nc") :
  "uship" unit "meter second-1" is not recognized; please report an issue
5: In read.netcdf("wh300.nc") :
  "vship" unit "meter second-1" is not recognized; please report an issue
6: In read.netcdf("wh300.nc") :
  "time" unit "days since 2024-01-01 00:00:00" is not recognized; please report an issue
> summary(d)
* Data Overview

                                          Min.     Mean      Max.   Dim.      NAs    OriginalName            
    lon [°E]                              -89.089  -87.465   -86.86 3961      0      "lon"                   
    lat [°N]                              29.395   29.797    30.351 3961      0      "lat"                   
    depth [meter]                         9.08     78.08     147.08 "70x3961" 0      "depth"                 
    u [meter second-1]                    -3.6754  -0.059641 3.738  "70x3961" 192949 "u"                     
    v [meter second-1]                    -2.4752  -0.083724 2.4604 "70x3961" 192949 "v"                     
    amp                                   41       71.276    231    "70x3961" 0      "amp"                   
    pg                                    0        30.811    100    "70x3961" 0      "pg"                    
    pflag                                 0        3.7513    6      "70x3961" 0      "pflag"                 
    heading [°]                           -179.89  0.79049   179.95 3961      0      "heading"               
    tr_temp [°C]                          27.084   28.984    29.754 3961      0      "tr_temp"               
    num_pings                             75       75.001    76     3961      0      "num_pings"             
    uship [meter second-1]                -5.4196  0.01776   5.4258 3961      1      "uship"                 
    vship [meter second-1]                -5.0293  -0.025475 4.7256 3961      1      "vship"                 
    index_config_start                    0        1950      3900   14        0      "index_config_start"    
    ensemble_seconds [s]                  120      120       120    14        0      "ensemble_seconds"      
    num_depth_bins                        70       70        70     14        0      "num_depth_bins"        
    transducer_depth [m]                  3        3         3      14        0      "transducer_depth"      
    depth_bin_length [m]                  2        2         2      14        0      "depth_bin_length"      
    pulse_length [m]                      1.99     1.9993    2      14        0      "pulse_length"          
    blank_length [m]                      4        4         4      14        0      "blank_length"          
    ping_interval [s]                     0.028284 0.13969   1.5797 14        0      "ping_interval"         
    transducer_orientation [°]            -21.39   -21.39    -21.39 14        0      "transducer_orientation"
    time [days since 2024-01-01 00:00:00] 273      275.75    278.5  3961      0      "time"                  

* Processing Log

    - 2024-11-15 00:04:14 UTC: `Create oce object`
    - 2024-11-15 00:04:14 UTC: `read.netcdf("wh300.nc")`