Reading OpenDAP

[jreniel]

I want to apologize in advance if this is not the right place to ask, if I'm asking a "silly" question or if this question has been asked hundreds of times. I have been porting all of my Python work into Rust with a considerably high degree of success. Currently, when using netCDF4.Dataset in Python, one can directly open an DAP connection directly and just request the variables needed. The netcdf docs for Rust mention the capability of opening a DAP connection directly, but I haven't seen examples, and I have tried doing it in Rust with no success. It seems like I'd need to download the netcdf files and open them locally. Is there a way to mimic the "streaming" behavior of the netCDF4.Dataset using georust/netcdf ? Thanks again! -J.

[mulimoen]

It should work like in python, but it requires the dap support to be included in the library linked (netcdf-c). It should be possible to force this by adding netcdf-sys = { version="*', features =["dap"]} to your Cargo.toml

[jreniel]

Hi @mulimoen I have modified my Cargo.toml as follows:

cargo add netcdf --features static,ndarray

cargo add netcdf-sys --features dap,static,memio

But now I get:

 --- stderr
 thread 'main' panicked at /home/jreniel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/netcdf-sys-0.6.0/build.rs:135:13:
 DAP requested but not found in this installation of netCDF
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Any help is appreciated, Thanks!

[mulimoen]

Could you check if cargo add netcdf-src --features dap enables dap support? If it does I can fix this in a new (patch) release of netcdf-sys

[mulimoen]

I fixed an incompatability for the static and dap features. Instead of the above, could you run cargo update netcdf-sys and try again?

[jreniel]

That fixed it, thanks a lot! You guys are rock stars!

[jreniel]

OK, I reopened because although it compiled, I'm still having one issue. This works in Python:

from netCDF4 import Dataset
nc = Dataset("https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide/k1.hamtide11a.nc")

But the same URL won't work with the Rust NetCDF API:

use netcdf;
let url = "https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide/k1.hamtide11a.nc";
let nc = netcdf::open(&url.to_string());
dbg!(&nc);

Results in:

[src/bctides/src/hamtide.rs:56:9] &url.to_string() = "https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide/k1.hamtide11a.nc"
Error:curl error: SSL peer certificate or SSH remote key was not OK
curl error details: 
Warning:oc_open: Could not read url
[src/bctides/src/hamtide.rs:58:9] &nc = Err(
    Netcdf(
        -68,
    ),
)

Any ideas on how to fix this? Thanks!

[mulimoen]

Bit of a journey finding out what needed to be fixed here. The basic problem is curl not being able to find the certificate.

The solution is to tell it where to find them. This functionality was fixed in netcdf-c, but after the release of 4.9.2, which means another patch was needed, and bubbling this up to the top crate.

[mulimoen]

New versions are up, you might want to update using cargo update netcdf to pull in the fixes

[jreniel]

Thanks for checking this, but unfortunately, even after doing cargo clean and double checking my Cargo.toml:

netcdf = { version = "0.9.2", features = ["static", "ndarray"] }
netcdf-sys = { version = "0.6.2", features = ["dap", "static", "memio"] }

I still get the same error:

[src/bctides/src/hamtide.rs:54:9] &this_url.to_string() = "https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide/q1.hamtide11a.nc"
Error:curl error: SSL peer certificate or SSH remote key was not OK                                                                          
curl error details:                                                                                                                          
Warning:oc_open: Could not read url                                                                                                          
[src/bctides/src/hamtide.rs:56:9] &nc = Err(                                                                                                 
    Netcdf(                                                                                                                                  
        -68,                                                                                                                                 
    ),                                                                                                                                       
)                                                                                                                                            

[mulimoen]

Did you call netcdf::rc::set("HTTP.SSL.CAPATH", "/etc/ssl/certs/").unwrap(); before opening the file?

[jreniel]

Did you call netcdf::rc::set("HTTP.SSL.CAPATH", "/etc/ssl/certs/").unwrap(); before opening the file?

Ah, that explains it! I added it and it worked as expected. Thanks!

Note: I did read https://github.com/georust/netcdf/pull/144 but I misunderstood that this call had been added as an explicit call inside the library.

[mulimoen]

Happy to hear it works for you. I find it strange that the curl library does not use a sane default, I don't know what e.g. ncdump does to handle this.

[mulimoen]

I found https://www.github.com/alexcrichton/curl-rust/pull/446 which describes why a default path is not set for the static build.

[jreniel]

Cool, thanks for the refs!

I hid the call on my struct constructor and made it conditional for #[cfg(unix)] and if it's not already set. This should be sufficient for the time being.

Rust is awesome! The fact that no one else had caught this means Rust is still growing, but the fact that it was fixed so quickly means it's growing fast! Cheers!