DAP4 usage?

[gauteh]

Hi,

I am working on a lightweight DAP-server (dars) and I am wondering if I should go straight for DAP4 (or only support DAP4). In that process I tested out what clients are ready for DAP4. But I cannot seem to get ncdump or python-netcdf4 to work with DAP4. By using ncdump (v4.7.3 from conda or v4.6.0 on ubuntu) I get different segfaults, e.g.:

ncdump -h dap4://remotetest.unidata.ucar.edu/d4ts/testfiles/test_atomic_array.nc
zsh: segmentation fault (core dumped)  ncdump -h

I get segfaults when compiling ncdump from master.

I have tried various other servers (thredds, hyrax running locally or testservers online), or other ways to specify dap4 (using the [dap4]http://... or http://...#dap4), but with various failure modes.

How is this intended to work?

Ref: https://github.com/OPENDAP/libdap4/issues/116#issuecomment-625008828

[DennisHeimbigner]

DAP2 is still widely used. However DAP4 is a better and more powerful protocol. It is also cleaner. DAP2 has a lot of special cases because of its history. So, in your shoes, I would try to build a DAP4 server. Where will your server get its data and in what format? I speculate you plan to store the data in a local file system.

Several other things.

1. The remote test server is down for now because of Java related problems. So it cannot be used. That is probably why the crashes (though it should fail more gracefully).
2. You might see if you can talk to the opendap dap4 server; URL escapes me at the moment.
3. DAP4 usage has been minimal to date (but is growing), so the code has notbeen tested as thoroughly as I would like. =Dennis Heimbigner Unidata

[gauteh]

Dennis Heimbigner writes on May 7, 2020 20:25:

DAP2 is still widely used. However DAP4 is a better and more powerful protocol. It is also cleaner. DAP2 has a lot of special cases because of its history. So, in your shoes, I would try to build a DAP4 server. Where will your server get its data and in what format? I speculate you plan to store the data in a local file system.

Several other things.

1. The remote test server is down for now because of Java related problems. So it cannot be used. That is probably why the crashes (though it should fail more gracefully).
2. You might see if you can talk to the opendap dap4 server; URL escapes me at the moment.
3. DAP4 usage has been minimal to date (but is growing), so the code has notbeen tested as thoroughly as I would like. =Dennis Heimbigner Unidata

Thanks for your answer. I would like to go straight for DAP4, but I can't seem to get it to work netcdf-c based clients. The following attempts all fail (netcdf 4.6.0):

$ ncdump -h dap4://test.opendap.org/opendap/data/nc/coads_climatology.nc
$ ncdump -h [dap4]http://test.opendap.org/opendap/data/nc/coads_climatology.nc

(trying other files gives various other errors)

while regular DAP2 works:

$ ncdump -h http://test.opendap.org/opendap/data/nc/coads_climatology.nc

http://test.opendap.org/opendap/data/nc/coads_climatology.nc.dmr is present and looks alright.

I am building a regular file-based server at first, but it will be easy to modify into using cloud based object storage. It will only support DAP/* with perhaps a very basic index. No WMS, catalog, etc. And probably only basic constraints at first (slicing, not filtering). The idea being that it can be plugged as an async and lightweight server for data-transfer, but not try re-invent the other parts that Thredds and Hyrax is already doing.

Regards, Gaute

[DennisHeimbigner]

can you try our latest release?

Sent from my iPad

On May 8, 2020, at 1:57 AM, Gaute Hope notifications@github.com wrote:

Dennis Heimbigner writes on May 7, 2020 20:25:

DAP2 is still widely used. However DAP4 is a better and more powerful protocol. It is also cleaner. DAP2 has a lot of special cases because of its history. So, in your shoes, I would try to build a DAP4 server. Where will your server get its data and in what format? I speculate you plan to store the data in a local file system.

Several other things.

1. The remote test server is down for now because of Java related problems. So it cannot be used. That is probably why the crashes (though it should fail more gracefully).
2. You might see if you can talk to the opendap dap4 server; URL escapes me at the moment.
3. DAP4 usage has been minimal to date (but is growing), so the code has notbeen tested as thoroughly as I would like. =Dennis Heimbigner Unidata

Thanks for your answer. I would like to go straight for DAP4, but I can't seem to get it to work netcdf-c based clients. The following attempts all fail (netcdf 4.6.0):

$ ncdump -h dap4://test.opendap.org/opendap/data/nc/coads_climatology.nc
$ ncdump -h [dap4]http://test.opendap.org/opendap/data/nc/coads_climatology.nc

(trying other files gives various other errors)

while regular DAP2 works:

$ ncdump -h http://test.opendap.org/opendap/data/nc/coads_climatology.nc

http://test.opendap.org/opendap/data/nc/coads_climatology.nc.dmr is present and looks alright.

I am building a regular file-based server at first, but it will be easy to modify into using cloud based object storage. It will only support DAP/* with perhaps a very basic index. No WMS, catalog, etc. And probably only basic constraints at first (slicing, not filtering). The idea being that it can be plugged as an async and lightweight server for data-transfer, but not try re-invent the other parts that Thredds and Hyrax is already doing.

Regards, Gaute — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[gauteh]

On b490c457e9d3356426f116437a089a7ec7640c13:

$ ncdump/ncdump --version
...
netcdf library version 4.8.0-development of May  9 2020 09:32:42 $

$ ncdump/ncdump -h dap4://test.opendap.org/opendap/data/nc/coads_climatology.nc
ncdump/ncdump: dap4://test.opendap.org/opendap/data/nc/coads_climatology.nc:
dap4://test.opendap.org/opendap/data/nc/coads_climatology.nc: NetCDF:
Variable has no data in DAP request

$ ncdump/ncdump -h
[dap4]http://test.opendap.org/opendap/data/nc/coads_climatology.nc
ncdump/ncdump: [dap4]http://test.opendap.org/opendap/data/nc/coads_climatology.nc:
[dap4]http://test.opendap.org/opendap/data/nc/coads_climatology.nc:
NetCDF: Variable has no data in DAP request

Regards, Gaute

Dennis Heimbigner writes on May 8, 2020 17:15:

can you try our latest release?

Sent from my iPad

On May 8, 2020, at 1:57 AM, Gaute Hope notifications@github.com wrote:

Dennis Heimbigner writes on May 7, 2020 20:25:

DAP2 is still widely used. However DAP4 is a better and more powerful protocol. It is also cleaner. DAP2 has a lot of special cases because of its history. So, in your shoes, I would try to build a DAP4 server. Where will your server get its data and in what format? I speculate you plan to store the data in a local file system.

Several other things.

1. The remote test server is down for now because of Java related problems. So it cannot be used. That is probably why the crashes (though it should fail more gracefully).
2. You might see if you can talk to the opendap dap4 server; URL escapes me at the moment.
3. DAP4 usage has been minimal to date (but is growing), so the code has notbeen tested as thoroughly as I would like. =Dennis Heimbigner Unidata

Thanks for your answer. I would like to go straight for DAP4, but I can't seem to get it to work netcdf-c based clients. The following attempts all fail (netcdf 4.6.0):

$ ncdump -h dap4://test.opendap.org/opendap/data/nc/coads_climatology.nc
$ ncdump -h [dap4]http://test.opendap.org/opendap/data/nc/coads_climatology.nc

(trying other files gives various other errors)

while regular DAP2 works:

$ ncdump -h http://test.opendap.org/opendap/data/nc/coads_climatology.nc

http://test.opendap.org/opendap/data/nc/coads_climatology.nc.dmr is present and looks alright.

I am building a regular file-based server at first, but it will be easy to modify into using cloud based object storage. It will only support DAP/* with perhaps a very basic index. No WMS, catalog, etc. And probably only basic constraints at first (slicing, not filtering). The idea being that it can be plugged as an async and lightweight server for data-transfer, but not try re-invent the other parts that Thredds and Hyrax is already doing.

Regards, Gaute — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/Unidata/netcdf-c/issues/1712#issuecomment-625864657

[DennisHeimbigner]

Ok, Let me see what is going on.