Remote ncdump/DAP command works on OSX/Linux, fails on Windows

WardF commented 5 years ago

As reported, the following command works on OSX/Linux and fails on Windows using MSVC. It was reported as a failure in 4.6.1 but I'm seeing the same issue with 4.7.0.

I'm seeing the issue in the Windows 10 CMD shell as well as an MSYS2 bash prompt.

$ ./ncdump https://www.ncei.noaa.gov/thredds/dodsC/namanl/201609/20160929/namanl_218_20160929_0000_000.grb

curl error details:
ncdump: https://www.ncei.noaa.gov/thredds/dodsC/namanl/201609/20160929/namanl_218_20160929_0000_000.grb: https://www.ncei.noaa.gov/thredds/dodsC/namanl/201609/20160929/namanl_218_20160929_0000_000.grb: NetCDF: I/O failure

WardF commented 5 years ago

@DennisHeimbigner Does this command work for you on Windows in a cygwin shell? I suspect the issue is something to do with the string parsing on Windows, but I'm not certain. I'm setting up a linux subsystem environment to test to see if that works.

WardF commented 5 years ago

Using nccopy to copy the file locally, ncdump now works as expected.

DennisHeimbigner commented 5 years ago

Works ok under CYGWIN bash. Couple of possibilities:

the url might need escaping to work
it is possible that the winpath code is getting invoked when it should not be.

DennisHeimbigner commented 5 years ago

So, another thing to try is to prefix the url with [log][show=fetch] to get some extra debug info.

WardF commented 5 years ago

@DennisHeimbigner Using [log][show=fetch] gives the following:

$ ./ncdump [log][show=fetch]http://www.ncei.noaa.gov/thredds/dodsC/namanl/201609/20160929/namanl_218_20160929_0000_000.grb Log:fetch: http://www.ncei.noaa.gov/thredds/dodsC/namanl/201609/20160929/namanl_218_20160929_0000_000.grb.dds Log:curl error: Unsupported protocol curl error details: Log:oc_open: Could not read url Log:fetch complete. D:\netcdf-c\build\ncdump\ncdump.exe: [log][show=fetch]http://www.ncei.noaa.gov/thredds/dodsC/namanl/201609/20160929/namanl_218_20160929_0000_000.grb: [log][show=fetch]http://www.ncei.noaa.gov/thredds/dodsC/namanl/201609/20160929/namanl_218_20160929_0000_000.grb: NetCDF: I/O failure

WardF commented 5 years ago

@DennisHeimbigner I just noticed that the original report was for an https url. Is there a way to see if the local curl install supports https?

WardF commented 5 years ago

I'm wondering if the issue is the https:// url and a libcurl that doesn't support https. I've installed openssl and am rebuilding my libcurl with openssl support. I'll see if this makes a difference.

DennisHeimbigner commented 5 years ago

Well, I did see when using the [log] prefix a complaint from curl about unsupported protocol. So that may be it.

DennisHeimbigner commented 5 years ago

And yes, if you do curl-config --protocols it should list supported protocols

WardF commented 5 years ago

Thanks @DennisHeimbigner . I installed openssl on Windows, rebuilt libcurl using the Visual Studio compiler, and was then able to access the url. Mystery solved!

WardF commented 5 years ago

Reopened because additional information has come to light.

WardF commented 5 years ago

I finally was able to get openssl 1.0.2l installed, now I need to get libcurl 7.60.0. OpenSSL 1.0.2l and libcurl 7.55.1 do not encounter an error.

WardF commented 5 years ago

Checking with libcurl 7.60.0 and OpenSSL 1.0.2l on my system, using the current master dev branch, fails to reveal the reported issue. Next check will be netCDF 4.6.1.

WardF commented 5 years ago

I will double check my work but I am not able to duplicate this issue with netCDF 4.6.1, OpenSSL 1.0.2l and libcurl 7.60.0 on my system as of this moment. I'll try to do a clean install of everything just to minimize the chance of user error, and report on that.

ellenjohnson commented 5 years ago

Hi Ward and Dennis -- thanks for reopening this and investigating. Just letting you know we had another user report this exact same issue with this GRIB OPeNDAP endpoint: "https://thredds-test.unidata.ucar.edu/thredds/dodsC/grib/NCEP/WW3/Regional_US_East_Coast/Best" Same error -- Invalid argument (NC_EINVAL). Same configuration -- only on Windows, only on GRIB endpoints, only in netcdf 4.6.1 + curl 7.60.0 + openssl 1.0.2.l.

WardF commented 5 years ago

@DennisHeimbigner is it possible this is a TDS issue? I'm putting together an email and some log files right now, but more work to diagnose this is needed before I have any concrete conclusions.

DennisHeimbigner commented 5 years ago

It is possible, but if so, then it should be easy to show by testing these 3 URLS in your web browser.

It is also possible that the dds or das being produced by tds is not being properly parsed by netcdf-c. I have not looked at the output in detail. Also, the fact that it works for some cases, makes this a low probability.

DennisHeimbigner commented 5 years ago

I have created a separate branch called windap.tmp that turns on a bunch of debug output. Please try to use it to see if the error you are seeing shows up. It should be sufficient if you send the output from this:

..../netcdf-c/ncdump/ncdump -h https://thredds-test.unidata.ucar.edu/thredds/dodsC/grib/NCEP/WW3/Regional_US_East_Coast/Best

I used the -h flag to keep the output down, but if the error does not show, then try again with the -h flag removed.

ellenjohnson commented 5 years ago

Thanks Dennis -- I've been trying to build the special branch on Windows and it's a bit slow-going. I got it to at least the build stage but it's failing with some weird errors, I think it's our configuration system here, though I tried to build it standalone and got build errors there too. I'll let you know when I make more progress.

ellenjohnson commented 5 years ago

I am so close to getting this building and am down to three link errors, I think all due to m4 not being found.
During CMake configure, I get: m4 not found Cannot find source files putget.c and test_get.c Cannot determine linker language for targets netcdf3 and nc_test Then during CMake build, I get three unresolved linker errors: nc_initialize.obj : error LNK2019: unresolved external symbol NC3_initialize referenced in function nc_initialize [X:\fromDennis\netcdf\build\liblib\netcdf.vcxproj] nc_initialize.obj : error LNK2019: unresolved external symbol NC3_finalize referenced in function nc_finalize [X:\fromDennis\netcdf\build\liblib\netcdf.vcxproj] dfile.obj : error LNK2001: unresolved external symbol NC3_dispatch_table [X:\fromDennis\netcdf\build\liblib\netcdf.vcxproj] X:\fromDennis\netcdf\build\liblib\netcdf.dll : fatal error LNK1120: 3 unresolved externals [X:\fromDennis\netcdf\build\liblib\netcdf.vcxproj]

I tried adding the path to Windows version of m4 in my system path environment variable, but that didn't work. Any thoughts? Thanks!

ellenjohnson commented 5 years ago

Update: got past m4 thing by starting up a new command window (oops, I had forgot I need to do that after changing system path). So I no longer get unresolved symbol link errors, but am getting build errors with libsrc\posixio.c:

"X:\fromDennis\netcdf\build\ALL_BUILD.vcxproj" (default target) (1) -> "X:\fromDennis\netcdf\build\examples\C\C_tests_format.vcxproj" (default target) (3) -> "X:\fromDennis\netcdf\build\liblib\netcdf.vcxproj" (default target) (4) -> "X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj" (default target) (8) -> (ClCompile target) -> x:\fromdennis\netcdf\libsrc\posixio.c(286): error C2065: 'ssize_t': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(286): error C2146: syntax error: missing ';' before identifier 'partial' [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(286): error C2065: 'partial': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(310): error C2065: 'partial': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(311): error C2065: 'partial': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(313): error C2065: 'partial': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(314): error C2065: 'partial': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(316): error C2065: 'partial': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(340): error C2065: 'ssize_t': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(340): error C2146: syntax error: missing ';' before identifier 'nread' [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(340): error C2065: 'nread': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(382): error C2065: 'nread': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(383): error C2065: 'nread': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(386): error C2065: 'nread': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj] x:\fromdennis\netcdf\libsrc\posixio.c(386): error C2065: 'ssize_t': undeclared identifier [X:\fromDennis\netcdf\build\libsrc\netcdf3.vcxproj]

Any suggestions? Thanks!

ellenjohnson commented 5 years ago

I finally figured out all my CMake build and link problems with the netcdf-c windap.tmp branch Dennis setup. I built it using the problematic combo of curl 7.60.0 and openssl 1.0.2.l.

As Dennis recommended, I tried this endpoint with -h flag: ncdump -h https://thredds-test.unidata.ucar.edu/thredds/dodsC/grib/NCEP/WW3/Regional_US_East_Coast/Best

This gives a ton of output, of which I included a small snippet below. I emailed Dennis and Ward the full output in a txt file. I also verified using MATLAB R2019a that this gives same NC_EINVAL error that we see with the other OPeNDAP GRIB endpoints.

$dapthrow: err=-125 line=2559 file=x:\fromdennis\netcdf\libdap2\ncd2dispatch.c dimensions: lat = 331 ; lon = 301 ; ordered_sequence_of_data = 2 ; time = 287 ; variables: float lat(lat) ; $dapthrow: err=-125 line=2559 file=x:\fromdennis\netcdf\libdap2\ncd2dispatch.c lat:units = "degrees_north" ; $dapthrow: err=-125 line=2559 file=x:\fromdennis\netcdf\libdap2\ncd2dispatch.c lat:_CoordinateAxisType = "Lat" ; float lon(lon) ; $dapthrow: err=-125 line=2559 file=x:\fromdennis\netcdf\libdap2\ncd2dispatch.c lon:units = "degrees_east" ; $dapthrow: err=-125 line=2559 file=x:\fromdennis\netcdf\libdap2\ncd2dispatch.c lon:_CoordinateAxisType = "Lon" ; double time(time) ; $dapthrow: err=-125 line=2559 file=x:\fromdennis\netcdf\libdap2\ncd2dispatch.c time:units = "Hour since 2019-06-02T00:00:00Z" ; $dapthrow: err=-125 line=2559 file=x:\fromdennis\netcdf\libdap2\ncd2dispatch.c time:standard_name = "time" ; $dapthrow: err=-125 line=2559 file=x:\fromdennis\netcdf\libdap2\ncd2dispatch.c time:long_name = "GRIB forecast or observation time" ;

DennisHeimbigner commented 5 years ago

To refresh my memory, what version of netcdf-c were you using to generate the output? 4.6.1?

ellenjohnson commented 5 years ago

To generate that huge output file with the extra debug info, I built whatever netcdf version is in the windap.tmp branch. I assume that's your latest development version (4.7 or similar)? I built it fron scratch using curl 7.60.0 and openssl 1.0.2l (the seemingly problematic combo). I reported the initial bug using our current shipping version of netcdf (4.6.1). That was before you setup the windap.tmp branch.

DennisHeimbigner commented 5 years ago

Oh yes Unfortunately that output is just not precise enough. Are you in a position to do this in visual studio?

Set a breakpoint in the function dapbreakpoint() (in file libdap2/dapdebug.c)
When that breakpoint is hit and the argument is -125, then get a list of the callstack to send to me.

ellenjohnson commented 5 years ago

My Visual Studio chops are a bit rusty as I'm more of a Linux person. I was able to generate the above output showing error -125 by building and installing netcdf from windap.tmp branch completely from Windows command line using CMake, then running ncdump.exe from command line. I'm assuming I can try to run the ncdump solution file in Visual Studio?

DennisHeimbigner commented 5 years ago

That should work. I use the following cheat sheet to remind me how to do this, it might be helpful to you also. The big issue is the PATH= arguments. The first item points to the netcdf-c library, the second points to where you have the hdf5 and libcurl etc libraries. Anyway for what it is worth:

Start VS
Open File->Open->ProjectSolns build/ZERO_CHECK.vcxproj or build/ALL_BUILD.vcxproj
Ensure the mode is set to Debug
From the Solution explorer window on the right, choose the program to run.
Make the program the default: Project>Set as start project
Set Env and cmd args under Project>Properties>Debugging Verify that the paths exist Env: PATH=d:\git\netcdf-c\build\liblib;c:\tools\nccmake\bin;%PATH% Cmd args: whatever, probably should use absolute paths e.g. for ncdump -h https://thredds-test.unidata.ucar.edu/thredds/dodsC/grib/NCEP/WW3/Regional_US_East_Coast/Best
Try go to Tools->Options->Debugging->Symbols and select checkbox "Microsoft Symbol Servers", Visual Studio will download PDBs automatically.
Build/clean and Build/rebuild project
Debug->Start Debugging

DennisHeimbigner commented 5 years ago

Ward- are you in a position to get me that stack trace?

WardF commented 5 years ago

Sorry for my silence on this, I've been working with our intern trying to sort out some of the C++ issues. Let me take a look. I should be able to get the stack trace for you.

WardF commented 5 years ago

@DennisHeimbigner you need this specifically from the windap.tmp branch?

DennisHeimbigner commented 5 years ago

No, but for whatever branch you use, you need to modify libdap2/dapdebug.h to turn on some flags by changing

if 0

define DAPDEBUG 1

define OCDEBUG 1

endif

to

if 1

define DAPDEBUG 1

define OCDEBUG 1

endif

This will make dapbreakpoint be defined.

WardF commented 5 years ago

Ok, thanks @DennisHeimbigner. I've got the branch built and the change you specified has been made. I'm still accessing the file without issue (e.g., I never hit a breakpoint at dapbreakpoint() where err == -125. Let me try the libraries @ellenjohnson provided and see if that changes things.

ellenjohnson commented 5 years ago

Thanks Dennis and Ward! Ward, I know you just tried the windap.tmp branch without error -- was that linking with curl 7.60.0 and openssl 1.0.2l? Or newer versions?

Just a reminder the libraries I sent over to you are our shipping R2019a binaries for netcdf 4.6.1, curl 7.60.0, and openssl 1.0.2l. (Note curl was built using OpenSSL and not WinSSL. A colleague here tried to swap out OpenSSL with WinSSL and got the same error).

DennisHeimbigner commented 5 years ago

So I think I have determined that the -125 error is a red-herring. However, in looking at the ncdump_fullOutput.txt, I did not see any occurrence of NC_EINVAL (-36), which if I understand, was the original problem.

ellenjohnson commented 5 years ago

@DennisHeimbigner, I noticed that also -- that i didn't see NC_EINVAL in the output, but the windap.tmp gave so much output that I didn't know if it successfully read all the variables or not. Is there any way to build the windap.tmp branch to just have regular ncdump output to see if I get the original error? I haven't had much success debugging in VisualStudio, but maybe it's a moot point if I'm not getting the NC_EINVAL error.

@WardF: I'm hoping you can reproduce the original error with the binaries I sent you.

@Both: Is there a place in the netcdf code (i.e., libdap2) where you suspect the error might be coming from? If so I can try to put some printf statements in, or try more Visual Studio debugging.

WardF commented 5 years ago

@ellenjohnson It was with the most recent versions. I will test the version you provided next. My laptop has spent the day running a surface scan because of an error, I will reproduce this as soon as I can launch my Windows environment; this is at the top of my priority list right now!

DennisHeimbigner commented 5 years ago

windap.tmp minus all the debug output should be the same as the current master branch. So you could see if the master branch fails

DennisHeimbigner commented 5 years ago

Oh, also, at this point, I just do not have enough info to even figure out where to put such print statements.

DennisHeimbigner commented 5 years ago

If we are lucky, Ward will have a failing version under visual studio and during our local meeting tomorrow maybe we can debug the problem.

ellenjohnson commented 5 years ago

I don't know if this helps -- my Mapping Toolbox colleague wrote a script to test 61 OPeNDAP THREDDS servers hosting GRIB data using our ncinfo function in MATLAB R2019a on Windows. Of the 61, only 7 worked, and 54 failed (NC_EINVAL). They all worked in R2018b. I'm going to try the same servers using ncdump with same libraries in R2019a (the ones I sent to Ward). I'm assuming they'll have same behavior because that's what I saw before using ncdump with the other endpoint. Here are the findings in case something jumps out at you about the ones that passed vs. failed:

Passed: "https://thredds.unidata.ucar.edu/thredds/dodsC/grib/NCEP/MRMS/NLDN/TP" "https://thredds.unidata.ucar.edu/thredds/dodsC/grib/NCEP/MRMS/RotationTrackML/TP" "https://thredds.unidata.ucar.edu/thredds/dodsC/grib/NCEP/MRMS/RotationTrack/TP" "https://thredds.unidata.ucar.edu/thredds/dodsC/grib/NCEP/MRMS/BaseRef/TP" "https://thredds.unidata.ucar.edu/thredds/dodsC/grib/NCEP/MRMS/Precip/TP" "https://thredds.unidata.ucar.edu/thredds/dodsC/grib/NCEP/MRMS/Radar/TP" "https://thredds.unidata.ucar.edu/thredds/dodsC/grib/NCEP/MRMS/Model/TP"

WardF commented 5 years ago

@ellenjohnson We are still trying to duplicate this issue, and @DennisHeimbigner just pointed out that the code you are calling does not require HDF5. Do you still see the issue if you compile netCDF without HDF5 support?

WardF commented 5 years ago

@ellenjohnson That isn't a high priority question, it shouldn't matter. But we are still not able to duplicate this issue, even running against the libraries you have provided. I will continue to try to duplicate this. I will also try to create a distributable file I can send to you and hopefully see if you encounter the same (lack of) issue.

WardF commented 5 years ago

Summary (note to self)

openssl 1.0.2l
libcurl 7.6.0
hdf5 1.8.12 (shouldn't matter since this is DAP2, but for completeness sake)

WardF commented 5 years ago

@ellenjohnson Attached is a zip file. Within is a version of ncdump 4.6.1 compiled today, against libcurl 7.6.0 and openssl 1.0.2l. The openssl runtimes are the ones you kindly provided. The HDF5 libraries were not used, as I built it without netcdf4 support (which shouldn't matter for this).

I'm able to run ncdump -h https://thredds-test.unidata.ucar.edu/thredds/dodsC/grib/NCEP/WW3/Regional_US_East_Coast/Best without issue using the ncdump.exe contained in this zip file. Can you try running it, and let me know if you see an error?

binaries-forellen.zip

Thanks!

ellenjohnson commented 5 years ago

@WardF: Thank you for the binaries, and yes, I can run the ncdump command on that endpoint without error! This is weird, especially since you can't reproduce with the binaries I sent.
Questions and comments:

I think you mean libcurl 7.60.0 (not 7.6.0, right?)
What version of Visual Studio did you use -- Visual Studio 2017 or 2015? (I see the USING-Debug-x64-VS15 file including in the binaries you sent, but not sure if that's the VS product name or version number.)
I can try again to build netcdf, this time without hdf5. Have you tried it? You guys can probably do it a lot faster than me. Yes we are using hdf5 1.8.12.
Can you send me your libnetcdf.settings file or any other info that gives me details of how you built netcdf 4.6.1 especially with regard to curl and openssl? Thanks!

Unidata / netcdf-c

Remote ncdump/DAP command works on OSX/Linux, fails on Windows #1393

if 0

define DAPDEBUG 1

define OCDEBUG 1

endif

if 1

define DAPDEBUG 1

define OCDEBUG 1

endif