Closed idixon closed 5 years ago
piyushrpt
commented
5 years ago gdalinfo on your local machine is correctly picking up the NETCDF driver whereas the version with vsicurl seems to be using the HDF5 driver, which is a fallback option when the file is not recognized by the NETCDF driver. Could you provide the output of the gdalinfo vsicurl version with CPL_DEBUG=ON set.
rouault
commented
5 years ago /vsicurl/ support in the netCDF driver requires a recent enough version of libnetcdf which installs the netcdf_mem.h header file
idixon
commented
5 years ago @piyushrpt
bash-4.2# gdalinfo /vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc HTTP: libcurl/7.61.1 OpenSSL/1.0.2k zlib/1.2.8 libidn2/0.16 libpsl/0.6.2 (+libicu/50.1.2) libssh2/1.4.2 nghttp2/1.21.1 VSICURL: GetFileSize(https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)=63099985 response_code=200 VSICURL: Downloading 0-16383 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 VSICURL: GetFileList(/vsicurl/https://s3.amazonaws.com/idixon) VSICURL: GetFileSize(https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.aux)=0 response_code=404 VSICURL: GetFileSize(https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.AUX)=0 response_code=404 VSICURL: Downloading 54558720-54575103 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 VSICURL: Downloading 63078400-63094783 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 VSICURL: Downloading 54624256-54640639 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 VSICURL: Downloading 63094784-63099984 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 VSICURL: Downloading 54575104-54591487 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 VSICURL: Downloading 54591488-54624255 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 VSICURL: Downloading 63045632-63062015 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 VSICURL: Downloading 32260096-32276479 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 VSICURL: Downloading 52363264-52379647 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 VSICURL: Downloading 63062016-63078399 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)... VSICURL: Got response_code=206 HDF5: Skipping unsupported //productBoundingBox of type Unknown HDF5: Skipping unsupported //science/grids/corrections/derived/ionosphere/ionosphere of type Unknown HDF5: Skipping unsupported //science/grids/corrections/external/tides/solidEarthTide of type Unknown HDF5: Skipping unsupported //science/grids/corrections/external/troposphere/troposphereHydrostatic of type Unknown HDF5: Skipping unsupported //science/grids/corrections/external/troposphere/troposphereWet of type Unknown HDF5: Skipping unsupported //science/radarMetaData/DEM of type Unknown HDF5: Skipping unsupported //science/radarMetaData/ISCEversion of type Unknown HDF5: Skipping unsupported //science/radarMetaData/azimuthZeroDopplerEndTime of type Unknown HDF5: Skipping unsupported //science/radarMetaData/azimuthZeroDopplerStartTime of type Unknown HDF5: Skipping unsupported //science/radarMetaData/inputSLC/reference/L1InputGranules of type Unknown HDF5: Skipping unsupported //science/radarMetaData/inputSLC/reference/orbitType of type Unknown HDF5: Skipping unsupported //science/radarMetaData/inputSLC/secondary/L1InputGranules of type Unknown HDF5: Skipping unsupported //science/radarMetaData/inputSLC/secondary/orbitType of type Unknown HDF5: Skipping unsupported //science/radarMetaData/missionID of type Unknown HDF5: Skipping unsupported //science/radarMetaData/productType of type Unknown HDF5: Skipping unsupported //science/radarMetaData/unwrapMethod of type Unknown GDAL: GDALOpen(/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc, this=0x25f7050) succeeds as HDF5. Driver: HDF5/Hierarchical Data Format Release 5 Files: /vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc Size is 512, 512 Metadata: author=David Bekaert Conventions=CF-1.6 crs_polygon_long_name=crs_polygon crs_polygon_spatial_ref=GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]] crs_polygon_standard_name=crs_polygon institution=Jet Propulsion Laboratory matchup_CLASS=DIMENSION_SCALE matchup_NAME=This is a netCDF dimension but not a netCDF variable. 0 matchup_REFERENCE_LIST= matchupNetcdf4Dimid=0 ogr_geometry_field=productBoundingBox ogr_layer_name=productBoundingBox ogr_layer_type=POLYGON product_type=UNW GEO IFG references=https://aria.jpl.nasa.gov/ science_grids_data_amplitude_description=Amplitude science_grids_data_amplitude_grid_mapping=crs science_grids_data_amplitude_long_name=amplitude science_grids_data_amplitude_standard_name=amplitude science_grids_data_amplitude_units=watt science_grids_data_amplitude__FillValue=0 science_grids_data_amplitudeNetcdf4Dimid=1 science_grids_data_coherence_description=Coherence science_grids_data_coherence_grid_mapping=crs science_grids_data_coherence_long_name=coherence science_grids_data_coherence_standard_name=coherence science_grids_data_coherence_units=unitless science_grids_data_coherenceFillValue=0 science_grids_data_coherenceNetcdf4Dimid=1 science_grids_data_connectedComponents_description=Snaphu connected components science_grids_data_connectedComponents_grid_mapping=crs science_grids_data_connectedComponents_long_name=connectedComponents science_grids_data_connectedComponents_standard_name=connectedComponents science_grids_data_connectedComponents_units=unitless science_grids_data_connectedComponentsFillValue=-1 science_grids_data_connectedComponentsNetcdf4Dimid=1 science_grids_data_crs_crs_wkt=GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]] science_grids_data_crs_grid_mapping_name=latitude_longitude science_grids_data_crs_inverse_flattening=298.257223563 science_grids_data_crs_longitude_of_prime_meridian=0 science_grids_data_crs_long_name=crs science_grids_data_crs_semi_major_axis=6378137 science_grids_data_crs_standard_name=crs science_grids_data_latitude_CLASS=DIMENSION_SCALE science_grids_data_latitude_long_name=latitude science_grids_data_latitude_NAME=latitude science_grids_data_latitude_REFERENCE_LIST= science_grids_data_latitude_standard_name=latitude science_grids_data_latitude_units=degrees_north science_grids_data_latitudeCoordinateAxisType=Lat science_grids_data_latitudeNetcdf4Dimid=1 science_grids_data_longitude_CLASS=DIMENSION_SCALE science_grids_data_longitude_long_name=longitude science_grids_data_longitude_NAME=longitude science_grids_data_longitude_REFERENCE_LIST= science_grids_data_longitude_standard_name=longitude science_grids_data_longitude_units=degrees_east science_grids_data_longitudeCoordinateAxisType=Lon science_grids_data_longitudeNetcdf4Dimid=2 science_grids_data_unwrappedPhase_description=Unwrapped phase science_grids_data_unwrappedPhase_grid_mapping=crs science_grids_data_unwrappedPhase_long_name=unwrappedPhase science_grids_data_unwrappedPhase_standard_name=unwrappedPhase science_grids_data_unwrappedPhase_units=rad science_grids_data_unwrappedPhaseFillValue=0 science_grids_data_unwrappedPhaseNetcdf4Dimid=1 science_grids_imagingGeometry_azimuthAngle_description=Azimuth angle science_grids_imagingGeometry_azimuthAngle_grid_mapping=crsMeta science_grids_imagingGeometry_azimuthAngle_long_name=azimuthAngle science_grids_imagingGeometry_azimuthAngle_standard_name=azimuthAngle science_grids_imagingGeometry_azimuthAngle_units=degrees science_grids_imagingGeometry_azimuthAngleFillValue=0 science_grids_imagingGeometry_azimuthAngleNetcdf4Dimid=5 science_grids_imagingGeometry_crsMeta_crs_wkt=GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]] science_grids_imagingGeometry_crsMeta_grid_mapping_name=latitude_longitude science_grids_imagingGeometry_crsMeta_inverse_flattening=298.257223563 science_grids_imagingGeometry_crsMeta_longitude_of_prime_meridian=0 science_grids_imagingGeometry_crsMeta_long_name=crsMeta science_grids_imagingGeometry_crsMeta_semi_major_axis=6378137 science_grids_imagingGeometry_crsMeta_standard_name=crsMeta science_grids_imagingGeometry_heightsMeta_CLASS=DIMENSION_SCALE science_grids_imagingGeometry_heightsMeta_long_name=height science_grids_imagingGeometry_heightsMeta_NAME=heightsMeta science_grids_imagingGeometry_heightsMeta_positive=up science_grids_imagingGeometry_heightsMeta_REFERENCE_LIST= science_grids_imagingGeometry_heightsMeta_standard_name=height science_grids_imagingGeometry_heightsMeta_units=meter science_grids_imagingGeometry_heightsMetaCoordinateAxisType=Lev science_grids_imagingGeometry_heightsMetaNetcdf4Dimid=5 science_grids_imagingGeometry_incidenceAngle_description=Incidence angle science_grids_imagingGeometry_incidenceAngle_grid_mapping=crsMeta science_grids_imagingGeometry_incidenceAngle_long_name=incidenceAngle science_grids_imagingGeometry_incidenceAngle_standard_name=incidenceAngle science_grids_imagingGeometry_incidenceAngle_units=degrees science_grids_imagingGeometry_incidenceAngleFillValue=0 science_grids_imagingGeometry_incidenceAngleNetcdf4Dimid=5 science_grids_imagingGeometry_latitudeMeta_CLASS=DIMENSION_SCALE science_grids_imagingGeometry_latitudeMeta_long_name=latitude science_grids_imagingGeometry_latitudeMeta_NAME=latitudeMeta science_grids_imagingGeometry_latitudeMeta_REFERENCE_LIST= science_grids_imagingGeometry_latitudeMeta_standard_name=latitude science_grids_imagingGeometry_latitudeMeta_units=degrees_north science_grids_imagingGeometry_latitudeMetaCoordinateAxisType=Lat science_grids_imagingGeometry_latitudeMetaNetcdf4Dimid=3 science_grids_imagingGeometry_longitudeMeta_CLASS=DIMENSION_SCALE science_grids_imagingGeometry_longitudeMeta_long_name=longitude science_grids_imagingGeometry_longitudeMeta_NAME=longitudeMeta science_grids_imagingGeometry_longitudeMeta_REFERENCE_LIST= science_grids_imagingGeometry_longitudeMeta_standard_name=longitude science_grids_imagingGeometry_longitudeMeta_units=degrees_east science_grids_imagingGeometry_longitudeMetaCoordinateAxisType=Lon science_grids_imagingGeometry_longitudeMetaNetcdf4Dimid=4 science_grids_imagingGeometry_lookAngle_description=Look angle science_grids_imagingGeometry_lookAngle_grid_mapping=crsMeta science_grids_imagingGeometry_lookAngle_long_name=lookAngle science_grids_imagingGeometry_lookAngle_standard_name=lookAngle science_grids_imagingGeometry_lookAngle_units=degrees science_grids_imagingGeometry_lookAngleFillValue=0 science_grids_imagingGeometry_lookAngleNetcdf4Dimid=5 science_grids_imagingGeometry_parallelBaseline_description=Parallel baseline science_grids_imagingGeometry_parallelBaseline_grid_mapping=crsMeta science_grids_imagingGeometry_parallelBaseline_long_name=parallelBaseline science_grids_imagingGeometry_parallelBaseline_standard_name=parallelBaseline science_grids_imagingGeometry_parallelBaseline_units=meter science_grids_imagingGeometry_parallelBaselineFillValue=0 science_grids_imagingGeometry_parallelBaselineNetcdf4Dimid=5 science_grids_imagingGeometry_perpendicularBaseline_description=Perpendicular baseline science_grids_imagingGeometry_perpendicularBaseline_grid_mapping=crsMeta science_grids_imagingGeometry_perpendicularBaseline_long_name=perpendicularBaseline science_grids_imagingGeometry_perpendicularBaseline_standard_name=perpendicularBaseline science_grids_imagingGeometry_perpendicularBaseline_units=meter science_grids_imagingGeometry_perpendicularBaselineFillValue=0 science_grids_imagingGeometry_perpendicularBaseline__Netcdf4Dimid=5 science_radarMetaData_azimuthTimeInterval_description=Azimuth time interval science_radarMetaData_azimuthTimeInterval_long_name=azimuthTimeInterval science_radarMetaData_azimuthTimeInterval_standard_name=azimuthTimeInterval science_radarMetaData_azimuthTimeInterval_units=seconds science_radarMetaData_centerFrequency_description=Radar center frequency science_radarMetaData_centerFrequency_long_name=centerFrequency science_radarMetaData_centerFrequency_standard_name=centerFrequency science_radarMetaData_centerFrequency_units=Hertz science_radarMetaData_ESDthreshold_description=ESD threshold, -1 = no ESD applied science_radarMetaData_ESDthreshold_long_name=ESDthreshold science_radarMetaData_ESDthreshold_standard_name=ESDthreshold science_radarMetaData_ESDthreshold_units=unitless science_radarMetaData_slantRangeEnd_description=Slant range end science_radarMetaData_slantRangeEnd_long_name=slantRangeEnd science_radarMetaData_slantRangeEnd_standard_name=slantRangeEnd science_radarMetaData_slantRangeEnd_units=meter science_radarMetaData_slantRangeSpacing_description=Slant range pixel spacing science_radarMetaData_slantRangeSpacing_long_name=slantRangeSpacing science_radarMetaData_slantRangeSpacing_standard_name=slantRangeSpacing science_radarMetaData_slantRangeSpacing_units=meter science_radarMetaData_slantRangeStart_description=Slant range start science_radarMetaData_slantRangeStart_long_name=slantRangeStart science_radarMetaData_slantRangeStart_standard_name=slantRangeStart science_radarMetaData_slantRangeStart_units=meter science_radarMetaData_wavelength_description=Radar wavelength science_radarMetaData_wavelength_long_name=wavelength science_radarMetaData_wavelength_standard_name=wavelength science_radarMetaData_wavelength_units=meter source=Contains modified Copernicus Sentinel data processed by ESA and ARIA NASA/JPL title=ARIA standard product UNW GEO IFG version=1b wkt_count_CLASS=DIMENSION_SCALE wkt_count_NAME=This is a netCDF dimension but not a netCDF variable. 1 wkt_count_REFERENCE_LIST= wkt_countNetcdf4Dimid=7 wkt_length_CLASS=DIMENSION_SCALE wkt_length_NAME=This is a netCDF dimension but not a netCDF variable. 464 wkt_length_REFERENCE_LIST= wkt_length__Netcdf4Dimid=6 Subdatasets: SUBDATASET_1_NAME=HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/data/amplitude SUBDATASET_1_DESC=[2370x3926] //science/grids/data/amplitude (32-bit floating-point) SUBDATASET_2_NAME=HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/data/coherence SUBDATASET_2_DESC=[2370x3926] //science/grids/data/coherence (32-bit floating-point) SUBDATASET_3_NAME=HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/data/connectedComponents SUBDATASET_3_DESC=[2370x3926] //science/grids/data/connectedComponents (32-bit floating-point) SUBDATASET_4_NAME=HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/data/unwrappedPhase SUBDATASET_4_DESC=[2370x3926] //science/grids/data/unwrappedPhase (32-bit floating-point) SUBDATASET_5_NAME=HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/imagingGeometry/azimuthAngle SUBDATASET_5_DESC=[4x26x38] //science/grids/imagingGeometry/azimuthAngle (32-bit floating-point) SUBDATASET_6_NAME=HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/imagingGeometry/incidenceAngle SUBDATASET_6_DESC=[4x26x38] //science/grids/imagingGeometry/incidenceAngle (32-bit floating-point) SUBDATASET_7_NAME=HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/imagingGeometry/lookAngle SUBDATASET_7_DESC=[4x26x38] //science/grids/imagingGeometry/lookAngle (32-bit floating-point) SUBDATASET_8_NAME=HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/imagingGeometry/parallelBaseline SUBDATASET_8_DESC=[4x26x38] //science/grids/imagingGeometry/parallelBaseline (32-bit floating-point) SUBDATASET_9_NAME=HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/imagingGeometry/perpendicularBaseline SUBDATASET_9_DESC=[4x26x38] //science/grids/imagingGeometry/perpendicularBaseline (32-bit floating-point) Corner Coordinates: Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 512.0) Upper Right ( 512.0, 0.0) Lower Right ( 512.0, 512.0) Center ( 256.0, 256.0) GDALPamDataset: In destructor with dirty metadata. GDAL: GDALClose(/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc, this=0x25f7050)
idixon
commented
5 years ago @rouault
We were running against libnetcdf v4.3.3.1 which lacked the netcdf_mem.h header file. I built v4.6.2 by hand and confirmed that netcdf_mem.h is now there. I am still seeing GDAL selecting the wrong driver over vsicurl. An example is in the response above.
rouault
commented
5 years ago Did you 'make clean' after running GDAL configure again ? And do you run a recent enough Linux kernel ( >=4.3) so as to see "userfaultfd support: yes" at the end of ./configure output ?
idixon
commented
5 years ago @rouault
I did a fresh build in a new container for the new version of netcdf.
I see:
userfaultfd support: yesabout 10 lines from the end of my ./configure output.
rouault
commented
5 years ago Does gdalinfo --format netcdf | grep Virtual report Supports: Virtual IO - eg. /vsimem/ ?
It does for me and gdalinfo /vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc opens with the netCDF driver
idixon
commented
5 years ago It does not. I'll investigate. Thanks!
rouault
commented
5 years ago what does uname -a report ? The Linux kernel header might be present but if the runtime version number is not >= 4.3, then it is not available at runtime
idixon
commented
5 years ago Linux 7df9fe41b690 4.14.101-91.76.amzn2.x86_64 #1 SMP Tue Feb 19 17:51:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
rouault
commented
5 years ago hum, well,then you can then try running under gdb, and set a break point on CPLIsUserFaultMappingSupported. It should be called and return true
idixon
commented
5 years ago Ok, I think that's my next step. Thanks for the help @rouault !
idixon
commented
5 years ago @rouault
The plot thickens! I tried building on another architecture and looked though the logs more carefully this time. In the gdal configure I noticed:
checking libnetcdf compiler and linker flags with nc-config... got version="netCDF 4.6.2", prefix="/usr/local", libs="-L/usr/local/lib -lnetcdf", includedir="/usr/local/include" checking for nc_open in -lnetcdf... yes checking for netcdf-4 (and HDF5) support in libnetcdf... yes checking for HDF4 support in libnetcdf... no checking /usr/local/include/netcdf_mem.h usability... no checking /usr/local/include/netcdf_mem.h presence... yes configure: WARNING: /usr/local/include/netcdf_mem.h: present but cannot be compiled configure: WARNING: /usr/local/include/netcdf_mem.h: check for missing prerequisite headers? configure: WARNING: /usr/local/include/netcdf_mem.h: see the Autoconf documentation configure: WARNING: /usr/local/include/netcdf_mem.h: section "Present But Cannot Be Compiled" configure: WARNING: /usr/local/include/netcdf_mem.h: proceeding with the compiler's result
I'm fairly certain I built netcdf correctly. Do you think this could be an indication that build is getting fouled?
rouault
commented
5 years ago I'm fairly certain I built netcdf correctly. Do you think this could be an indication that build is getting fouled?
absolutely. It would seem that netcdf_mem.h cannot be included alone. Does "NetCDF has netcdf_mem.h: yes" appear at the end of configure output ? I'd bet not If so, try
echo '#include <stdio.h>' > conftest.c
echo '#include <netcdf_mem.h>' >> conftest.c
gcc -c conftest.c(I haven't tested with netcdf 4.6, so it might be possible that the header in this version depends on other headers)
idixon
commented
5 years ago @rouault
It gets worse.
I rolled back to netcdf version 4.6.0 after seeing a post describing a the same problem with v4.6.2 in another project. When I configured GDAL this time everything looks peachy:
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking C_WFLAGS for maximum warnings... -Wall -Wdeclaration-after-statement
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking CXX_WFLAGS for maximum warnings... -Wall
checking whether C compiler accepts -Werror -we10006... no
checking whether C compiler accepts -diag-disable 188,1684,2259,2304,3280,11074,11076... no
checking whether C compiler accepts -Wextra... yes
checking whether C compiler accepts -Winit-self... yes
checking whether C compiler accepts -Wunused-parameter... yes
checking whether C compiler accepts -Wmissing-prototypes... yes
checking whether C compiler accepts -Wmissing-declarations... yes
checking whether C compiler accepts -Wformat... yes
checking whether C compiler accepts -Wformat -Werror=format-security -Wno-format-nonliteral... yes
checking whether C compiler accepts -Wshorten-64-to-32... no
checking whether C compiler accepts -Wlogical-op... yes
checking whether C compiler accepts -Wshadow... yes
checking whether C compiler accepts -Werror=vla... yes
checking whether C compiler accepts -Werror=declaration-after-statement... yes
checking whether C compiler accepts -Wdate-time... no
checking whether C compiler accepts -Wnull-dereference... no
checking whether C compiler accepts -Wduplicated-cond... no
checking whether C++ compiler accepts -Wextra-semi... no
checking whether C compiler accepts -Wno-sign-compare... yes
checking whether C compiler accepts -Wcomma... no
checking whether C compiler accepts -Wfloat-conversion... no
checking whether C compiler accepts -Wdocumentation -Wno-documentation-deprecated-sync... no
checking if -std=gnu89 can be enabled... yes
checking whether C++ compiler accepts -Wunused-private-field... no
checking whether C++ compiler accepts -Wmissing-prototypes... no
checking whether C++ compiler accepts -Wmissing-declarations... yes
checking whether C++ compiler accepts -Wnon-virtual-dtor... yes
checking whether C++ compiler accepts -Wold-style-cast... yes
checking whether C++ compiler accepts -Weffc++... yes
checking if -Weffc++ should be enabled... no
checking whether C++ compiler accepts -Woverloaded-virtual... yes
checking whether C++ compiler accepts -fno-operator-names... yes
checking whether C++ compiler accepts -Wzero-as-null-pointer-constant... yes
checking if use C++14 compiler options... disabled by user
checking whether g++ supports C++11 features by default... no
checking whether g++ supports C++11 features with -std=c++11... yes
checking whether C++ compiler accepts -Wsuggest-override... no
checking if -Wimplicit-fallthrough can be enabled... no
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether g++ -std=c++11 accepts -g... (cached) yes
checking how to print strings... printf
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /bin/ld
checking if the linker (/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /bin/nm -B
checking the name lister (/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... dlltool
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking for gawk... gawk
checking command to parse /bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... g++ -std=c++11 -E
checking for ld used by g++ -std=c++11... /bin/ld -m elf_x86_64
checking if the linker (/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ -std=c++11 linker (/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ -std=c++11 option to produce PIC... -fPIC -DPIC
checking if g++ -std=c++11 PIC flag -fPIC -DPIC works... yes
checking if g++ -std=c++11 static flag -static works... no
checking if g++ -std=c++11 supports -c -o file.o... yes
checking if g++ -std=c++11 supports -c -o file.o... (cached) yes
checking whether the g++ -std=c++11 linker (/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking for std::isnan... yes
checking for dlopen in -ldl... yes
checking for nanosleep in -lrt... yes
checking for sin in -lm... yes
checking for ANSI C header files... (cached) yes
checking assert.h usability... yes
checking assert.h presence... yes
checking for assert.h... yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking for unistd.h... (cached) yes
checking dbmalloc.h usability... no
checking dbmalloc.h presence... no
checking for dbmalloc.h... no
checking for dlfcn.h... (cached) yes
checking for stdint.h... (cached) yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking locale.h usability... yes
checking locale.h presence... yes
checking for locale.h... yes
checking values.h usability... yes
checking values.h presence... yes
checking for values.h... yes
checking float.h usability... yes
checking float.h presence... yes
checking for float.h... yes
checking errno.h usability... yes
checking errno.h presence... yes
checking for errno.h... yes
checking direct.h usability... no
checking direct.h presence... no
checking for direct.h... no
checking linux/userfaultfd.h usability... yes
checking linux/userfaultfd.h presence... yes
checking for linux/userfaultfd.h... yes
checking whether byte ordering is bigendian... no
checking for 64bit integer type... long long
checking for 64bit file io... yes
checking for stat64... yes
checking for fopen64... yes
checking for ftruncate64... yes
checking size of int... 4
checking size of unsigned long... 8
checking size of void*... 8
checking for int8... no
checking for int16... no
checking for int32... no
checking for uintptr_t... yes
checking native cpu bit order... lsb2msb
checking for vprintf... yes
checking for _doprnt... no
checking for snprintf... yes
checking for vsnprintf... yes
checking for atoll... yes
checking for strtoull... yes
checking for strtof... yes
checking for getcwd... yes
checking whether strtof is declared... yes
checking for readlink... yes
checking for lstat... yes
checking for posix_spawnp... yes
checking for posix_memalign... yes
checking for vfork... yes
checking for mmap... yes
checking for sigaction... yes
checking for statvfs... yes
checking for statvfs64... yes
checking for getrlimit... yes
checking for RLIMIT_AS... yes
checking for gmtime_r... yes
checking for localtime_r... yes
checking for uselocale... yes
checking to enable debug build... no, CFLAGS="-g -O2"
checking whether GCC 4.1 atomic builtins are available... yes
checking whether GCC bswap builtins are available... yes
checking for __uint128_t... yes
checking whether SSE is available at compile time... yes
checking whether SSSE3 is available at compile time... yes
checking whether AVX is available at compile time... yes
checking to enable LTO (link time optimization) build... no
checking whether we should hide internal symbols... no
checking for local include/lib path... none
checking for pthread_create in -lpthread... yes
checking for PTHREAD_MUTEX_RECURSIVE... yes
checking for PTHREAD_MUTEX_ADAPTIVE_NP... yes
checking for pthread_spinlock_t... yes
checking for 5 args mremap()... yes
checking for _SC_PHYS_PAGES... yes
checking for deflateInit_ in -lz... yes
checking for inflateCopy in -lz... yes
using pre-installed libz
checking for ld used by GCC... /bin/ld -m elf_x86_64
checking if the linker (/bin/ld -m elf_x86_64) is GNU ld... yes
checking for shared library run path origin... /bin/sh: ./config.rpath: No such file or directory
done
checking for iconv... yes
checking for working iconv... yes
checking for iconv declaration...
extern size_t iconv (iconv_t cd, char * *inbuf, size_t *inbytesleft, char * *outbuf, size_t *outbytesleft);
using ICONV_CPP_CONST=""
checking for PROJ >= 6 library... checking for proj_create_from_wkt in -lproj... yes
checking proj.h usability... yes
checking proj.h presence... yes
checking for proj.h... yes
configure: proj.h found
checking for ZSTD_decompressStream in -lzstd... no
libzstd not found - ZSTD support disabled
checking for pg_config... no
checking for PostgreSQL... no
checking for G_gisinit_2 in -lgrass5... no
checking for ffopen in -lcfitsio... no
libcfitsio not found - FITS support disabled
checking for Mopen in -lcsf... no
checking csf.h usability... no
checking csf.h presence... no
checking for csf.h... no
using internal csf code.
checking for libpng... checking for png_set_IHDR in -lpng... no
checking png.h usability... no
checking png.h presence... no
checking for png.h... no
using internal png code.
checking for libcrunch... dds support disabled.
checking for gta_version in -lgta... no
libgta not found - GTA support disabled
checking for PCIDSK... using internal libpcidsk.
checking for libtiff... checking for TIFFScanlineSize64 in -ltiff... no
using internal TIFF code.
BigTIFF support enabled.
using internal GeoTIFF code.
checking for jpeg_read_scanlines in -ljpeg... no
checking jpeglib.h usability... no
checking jpeglib.h presence... no
checking for jpeglib.h... no
using internal jpeg code.
checking CharLS/interface.h usability... no
checking CharLS/interface.h presence... no
checking for CharLS/interface.h... no
checking CharLS/charls.h usability... no
checking CharLS/charls.h presence... no
checking for CharLS/charls.h... no
checking for jpeg12... enabled
checking for DGifOpenFileName in -lgif... no
checking gif_lib.h usability... no
checking gif_lib.h presence... no
checking for gif_lib.h... no
using internal gif code.
checking for pkg-config... /bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for OGDI... no
checking ecs.h usability... no
checking ecs.h presence... no
checking for ecs.h... no
checking for FMEObjects... no
SOSI support disabled.
checking for MongoCXX... MongoCXX not found.
checking for pkg-config... (cached) /bin/pkg-config
checking pkg-config is at least version 0.21... yes
checking for MONGOCXXV3... no
checking for SDreaddata in -lmfhdfalt... no
checking for SDreaddata in -lmfhdf... no
checking for SDreaddata in -lhdf4... no
checking for SDreaddata in -lmfhdf... no
checking for SDreaddata in -lmfhdf... no
checking for pkg-config... (cached) /bin/pkg-config
checking pkg-config is at least version 0.21... yes
checking for HDF5... no
checking for H5Fopen in -lhdf5... yes
checking for kea-config... no
checking for kea... no
checking for nc-config... /usr/local//bin/nc-config
checking libnetcdf compiler and linker flags with nc-config...
got version="netCDF 4.6.0", prefix="/usr/local",
libs="-L/usr/local/lib -lnetcdf", includedir="/usr/local/include"
checking for nc_open in -lnetcdf... yes
checking for netcdf-4 (and HDF5) support in libnetcdf... yes
checking for HDF4 support in libnetcdf... no
checking /usr/local/include/netcdf_mem.h usability... yes
checking /usr/local/include/netcdf_mem.h presence... yes
checking for /usr/local/include/netcdf_mem.h... yes
checking for jpc_decode in -ljasper... no
checking for jp2_decode in -ljasper... no
checking for pgx_decode in -ljasper... no
checking for pkg-config... (cached) /bin/pkg-config
checking pkg-config is at least version 0.21... yes
checking for OPENJPEG... no
checking for FileGDBAPI... FileGDBAPI not found.
checking for NCScbmOpenFileView in -lNCSEcw... no
checking for NCScbmOpenFileView in -lecwj2... no
checking for Kakadu JPEG2000 support... not requested.
configure: MrSID support disabled.
configure: MrSID/MG4 Lidar support disabled.
checking for JP2Lura support... checking for MSG... not requested
checking if Oracle support is enabled... no
enabled
checking for MySQL... no
checking for Ingres... no
checking for Xerces C++ Parser... no
checking for Expat XML Parser... no
checking for pkg-config... (cached) /bin/pkg-config
checking pkg-config is at least version 0.25... yes
checking for LIBKML... no
checking for Google libkml... no
check for unixODBC
checking for SQLConnect in -lodbc... no
checking for SQLInstallDriverEx in -lodbcinst... no
check for windows ODBC
checking for main in -lodbc32... no
checking for SQLInstallDriverEx in -lodbccp32... no
checking if Oracle support is enabled... no
checking for DODS... disabled
checking for curl-config... /bin/curl-config
found libcurl version 7.29.0
checking for curl_global_init in -lcurl... yes
checking for xml2-config... no
checking for spatialite... disabled
checking for sqlite3_open in -lsqlite3... yes
checking for SQLite3 library >= 3.0.0... yes
checking for sqlite3_column_table_name in -lsqlite3... yes
configure: rasterlite2 support disabled
checking pcre.h usability... no
checking pcre.h presence... no
checking for pcre.h... no
checking Checking for Teigha... disabled
configure: IBM Informix DataBlade not supported.
configure: checking whether we should include ESRI SDE support...
ESRI SDE support not requested.
checking for WebPDecodeRGB in -lwebp... no
libwebp not found - WEBP support disabled
checking for geos-config... no
checking for sfcgal-config... no
checking qhull/libqhull.h usability... no
checking qhull/libqhull.h presence... no
checking for qhull/libqhull.h... no
checking libqhull/libqhull.h usability... no
checking libqhull/libqhull.h presence... no
checking for libqhull/libqhull.h... no
checking for OpenCL support... no
checking for FreeXL support... checking freexl.h usability... no
checking freexl.h presence... no
checking for freexl.h... no
checking for freexl_open in -lfreexl... no
checking for freexl_open in -lfreexl... no
checking for freexl_open in -lfreexl... no
checking for freexl_open in -lfreexl... no
checking for freexl_open in -lfreexl... no
libfreexl not found - FreeXL support disabled
checking for json_object_set_serializer in -ljson-c... no
using internal libjson-c code
checking whether to enable PAM... yes
checking whether to build PDF driver as a plugin... no
checking for poppler... disabled
checking for podofo... disabled
checking for pdfium... disabled
checking GDAL version information from gdal_version.h.in: 2.5.0
checking for perl bindings... disabled
checking for python bindings... checking for python... python
checking for python setuptools... found
checking whether we should include Java support... no
HDFS support not requested.
checking whether we should include MDB support... no
configure: checking whether we should include rasdaman support...
rasdaman support not requested.
checking whether we should include Armadillo support... no
checking for cryptopp... no
checking for crypto... checking for EVP_sha256 in -lcrypto... no
checking for LERC... enabled
checking for NULL... disabled
checking if sprintf can be overloaded for GDAL compilation... yes
configure: creating ./config.status
config.status: creating GDALmake.opt
config.status: WARNING: 'GDALmake.opt.in' seems to ignore the --datarootdir setting
config.status: creating port/cpl_config.h
config.status: port/cpl_config.h is unchanged
config.status: executing libtool commands
GDAL is now configured for x86_64-pc-linux-gnu
Installation directory: /usr/local
C compiler: gcc -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++ compiler: g++ -std=c++11 -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++14 support: no
LIBTOOL support: yes
LIBZ support: external
LIBLZMA support: no
ZSTD support: no
cryptopp support: no
crypto/openssl support: no
GRASS support: no
CFITSIO support: no
PCRaster support: internal
LIBPNG support: internal
DDS support: no
GTA support: no
LIBTIFF support: internal (BigTIFF=yes)
LIBGEOTIFF support: internal
LIBJPEG support: internal
12 bit JPEG: yes
12 bit JPEG-in-TIFF: yes
LIBGIF support: internal
JPEG-Lossless/CharLS: no
OGDI support: no
HDF4 support: no
HDF5 support: yes
Kea support: no
NetCDF support: yes
NetCDF has netcdf_mem.h: yes
Kakadu support: no
JasPer support: no
OpenJPEG support: no
ECW support: no
MrSID support: no
MrSID/MG4 Lidar support: no
JP2Lura support: no
MSG support: no
EPSILON support: no
WebP support: no
cURL support (wms/wcs/...):yes
PostgreSQL support: no
LERC support: yes
MySQL support: no
Ingres support: no
Xerces-C support: no
NAS support: no
Expat support: no
libxml2 support: no
Google libkml support: no
ODBC support: no
PGeo support: no
FGDB support: no
MDB support: no
PCIDSK support: internal
OCI support: no
GEORASTER support: no
SDE support: no
Rasdaman support: no
DODS support: no
SQLite support: yes
PCRE support: no
SpatiaLite support: no
RasterLite2 support: no
Teigha (DWG and DGNv8): no
INFORMIX DataBlade support:no
GEOS support: no
SFCGAL support: no
QHull support: internal
Poppler support: no
Podofo support: no
PDFium support: no
OpenCL support: no
Armadillo support: no
FreeXL support: no
SOSI support: no
MongoDB support: no
MongoCXX v3 support: no
HDFS support: no
userfaultfd support: yes
misc. gdal formats: aaigrid adrg aigrid airsar arg blx bmp bsb cals ceos ceos2 coasp cosar ctg dimap dted e00grid elas envisat ers fit gff gsg gxf hf2 idrisi ignfheightasciigrid ilwis ingr iris iso8211 jaxapalsar jdem kmlsuperoverlay l1b leveller map mrf msgn ngsgeoid nitf northwood pds prf r raw rmf rs2 safe saga sdts sentinel2 sgi sigdem srtmhgt terragen til tsx usgsdem xpm xyz zmap rik ozi grib eeda plmosaic rda wcs wms wmts pdf
disabled gdal formats:
misc. ogr formats: aeronavfaa arcgen avc bna cad csv dgn dxf edigeo geoconcept georss gml gmt gpsbabel gpx gtm htf jml mvt ntf openair openfilegdb pgdump rec s57 segukooa segy selafin shape sua svg sxf tiger vdv wasp xplane idrisi pds sdts amigocloud carto cloudant couchdb csw elastic gft ngw plscenes wfs
disabled ogr formats:
SWIG Bindings: python
PROJ >= 6: yes
enable GNM building: yes
enable pthread support: yes
enable POSIX iconv support:yes
hide internal symbols: noFull of excitement I ran gdal info only to see it was still selecting the HDF5 driver.
gdalinfo /vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc
Driver: HDF5/Hierarchical Data Format Release 5
Files: /vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc
Size is 512, 512What version of NETCDF are you running?
rouault
commented
5 years ago What version of NETCDF are you running?
hum, I've a weird setup currently... It appears I'm building against netcdf 4.4 from Ubuntu 16.04 but I manually added netcdf_mem.h in /usr/include, but at runtime I'm linking against a manually compiled netcdf 4.5.0, and somehow that works...
piyushrpt
commented
5 years ago @idixon ... do the debug messages show that it first attempts reading it with NETCDF before falling back on HDF5, now that you are building GDAL with netcdf_mem.h?
idixon
commented
5 years ago @piyushrpt
gdalinfo /vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc
HTTP: libcurl/7.29.0 NSS/3.36 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
VSICURL: GetFileSize(https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)=63099985 response_code=200
VSICURL: Downloading 0-16383 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
VSICURL: GetFileList(/vsicurl/https://s3.amazonaws.com/idixon)
VSICURL: GetFileSize(https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.aux)=0 response_code=404
VSICURL: GetFileSize(https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.AUX)=0 response_code=404
VSICURL: Downloading 54558720-54575103 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
VSICURL: Downloading 63078400-63094783 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
VSICURL: Downloading 54624256-54640639 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
VSICURL: Downloading 63094784-63099984 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
VSICURL: Downloading 54575104-54591487 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
VSICURL: Downloading 54591488-54624255 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
VSICURL: Downloading 63045632-63062015 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
VSICURL: Downloading 32260096-32276479 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
VSICURL: Downloading 52363264-52379647 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
VSICURL: Downloading 63062016-63078399 (https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc)...
VSICURL: Got response_code=206
HDF5: Skipping unsupported //productBoundingBox of type Unknown
HDF5: Skipping unsupported //science/grids/corrections/derived/ionosphere/ionosphere of type Unknown
HDF5: Skipping unsupported //science/grids/corrections/external/tides/solidEarthTide of type Unknown
HDF5: Skipping unsupported //science/grids/corrections/external/troposphere/troposphereHydrostatic of type Unknown
HDF5: Skipping unsupported //science/grids/corrections/external/troposphere/troposphereWet of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/DEM of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/ISCEversion of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/azimuthZeroDopplerEndTime of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/azimuthZeroDopplerStartTime of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/inputSLC/reference/L1InputGranules of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/inputSLC/reference/orbitType of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/inputSLC/secondary/L1InputGranules of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/inputSLC/secondary/orbitType of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/missionID of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/productType of type Unknown
HDF5: Skipping unsupported //science/radarMetaData/unwrapMethod of type Unknown
GDAL: GDALOpen(/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc, this=0x20f3370) succeeds as HDF5.
Driver: HDF5/Hierarchical Data Format Release 5It does not appear so. I'm going to try rebuilding with Even's versions today and see if that helps.
idixon
commented
5 years ago @piyushrpt
Are you seeing results like mine on your end?
piyushrpt
commented
5 years ago @idixon
The last time I tried (when the PR was merged), I was able to use my old proj library and it worked. I need to setup my proj6 before giving this a shot again.
rouault
commented
5 years ago I've just committed in a529c1a a fix needed to detect netcdf_mem.h in recent netcdf versions (tested with 4.6.2). gdalinfo /vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc is opened with the netcdf driver with this version
rouault
commented
5 years ago Change backported to release/2.4 branch
idixon
commented
5 years ago Everything looks great against NETCDF v4.6.2 now!
rouault
commented
5 years ago ok, closing that issue then
idixon
commented
5 years ago Thank you for the help @rouault and @piyushrpt!
Expected behavior and actual behavior.
Running GDAL against NETCDF-4 groups has different results depending on if the data is local or accessed using vsicurl.
Run locally gdal_translate extracts a geotiff with spatial information however a gdalinfo on the container doesn't appear to return the correct information.
Run over vsicurl it's the opposite problem. A gdalinfo on the container provides verbose information but a gdal_translate appears to be incomplete.
Ideally GDAL should behave the same locally and over vsicurl.
Steps to reproduce the problem.
A sample product has been staged here:
https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc
Running GDAL against the sample file (support was just recently added for NETCDF-4 groups):
bash-4.2# gdalinfo /vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc
bash-4.2# gdal_translate HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/data/unwrappedPhase unwrappedPhase.tif
bash-4.2# gdalinfo unwrappedPhase.tif
bash-4.2# curl -o S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc
bash-4.2# gdalinfo S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.n
bash-4.2# gdal_translate NETCDF:"S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc":/science/grids/data/unwrappedPhase phase.tif
bash-4.2# gdalinfo phase.tif
Operating system
Amazon Linux 2 AMI 2.0.20190115 x86_64 HVM gp2
GDAL version and provenance
GDAL 2.5.0dev-a2d4e6740d, released 2019/02/26