Open mdsumner opened 2 years ago
working example with bbox, crs, and bands in 'vrt_connection' branch: https://github.com/mdsumner/gdal/tree/vrt_connection
autotest/ is in the gdal source tree
gdalinfo "vrt://autotest/gdrivers/data/netcdf/MODIS_ARRAY.nc?a_ullr=0,1,1,0&a_srs=EPSG:3857&a_ullr=100,200,1000,-300&bands=1,1"
now add '-ot' output type
gdalinfo "vrt://autotest/gdrivers/data/netcdf/MODIS_ARRAY.nc?a_ullr=0,1,1,0&a_srs=EPSG:3031&a_ullr=100,200,1000,-300&bands=1,1&ot=Float64"
'-if' (doesn't seem to work)
'-expand' (fails sometimes ?? is there a transient temp file or something?)
gdalinfo "vrt://autotest/gcore/data/test_average_palette.tif?expand=rgba&if=NetCDF"
what is up with this? it doesn't work, and then it does ...
how to do GDAL executable
#include <iostream>
#include "gdal.h"
#include "ogr_spatialref.h"
bool this_way(char *input) {
OGRSpatialReference srs;
srs.SetFromUserInput(input);
char *txt = nullptr;
srs.exportToProj4(&txt);
std::cout << txt << "\n";
CPLFree(txt);
return true;
}
bool cpl_way(CPLString input) {
OGRSpatialReference srs;
srs.SetFromUserInput(input);
char *output = nullptr;
srs.exportToProj4(&output);
std::cout << output << "\n";
CPLFree(output);
return true;
}
bool that_way(char *input) {
OGRSpatialReference *srs = nullptr;
srs = new OGRSpatialReference;
srs->SetFromUserInput(input);
char *txt = nullptr;
srs->exportToProj4(&txt);
delete srs;
std::cout << txt << "\n";
CPLFree(txt);
return true;
}
int main() {
char input[] = "WGS84";
bool yes = this_way(input);
bool no = that_way(input);
CPLString input2{};
input2 = "WGS84";
bool cpl = cpl_way(input2);
return 1;
}
g++ -I/usr/include/gdal -o program program.cpp -L/usr/lib -lgdal -lproj
./program
+proj=longlat +datum=WGS84 +no_defs +proj=longlat +datum=WGS84 +no_defs +proj=longlat +datum=WGS84 +no_defs
so, you can do spaces in the args, or escape them, committed just now
gdalinfo "vrt://autotest/gdrivers/data/netcdf/MODIS_ARRAY.nc?a_srs=EPSG:3857&bands=1,1&a_ullr=0\ 10\ 100\ -10" gdalinfo "vrt://autotest/gdrivers/data/netcdf/MODIS_ARRAY.nc?a_srs=EPSG:3857&bands=1,1&a_ullr=0 10 100 -10"
you can't use commas in the decimals, but gdalinfo normally can so we'd better use it's parsing (it can mix them even)
the spaces fail when it comes to the VRT text, you get '&' but not %20
just my geoarrow-cpp notes
mkdir test
cp test.cpp test/
cd test
cp ../CMake* .
#nano CMakeLists.txt ## edit CMakeLists to ../src/geoarrow etc
cmake .
g++ test.cpp -L . -lgeoarrow -lm
./test
g++ test.cpp -L . -lgeoarrow -lm -o test
new, here we're consolidating
I've got most libraries we need, there is some remaining stuff around BLOSC, lz4, Zstandard, that family of compression stuff or whatever it is.
## arrow
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev # For C++
sudo apt upgrade
## R
# update indices
sudo apt update -qq
# install two helper packages we need
sudo apt install --no-install-recommends software-properties-common dirmngr
# add the signing key (by Michael Rutter) for these repos
# To verify key, run gpg --show-keys /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc
# Fingerprint: E298A3A825C0D65DFD57CBB651716619E084DAB9
wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | sudo tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc
# add the R 4.0 repo from CRAN -- adjust 'focal' to 'groovy' or 'bionic' as needed
sudo add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/"
sudo apt install --assume-yes \
libmagick++-dev libarchive-dev libnetcdf-dev libnode-dev \
libxml2-dev libcurl4-openssl-dev libssl-dev libgl1-mesa-dev libglu1-mesa-dev \
libudunits2-dev libprotobuf-dev protobuf-compiler imagemagick libgit2-dev r-base r-base-dev
sudo apt install g++ libxml2-dev libcurl4-openssl-dev cmake pkg-config make libsqlite3-dev python3-dev libtiff-dev lz4 libgtest-dev libkml-dev libparquet-dev libarrow-dataset-dev libxerces-c-dev libhdf4-dev libblosc-dev sqlite3
git clone https://github.com/osgeo/proj
cd proj
git checkout 9.1.1
mkdir build
cd build
cmake ..
sudo cmake --build . --parallel 31 --target install
git clone https://github.com/libgeos/geos
cd geos
git checkout 3.11.1
mkdir build
cd build
cmake ..
sudo cmake --build . --parallel 31 --target install
git clone https://github.com/osgeo/gdal
cd gdal
mkdir build
cd build
## finally got this to work, see https://gist.github.com/mdsumner/526af876cfddaa5ff245ab376b3cec84
##cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/usr
## without csharp or java (since 2023-03-15)
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/usr -DBUILD_JAVA_BINDINGS:BOOL=OFF -DBUILD_CSHARP_BINDINGS:BOOL=OFF
sudo cmake --build . --parallel 31 --target install
sudo apt install docker.io
## this doesn't work, it doesn't pick up the system libs above
#sudo usermod -aG docker ubuntu
#docker run --rm -ti -e PASSWORD=yourpassword -p 8787:8787 rocker/rstudio user/yourpassword
So using the above, or using the docker target builder from the gdal docker scripts, I needed to do
cp -R /usr/local/lib/python3/dist-packages/* /usr/local/lib/python3.8/dist-packages/
none of PYTHONPATH or LD_LIBRARY_PATH or any other config I could find would work , so I'll just hack this for now.
the docker way, with full
#~/Git/gdal-source/gdal/docker/ubuntu-full$
docker build . --target builder -t hypertidy/gdal-full
docker run -it --rm -v /home/gdal/Git/gdal-source:/gdal-source hypertidy/gdal-full
cd gdal-source/gdal/build2 ## use different
## cmake config and build/install as above
cd ../autotest
apt update
apt install python3-pip ## wtaf
python3.8 -m pip install -r ./requirements.txt
pytest gdrivers/mem.py::test_mem_2
getting smarter about docker, using gdal-source/ubuntu-small-mike
## build the runner
docker build . -t hypertidy/gdal-small-runner
docker build . --target builder -t hypertidy/gdal-small-builder
## run em
docker run -it --rm -v /home/gdal/Git/gdal-source:/gdal-source hypertidy/gdal-small-runner
docker run -it --rm -v /home/gdal/Git/gdal-source:/gdal-source hypertidy/gdal-small-builder
building netcdf with cmake (on gdalserver)
- # NetCDF C Configuration Summary
==============================
# General
-------
NetCDF Version: 4.9.2
Dispatch Version: 5
Configured On: Fri 12 May 2023 00:21:30 UTC
Host System: x86_64-Linux-5.4.0-148-generic
Build Directory: /home/gdal/source/netcdf-c-4.9.2
Install Prefix: /usr/local
Plugin Install Prefix: N.A.
# Compiling Options
-----------------
C Compiler: /usr/bin/cc
CFLAGS: -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -g -Wall -Wconversion
CPPFLAGS:
LDFLAGS: -Wl,--no-undefined
AM_CFLAGS:
AM_CPPFLAGS:
AM_LDFLAGS:
Shared Library: yes
Static Library: no
Extra libraries: -lhdf5_hl -lhdf5 -lpthread -lsz -lz -ldl -lm -lblosc -lzstd -lbz2 -lcurl -lxml2
XML Parser: libxml2
# Features
--------
Benchmarks: no
NetCDF-2 API: yes
HDF4 Support: no
HDF5 Support: yes
NetCDF-4 API: yes
CDF5 Support: yes
NC-4 Parallel Support: no
PnetCDF Support: no
DAP2 Support: yes
DAP4 Support: yes
Byte-Range Support: yes
S3 Support: no
NCZarr Support: yes
NCZarr Zip Support: no
Diskless Support: yes
MMap Support: yes
JNA Support: no
ERANGE Fill Support: yes
Relaxed Boundary Check: yes
Multi-Filter Support: yes
Quantization: yes
Logging: no
SZIP Write Support: yes
Standard Filters: deflate szip blosc zstd bz2
ZSTD Support: yes
Parallel Filters: yes
-- Configuring done
-- Generating done
-- Build files have been written to: /home/gdal/source/netcdf-c-4.9.2
for python plots over ssh I needed
sudo apt-get install python3-tk sudo apt-get install qt5-default
ok so on gdal server stick with this for now
sudo python3.8 -m pip install -r ./doc/requirements.txt
sudo python3.8 -m pip install -r ./autotest/requirements.txt
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/usr -DPython_LOOKUP_VERSION=3.8.10
sudo cmake --build . --parallel 31 --target install
here is the rocker dev-osgeo
https://github.com/rocker-org/rocker-versioned2/pkgs/container/geospatial
then I had to do
sudo apt update
sudo apt upgrade
sudo apt install python-dev-is-python3
wget https://bootstrap.pypa.io/get-pip.py
sudo python3 get-pip.py
sudo python3 -m pip install cryptography pandas datetime fsspec branca click pyparsing dask pystac_client xarray cachetools zarr odc-stac
sudo python3 -m pip install gdal gnm ogr osr --no-binary "gdal,gnm,ogr,osr"
sudo python3 -m pip install rasterio shapely fiona geopandas pyproj affine --no-binary "rasterio,shapely,fiona,geopandas,pyproj,affine"
sudo python3 -m pip install folium
sudo python3 -m pip install odc-stac odc-geo rioxarray --no-binary ":all:"
then it's all good in 3.10.12
scratch the above, do it all from source and use this: https://github.com/auspatious/dea-coastlines/blob/stable/Dockerfile
sudo apt install python3-pip
sudo python3 -m pip install affine branca cachetools click cryptography dask datetime fiona folium fsspec gdal geopandas gnm odc-geo odc-stac odc-stac ogr osr pandas pyparsing pyproj pystac_client rasterio rioxarray shapely xarray zarr --nobinary ":all:"
how to rebase https://adamj.eu/tech/2022/03/25/how-to-squash-and-rebase-a-git-branch/
Getting this down to much tighter
to build from source do
to lint and test
with rocker:
OLD NOTES BELOW
To check out and build GDAL from source, as ubuntu user
GDAL Build hints
https://gdal.org/build_hints.html
My notes
first PROJ (Note that 8.2 is still autotools, cmake required as of version 9.0 - I'll update)
now GDAL, with new cmake
then to check utility function
previous notes were here, I will update once I get this sorted: https://github.com/mdsumner/nectar/blob/master/gdalbuilder.md