Which drivers are missing in the binary windows sf/rgdal distros?

[edzer]

See https://github.com/r-spatial/sf/issues/408 -- the new sf 0.5-2 has a lot of new drivers on Windows now thanks to the great work of @jeroen (including WFS and WFS over https; https://github.com/rwinlib/gdal2/issues/5, and PostGIS it seems) -- are there other ones people strongly miss?

[jeroen]

If we can reach consensus on this I am willing to put together a homebrew formula for gdal2. But the homebrew folks want an informed decision on which features to enable in a given library.

[pat-s]

@jeroen are you aware of https://github.com/OSGeo/homebrew-osgeo4mac/blob/master/Formula/gdal2.rb?

[tim-salabim]

lwgeom is something I dearly miss on windows at work. We often get invalid, e.g. non-closed polygons in open data...

[edzer]

And the st_buffer(obj, 0) trick does not work?

[mdsumner]

NetCDF and DODS and HDF4 and HDF5 drivers are the primary raster ones from my perspective. (NetCDF driver can be OpenDAP-aware). The missing OpenDAP support is a lower level shared library issue that is a problem with the Windows ncdf4 and RNetCDF packages as well.

(DODS is the old system, sequentially replaced by OpenDAP and now Thredds - these are synonymous terms as far as I know, but "DODS" is the name of the GDAL driver, for raster and vector sources). GDAL has an independent driver DODS, but NetCDF itself can also be DODS/OpenDAP capable. Similar overlap occurs with NetCDF(4) and HDF5, and you can see conflicts with raw HTTP sources and these DODS/OpenDAP/Thredds sources because the "same syntax" triggers driver-choice on connect. All driver conflicts within a given GDAL build can be resolved by prepending the driver identifier to the data source string, as far as I know.

For vector support the main missing ones in the past were those relying on XML/xerces, but I haven't checked recently. I've mostly given up on Windows with the amazing utility of RStudio Server - but I am concerned that any back-end effforts here occur in too limited contexts, without sufficient oversight of the entire package and library landscape. It's kind of an "everyone and no one" responsibility.

Ultimately the ability for individuals to obtain and compile these choices as needed on different systems and share those tool chains back via scripts, docker, package and install artefacts would be super valuable. I really thought that r-hub would be the place for sharing/developing this broader guidance, but I'm not sure.

[mdsumner]

libcurl is missing, on Linux this works but not in Windows:

d <- sf::read_sf("http://www.natice.noaa.gov/pub/special/kml_archive/antarctic/2016/antarctic_2016001.kmz")
library(dplyr)
sf::st_drivers() %>% dplyr::filter(grepl("KML|CURL", name))
    name                        long_name write  copy is_raster is_vector
1 LIBKML Keyhole Markup Language (LIBKML)  TRUE FALSE     FALSE      TRUE
2    KML    Keyhole Markup Language (KML)  TRUE FALSE     FALSE      TRUE

There's families of web-source, compressed and XML forms that haven't been available in Windows, but I'm not across the topic. I'm happy to explore it - but I'll add other examples as I find them.

[mdsumner]

See active discussion and work at https://github.com/rwinlib/gdal2/issues/7

[tim-salabim]

Closing, as I think most of this is under control/resolved now. Feel free to re-open if you disagree

[lauren-obrien]

Sorry to revive this late, but is it possible to add KEA support?

[rsbivand]

@lauren-obrien Which platform? If Windows, we can ask Rtools42 developers to consider https://github.com/ubarsc/kealib/releases/download/kealib-1.4.14/kealib-1.4.14.tar.gz and activating the driver. If macOS, we could ask the recipes developer to take similar steps. On Linux, all may build GDAL from source with customised driver choices, but packaged binaries (Ubuntu, etc.) include only a subset of drivers.

[edzer]

According to https://twitter.com/opencpu/status/962341890585710592 KEA was in the 4.1 windows binaries, and might still be in the r-universe binaries.

[rsbivand]

It required an additional external library install and a cmake build on Fedora. Present if libkea is present locally. Have added this library to the ongoing attempt to convert MXE config files to cmake.

[rsbivand]

@lauren-obrien Is KEA used outside New Zealand? It seems to have an unfortunate circular dependence on libgdal unless one ensures that libkea does not look for and find GDAL.

[lauren-obrien]

Hi @rsbivand - Windows is my concern here, so RTools would be an option, at least for less restrictive IT environments.

Not sure of the extent of KEA's usage at present, although there's some activity on gis stackexchange that suggests wider use than just NZ. Given its feature set I think its worth promoting more, but I also think that's beside the point - as @edzer points out, the current state seems to represent a regression. I can't speak to the dependency issues, perhaps @gillins can help?

[gillins]

I also know a few organisations using KEA in Australia and the UK, but I don't have any actual numbers. I just keep maintaining it because it has been useful for me and hope it might be useful for others.... Unfortunately I'm not very good at promoting so will leave that for someone else.

I don't know much about R unfortunately so might not be much help (I usually ask @pierreroudier !). The conda-forge builds of gdal include KEA support so I'd normally recommend that to Python users (and R users on Linux/OSX). Unfortunately on Windows there is a conflict between builds of gdal for Python and what is needed for R (the old gcc vs MSVC problem) and this has been resolved by packaging a compatible 'minimum' gdal build without KEA for Windows (https://github.com/conda-forge/r-rgdal-feedstock/pull/25).

@rsbivand not exactly sure what you mean about circular dependencies but there are 2 ways to add KEA support. First is to just build libkea then add KEA support when configuring gdal (this is what we do with conda-forge). Second way (for situations where you already have a gdal build that is hard to reconfigure) is to build the GDAL plugin that is contained within the libkea sources (this is what the Arch Linux AUR does). Note that the option for doing this (LIBKEA_WITH_GDAL) defaults to OFF (at least it does in recent versions) so you will need to override that by setting to ON. I'm wondering if the second way would make more sense for you but again not sure of your actual situation but happy to answer any questions....

[rsbivand]

@gillins : thanks for filling in details. For those building GDAL from source, the switch to cmake is a bit challenging/new. I used LIBKEA_WITH_GDAL:BOOL=OFF to control for cmake finding libgdal. On Windows and macOS CRAN R only links static, so generating libkea.a needs extra steps, and care with the headers inherited from H5, but we're working on it. On my development systems (recent Fedora), all looks fine.

Note that rgdal is approachiing EOL and will be retired during 2023; the central packages using GDAL are terra, sf and vapour, maybe others will emerge too (gdalcubes, others).

R advises strongly against mixing build trains for binary Windows and macOS releases, so everything should be compiled and linked with the same compiler/linker as was used to build R itself. R advises against any conda or similar environments, which are not supported at all. R and CRAN binaries for Windows now (from 4.2) use UCRT (Universal C RunTime), not versioned CRT, to help remove special treatment of multibyte characters; underlying libraries have been rebuilt with UCRT for this reason.

[rsbivand]

@lauren-obrien I've raised the absence of KEA with Rtools42 developers; I'd also mention that has someone tried the Windows binaries: r-devel: flavour of sf, terra or rgdal before R 4.2 was released, this would have been found and fixed in time. Please do contribute by testing R's development version (with relevant packages) on real workflows before release to help avoid things coming unstuck.

[gillins]

That's a pity that conda isn't recommended - iy has a bunch of consistently built packages that may have been useful. Feel free to ping me with any build problems @rsbivand.

[mdsumner]

why don't the rtools developers maintain their own forum for feedback and public discussion? I've had some contact, I will suggest. The best improvement in decades came from Jeroen Ooms wonderful work with rwinlib. Sadly it seems CRAN won't build on that goodwill . I encourage everyone to reach out directly to CRAN to try to get this stuff out in the open. It's ridiculous how elite and closed this stuff still is