Closed aazaff closed 2 years ago
aazaff
commented
3 years ago For record keeping purposes, I should also note that this is an edited repost of a question I put on stack: https://stackoverflow.com/questions/68025597/install-sf-for-r-using-gdal-from-postgresapp
rsbivand
commented
3 years ago As of https://www.kyngchaos.com/software/archive/gdal-complete/, Kynchaos has is pretty up-to-date:
GDAL 3.2.2, UIIO 1.7.1 (GIF, JPEG, PNG, TIFF, JPEG2000, WebP), GEOS 3.9.1, PROJ 7.2.1, SQLite 3.35.2, Spatialite 5.0.1, Rasterlite2 1.1b1; requires Python 3.9 (not included).
Success installing R packages on this basis has been reported. What software a user uses to install R packages on MacOS from source is in principle their own business, only CRAN MacOS binaries are supported as such.
aazaff
commented
3 years ago That's great to know. I'll probably end up using it then so I can move on with my life, but I still think it would be a good thing if we could figure this out.
rsbivand
commented
3 years ago Unless you must install from source because you need other drivers than provided in the CRAN MacOS binary, please do use the CRAN binary. https://github.com/rsbivand/checkGDALDrivers checks drivers against the CRAN Windows binary, and only four ODBC-based drivers are missing on MacOS (like MSSQLSpatial and ODBC). The shared.rds file in that test rig is:
> readRDS("shared.rds")
$vecs
[1] "AeronavFAA" "AmigoCloud" "ARCGEN" "AVCBin"
[5] "AVCE00" "BNA" "CAD" "Carto"
[9] "Cloudant" "CouchDB" "CSV" "CSW"
[13] "DGN" "DXF" "EDIGEO" "EEDA"
[17] "ESRI Shapefile" "ESRIJSON" "Geoconcept" "GeoJSON"
[21] "GeoJSONSeq" "Geomedia" "GeoRSS" "GML"
[25] "GPKG" "GPSBabel" "GPSTrackMaker" "GPX"
[29] "HTF" "HTTP" "Idrisi" "JML"
[33] "JP2OpenJPEG" "KML" "MapInfo File" "MBTiles"
[37] "Memory" "MSSQLSpatial" "MVT" "netCDF"
[41] "NGW" "ODBC" "ODS" "OGR_GMT"
[45] "OGR_PDS" "OGR_SDTS" "OGR_VRT" "OpenAir"
[49] "OpenFileGDB" "OSM" "PCIDSK" "PDF"
[53] "PDS4" "PGDUMP" "PGeo" "PLSCENES"
[57] "PostgreSQL" "REC" "S57" "SEGUKOOA"
[61] "SEGY" "Selafin" "SQLite" "SUA"
[65] "SVG" "SXF" "TIGER" "TopoJSON"
[69] "UK .NTF" "VDV" "VFK" "Walk"
[73] "WAsP" "WFS" "XLS" "XLSX"
[77] "XPlane"
$rasts
[1] "AAIGrid" "ACE2" "ADRG"
[4] "AIG" "AirSAR" "ARG"
[7] "BAG" "BIGGIF" "BLX"
[10] "BMP" "BSB" "BT"
[13] "BYN" "CAD" "CALS"
[16] "CEOS" "COASP" "COSAR"
[19] "CPG" "CTable2" "CTG"
[22] "DAAS" "DERIVED" "DIMAP"
[25] "DIPEx" "DOQ1" "DOQ2"
[28] "DTED" "E00GRID" "ECRGTOC"
[31] "EEDAI" "EHdr" "EIR"
[34] "ELAS" "ENVI" "ERS"
[37] "ESAT" "FAST" "FIT"
[40] "FujiBAS" "GenBin" "GFF"
[43] "GIF" "GMT" "GPKG"
[46] "GRASSASCIIGrid" "GRIB" "GS7BG"
[49] "GSAG" "GSBG" "GSC"
[52] "GTiff" "GTX" "GXF"
[55] "HDF4" "HDF4Image" "HDF5"
[58] "HDF5Image" "HF2" "HFA"
[61] "HTTP" "IDA" "IGNFHeightASCIIGrid"
[64] "ILWIS" "INGR" "IRIS"
[67] "ISCE" "ISIS2" "ISIS3"
[70] "JAXAPALSAR" "JDEM" "JP2OpenJPEG"
[73] "JPEG" "KMLSUPEROVERLAY" "KRO"
[76] "L1B" "LAN" "LCP"
[79] "Leveller" "LOSLAS" "MAP"
[82] "MBTiles" "MEM" "MFF"
[85] "MFF2" "MRF" "MSGN"
[88] "NDF" "netCDF" "NGSGEOID"
[91] "NGW" "NITF" "NTv1"
[94] "NTv2" "NWT_GRC" "NWT_GRD"
[97] "OZI" "PAux" "PCIDSK"
[100] "PCRaster" "PDF" "PDS"
[103] "PDS4" "PLMOSAIC" "PLSCENES"
[106] "PNG" "PNM" "PostGISRaster"
[109] "PRF" "R" "Rasterlite"
[112] "RDA" "RIK" "RMF"
[115] "ROI_PAC" "RPFTOC" "RRASTER"
[118] "RS2" "RST" "SAFE"
[121] "SAGA" "SAR_CEOS" "SDTS"
[124] "SENTINEL2" "SGI" "SIGDEM"
[127] "SNODAS" "SRP" "SRTMHGT"
[130] "Terragen" "TIL" "TSX"
[133] "USGSDEM" "VICAR" "VRT"
[136] "WCS" "WEBP" "WMS"
[139] "WMTS" "XPM" "XYZ"
[142] "ZMap" They probably do not include the PostGIS extension in PostgreSQL, so if that is you case, Kyngchaos may be useful. Please report on progress.
aazaff
commented
3 years ago Okay. I have a long geoprocessing query running at the moment that won't be finished for a few days, so I cannot try kyngchaos yet. I will update before the end of the week though, so please do not close yet.
Just to clarify, when you say please use the CRAN binaries, you mean just install.packages("sf")? That's what I've been doing thus far and the PostgreSQL driver is not listed when doing sf::st_drivers().
Here is a more thorough reproduction of my problem.
# Install libraries if necessary and load them into the environment
if (suppressWarnings(require("RPostgreSQL"))==FALSE) {
install.packages("RPostgreSQL",repos="https://cran.microsoft.com/");
library("RPostgreSQL");
}
# sf technically part of velociraptr, but it is so important that I break it out separately.
if (suppressWarnings(require("sf"))==FALSE) {
install.packages("sf",repos="https://cran.microsoft.com/");
library("sf");
}# Load data
Ecoregions = sf::st_read("https://services.arcgis.com/F7DSX1DSNSiWmOqh/arcgis/rest/services/Marine_Ecoregions_Of_the_World_(MEOW)/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson")RPostgreSQL and sf (WORKS)# Establish postgresql connection
# This assume that you already have a postgres instance with PostGIS installed and a database named ecos
# and that you have comparable configuration and credentials
# This could theoretically be done entirely within R, as the sf packae ports most geoprocessing functions
# from postgis entirely into R, but the benefit of doing it in postgis is because that allows me to preserve
# a stable database rather than recreating data constantly
Driver<-dbDriver("PostgreSQL") # Establish database driver
Ecos<-dbConnect(Driver, dbname = "ecos", host = "localhost", port = 5432, user = "user")
# Write data to PostgreSQL database
sf::st_write(dsn=Ecos,obj=Ecoregions,layer=c("plates","meow"),row.names=FALSE)
dbSendQuery(Ecos,'ALTER TABLE plates.meow RENAME "geometry" TO geom;')
dbSendQuery(Ecos,"ALTER TABLE plates.meow ADD PRIMARY KEY (id);")
# dbSendQuery(Ecos,"CREATE INDEX ON plates.meow USING GiST (geom);")# An alternative, gdal-specific method of specifying the postgres connection credentials as above
# This is NECESSARY if you want to use gdal/PostGIS layer creation options in sf::st_write(), which I almost always do.
GDAL<- "PG:host=localhost port=5432 dbname=ecos user=user"
sf::st_write(Ecoregions,GDAL,layer="plates.meow",layer_options=c("GEOMETRY_NAME=geom","LAUNDER=true","FID=id","SPATIAL_INDEX=GIST"))
Error: PostgreSQL driver not available in supported drivers, see `st_drivers()'
sf::st_drivers()$name
[1] "ESRIC" "PCIDSK" "netCDF" "PDS4" "VICAR" "JP2OpenJPEG" "PDF" "MBTiles"
[9] "BAG" "EEDA" "OGCAPI" "ESRI Shapefile" "MapInfo File" "UK .NTF" "LVBAG" "OGR_SDTS"
[17] "S57" "DGN" "OGR_VRT" "REC" "Memory" "BNA" "CSV" "GML"
[25] "GPX" "KML" "GeoJSON" "GeoJSONSeq" "ESRIJSON" "TopoJSON" "OGR_GMT" "GPKG"
[33] "SQLite" "WAsP" "OpenFileGDB" "XPlane" "DXF" "CAD" "FlatGeobuf" "Geoconcept"
[41] "GeoRSS" "GPSTrackMaker" "VFK" "PGDUMP" "OSM" "GPSBabel" "SUA" "OpenAir"
[49] "OGR_PDS" "WFS" "OAPIF" "HTF" "AeronavFAA" "EDIGEO" "SVG" "CouchDB"
[57] "Cloudant" "Idrisi" "ARCGEN" "SEGUKOOA" "SEGY" "XLS" "ODS" "XLSX"
[65] "Elasticsearch" "Carto" "AmigoCloud" "SXF" "Selafin" "JML" "PLSCENES" "CSW"
[73] "VDV" "MVT" "NGW" "MapML" "TIGER" "AVCBin" "AVCE00" "HTTP" As an aside, see #1693 for why I prefer the latter (currently not working) version to the former. Both methods were working just last week before I wiped/reinstalled my OS, so I don't believe it has anything to do with my R script, but is related to my gdal installation and configuration.
rsbivand
commented
3 years ago I'll check the MacOS drivers in the CRAN binaries next week when I have access to that platform.
rsbivand
commented
3 years ago The CRAN MacOS binary sf package includes "PostgreSQL" among available drivers (checked for arm64). I do not use PostgreSQL or PostGIS, so cannot check whether the driver works, or whether its operation is configured to include PostGIS if that might be necessary (that is, whether it matters that GDAL itself is built against PostGIS-enabled PostgreSQL headers and libraries). @etiennebr do you know whether the presence of the driver is enough?
aazaff
commented
3 years ago Thanks so much for helping me investigate this.
I have uninstalled R completely from my system and reinstalled everything from scratch just to make sure we are all on the same page. Here is my workflow and output. the "PostgreSQL" driver is still not listed after sf::st_drivers() (intel btw, since you mentioned arm64).
R.app, .Rapp.history, and /Library/Frameworks/R.framework
R version 4.1.0 (2021-05-18) -- "Camp Pontanezen"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin17.0 (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
[R.app GUI 1.76 (7976) x86_64-apple-darwin17.0]
> install.packages("sf",repos="https://cran.microsoft.com/")
also installing the dependencies ‘proxy’, ‘cpp11’, ‘e1071’, ‘wk’, ‘classInt’, ‘DBI’, ‘magrittr’, ‘Rcpp’, ‘s2’, ‘units’
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/proxy_0.4-26.tgz'
Content type 'application/octet-stream' length 232221 bytes (226 KB)
==================================================
downloaded 226 KB
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/cpp11_0.3.1.tgz'
Content type 'application/octet-stream' length 208052 bytes (203 KB)
==================================================
downloaded 203 KB
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/e1071_1.7-7.tgz'
Content type 'application/octet-stream' length 908290 bytes (887 KB)
==================================================
downloaded 887 KB
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/wk_0.4.1.tgz'
Content type 'application/octet-stream' length 532751 bytes (520 KB)
==================================================
downloaded 520 KB
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/classInt_0.4-3.tgz'
Content type 'application/octet-stream' length 453300 bytes (442 KB)
==================================================
downloaded 442 KB
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/DBI_1.1.1.tgz'
Content type 'application/octet-stream' length 670893 bytes (655 KB)
==================================================
downloaded 655 KB
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/magrittr_2.0.1.tgz'
Content type 'application/octet-stream' length 224197 bytes (218 KB)
==================================================
downloaded 218 KB
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/Rcpp_1.0.6.tgz'
Content type 'application/octet-stream' length 3205553 bytes (3.1 MB)
==================================================
downloaded 3.1 MB
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/s2_1.0.6.tgz'
Content type 'application/octet-stream' length 10879373 bytes (10.4 MB)
==================================================
downloaded 10.4 MB
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/units_0.7-2.tgz'
Content type 'application/octet-stream' length 1260841 bytes (1.2 MB)
==================================================
downloaded 1.2 MB
trying URL 'https://cran.microsoft.com/bin/macosx/contrib/4.1/sf_1.0-0.tgz'
Content type 'application/octet-stream' length 92779161 bytes (88.5 MB)
==================================================
downloaded 88.5 MB
The downloaded binary packages are in
/var/folders/1x/gd0b7qc97j78w7hh53s3vm1w0000gn/T//Rtmpjkjmei/downloaded_packages
> library(sf)
Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1
> sf::st_drivers()
name long_name write copy is_raster is_vector vsi
ESRIC ESRIC Esri Compact Cache FALSE FALSE TRUE TRUE TRUE
PCIDSK PCIDSK PCIDSK Database File TRUE FALSE TRUE TRUE TRUE
netCDF netCDF Network Common Data Format TRUE TRUE TRUE TRUE FALSE
PDS4 PDS4 NASA Planetary Data System 4 TRUE TRUE TRUE TRUE TRUE
VICAR VICAR MIPL VICAR file TRUE TRUE TRUE TRUE TRUE
JP2OpenJPEG JP2OpenJPEG JPEG-2000 driver based on OpenJPEG library FALSE TRUE TRUE TRUE TRUE
PDF PDF Geospatial PDF TRUE TRUE TRUE TRUE FALSE
MBTiles MBTiles MBTiles TRUE TRUE TRUE TRUE TRUE
BAG BAG Bathymetry Attributed Grid TRUE TRUE TRUE TRUE TRUE
EEDA EEDA Earth Engine Data API FALSE FALSE FALSE TRUE FALSE
OGCAPI OGCAPI OGCAPI FALSE FALSE TRUE TRUE TRUE
ESRI Shapefile ESRI Shapefile ESRI Shapefile TRUE FALSE FALSE TRUE TRUE
MapInfo File MapInfo File MapInfo File TRUE FALSE FALSE TRUE TRUE
UK .NTF UK .NTF UK .NTF FALSE FALSE FALSE TRUE TRUE
LVBAG LVBAG Kadaster LV BAG Extract 2.0 FALSE FALSE FALSE TRUE TRUE
OGR_SDTS OGR_SDTS SDTS FALSE FALSE FALSE TRUE TRUE
S57 S57 IHO S-57 (ENC) TRUE FALSE FALSE TRUE TRUE
DGN DGN Microstation DGN TRUE FALSE FALSE TRUE TRUE
OGR_VRT OGR_VRT VRT - Virtual Datasource FALSE FALSE FALSE TRUE TRUE
REC REC EPIInfo .REC FALSE FALSE FALSE TRUE FALSE
Memory Memory Memory TRUE FALSE FALSE TRUE FALSE
BNA BNA Atlas BNA TRUE FALSE FALSE TRUE TRUE
CSV CSV Comma Separated Value (.csv) TRUE FALSE FALSE TRUE TRUE
GML GML Geography Markup Language (GML) TRUE FALSE FALSE TRUE TRUE
GPX GPX GPX TRUE FALSE FALSE TRUE TRUE
KML KML Keyhole Markup Language (KML) TRUE FALSE FALSE TRUE TRUE
GeoJSON GeoJSON GeoJSON TRUE FALSE FALSE TRUE TRUE
GeoJSONSeq GeoJSONSeq GeoJSON Sequence TRUE FALSE FALSE TRUE TRUE
ESRIJSON ESRIJSON ESRIJSON FALSE FALSE FALSE TRUE TRUE
TopoJSON TopoJSON TopoJSON FALSE FALSE FALSE TRUE TRUE
OGR_GMT OGR_GMT GMT ASCII Vectors (.gmt) TRUE FALSE FALSE TRUE TRUE
GPKG GPKG GeoPackage TRUE TRUE TRUE TRUE TRUE
SQLite SQLite SQLite / Spatialite TRUE FALSE FALSE TRUE TRUE
WAsP WAsP WAsP .map format TRUE FALSE FALSE TRUE TRUE
OpenFileGDB OpenFileGDB ESRI FileGDB FALSE FALSE FALSE TRUE TRUE
XPlane XPlane X-Plane/Flightgear aeronautical data FALSE FALSE FALSE TRUE TRUE
DXF DXF AutoCAD DXF TRUE FALSE FALSE TRUE TRUE
CAD CAD AutoCAD Driver FALSE FALSE TRUE TRUE TRUE
FlatGeobuf FlatGeobuf FlatGeobuf TRUE FALSE FALSE TRUE TRUE
Geoconcept Geoconcept Geoconcept TRUE FALSE FALSE TRUE TRUE
GeoRSS GeoRSS GeoRSS TRUE FALSE FALSE TRUE TRUE
GPSTrackMaker GPSTrackMaker GPSTrackMaker TRUE FALSE FALSE TRUE TRUE
VFK VFK Czech Cadastral Exchange Data Format FALSE FALSE FALSE TRUE FALSE
PGDUMP PGDUMP PostgreSQL SQL dump TRUE FALSE FALSE TRUE TRUE
OSM OSM OpenStreetMap XML and PBF FALSE FALSE FALSE TRUE TRUE
GPSBabel GPSBabel GPSBabel TRUE FALSE FALSE TRUE FALSE
SUA SUA Tim Newport-Peace's Special Use Airspace Format FALSE FALSE FALSE TRUE TRUE
OpenAir OpenAir OpenAir FALSE FALSE FALSE TRUE TRUE
OGR_PDS OGR_PDS Planetary Data Systems TABLE FALSE FALSE FALSE TRUE TRUE
WFS WFS OGC WFS (Web Feature Service) FALSE FALSE FALSE TRUE TRUE
OAPIF OAPIF OGC API - Features FALSE FALSE FALSE TRUE FALSE
HTF HTF Hydrographic Transfer Vector FALSE FALSE FALSE TRUE TRUE
AeronavFAA AeronavFAA Aeronav FAA FALSE FALSE FALSE TRUE TRUE
EDIGEO EDIGEO French EDIGEO exchange format FALSE FALSE FALSE TRUE TRUE
SVG SVG Scalable Vector Graphics FALSE FALSE FALSE TRUE TRUE
CouchDB CouchDB CouchDB / GeoCouch TRUE FALSE FALSE TRUE FALSE
Cloudant Cloudant Cloudant / CouchDB TRUE FALSE FALSE TRUE FALSE
Idrisi Idrisi Idrisi Vector (.vct) FALSE FALSE FALSE TRUE TRUE
ARCGEN ARCGEN Arc/Info Generate FALSE FALSE FALSE TRUE TRUE
SEGUKOOA SEGUKOOA SEG-P1 / UKOOA P1/90 FALSE FALSE FALSE TRUE TRUE
SEGY SEGY SEG-Y FALSE FALSE FALSE TRUE TRUE
XLS XLS MS Excel format FALSE FALSE FALSE TRUE FALSE
ODS ODS Open Document/ LibreOffice / OpenOffice Spreadsheet TRUE FALSE FALSE TRUE TRUE
XLSX XLSX MS Office Open XML spreadsheet TRUE FALSE FALSE TRUE TRUE
Elasticsearch Elasticsearch Elastic Search TRUE FALSE FALSE TRUE FALSE
Carto Carto Carto TRUE FALSE FALSE TRUE FALSE
AmigoCloud AmigoCloud AmigoCloud TRUE FALSE FALSE TRUE FALSE
SXF SXF Storage and eXchange Format FALSE FALSE FALSE TRUE TRUE
Selafin Selafin Selafin TRUE FALSE FALSE TRUE TRUE
JML JML OpenJUMP JML TRUE FALSE FALSE TRUE TRUE
PLSCENES PLSCENES Planet Labs Scenes API FALSE FALSE TRUE TRUE FALSE
CSW CSW OGC CSW (Catalog Service for the Web) FALSE FALSE FALSE TRUE FALSE
VDV VDV VDV-451/VDV-452/INTREST Data Format TRUE FALSE FALSE TRUE TRUE
MVT MVT Mapbox Vector Tiles TRUE FALSE FALSE TRUE TRUE
NGW NGW NextGIS Web TRUE TRUE TRUE TRUE FALSE
MapML MapML MapML TRUE FALSE FALSE TRUE TRUE
TIGER TIGER U.S. Census TIGER/Line TRUE FALSE FALSE TRUE TRUE
AVCBin AVCBin Arc/Info Binary Coverage FALSE FALSE FALSE TRUE TRUE
AVCE00 AVCE00 Arc/Info E00 (ASCII) Coverage FALSE FALSE FALSE TRUE TRUE
HTTP HTTP HTTP Fetching Wrapper FALSE FALSE TRUE TRUE FALSE
> "HTTP"%in%sf::st_drivers()$name
[1] TRUE
> "PostgreSQL"%in%sf::st_drivers()$name
[1] FALSE@aazaff confirmed:
CRAN MacOS sf x86_64 binary:
CRAN MacOS sf aarch64 binary:
which I don't understand, as the recipes should be the same, I think: https://github.com/R-macos/recipes @s-u - libpq is in the recipes, and is picked up by aarch64 but not by x86_64 when GDAL is built; is this just a consequence of the external software binaries having been built earlier for x86_64 than aarch64? I haven't tested the "PostgreSQL" driver on aarch64, only that it is present.
aazaff
commented
3 years ago I'm glad that we've isolated the issue. Unfortunately, we have reached the limits of my knowledge. Is there someone else we could ask or a different repo/forum? I didn't get any traction from stack on my original question, but maybe we could post a new one specifically about the aarch64 v intel issue?
rsbivand
commented
3 years ago No need, we wait until Simon Urbanek responds. A new sf reached CRAN today, so new binaries will need generating. If we can get him to update the GDAL binary before that gets triggered, it will just happen.
Roger Bivand Falsensvei 32 5063 Bergen
tir. 29. jun. 2021, 18:02 skrev Andrew Zaffos @.***>:
I'm glad that we've isolated the issue. Unfortunately, we have reached the limits of my knowledge. Is there someone else we could ask or a different repo/forum? I didn't get any traction from stack on my original question, but maybe we could post a new one specifically about the aarch64 v intel issue?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r-spatial/sf/issues/1703#issuecomment-870725659, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNZ3BFPKSLFC6LHCNUG4HDTVHVDHANCNFSM47GHX4AQ .
s-u
commented
3 years ago @rsbivand I fear this is the old problem of the configure script being broken in GDAL - it fails to detect PostgreSQL because it is using the wrong flags (this is true for other libraries as well - they fail to include --static in pkg-config for static libraries). I think I may have manually hacked it for the arm64 build as there were so many issues in GDAL that it I had to manually modify it anyway. Sadly, the list of manual for GDAL overrides gets longer and longer - it would be great if they hired someone who actually knows autoconf to clean up that mess and use the testing framework properly (apart from the fact that they don't seem to test static builds one of the major issues is that they run tests by removing all flags for each test so they don't actually even test what will be used at compile time which is a capital sin in autoconf scripts and also it prevents the user from fixing their bad flag detection). Anyway, long story short, I'll see if I can manually modify GDAL again in x86_64 as well... maybe I should add all those patches to recipes, the only issue with that it that it's likely to break on each update (I'm still hopeful that they fix the bugs sometime ;)).
rsbivand
commented
3 years ago Thanks very much, @s-u ! I wonder whether you use version control on the corrections you make to the configure scripts ? The idea of hiring someone is attractive, also in the context of "us" trying to set up an OSGeo R-spatial community @Robinlovelace https://github.com/r-spatial/discuss/issues/44. Would the target be something like this? To ensure that static works robustly (for GDAL, PROJ, GEOS, other OSGeo?) under a number of scenarios to ensure that cross-platform downstream uses build cleanly when they choose to build static, as we do for MacOS and Windows binary packages.
Robinlovelace
commented
3 years ago Thanks for the link to this issue. I'm still awaiting feedback from our application, I guess I should check they actually got it!
kadyb
commented
3 years ago @Robinlovelace, if you haven't seen, here is the answer from Jody Garnett: https://lists.osgeo.org/pipermail/incubator/2021-May/004331.html
rsbivand
commented
3 years ago @Robinlovelace The licence issue maybe simply involves pointing to the key-value pair in all DESCRIPTION files. It will also help to tabulate these for packages in the R-spatial and rspatiial github organisations. I think that @jodygarnett is narrowing the actual proposal down to sf, so it would be helpful to get some feedback on OSGeo's reading of the proposal (not sf as an OSGeo member project, but R-spatial with many additions as a community member).
aazaff
commented
3 years ago This seems to have evolved into more of a thread for R-spatial leadership, but I just wanted to add a thought.
The current strategy of having @s-u manually configure stable static builds of sf for all platforms makes a lot of sense. I'm certainly looking forward to the promised x86_64 binary! However, I wonder if this strategy will be sustainable in the long-term? Particularly given this comment from the previous thread:
A heads' up wrt. an ongoing thread on the gdal-dev list. It started with Even Rouault asking which drivers could be dropped from GDAL to ease maintenance, and has widened to cover the struggle independent FOSS developers have to secure income.
Dropping some of the obscure gdal drivers makes a lot of sense and I can see why they're exploring it, but this might lead to people needing to retroactively add dropped drivers back into their libraries. As someone who has at various points had versions of gdal from brew, sf/rgdal, QGIS, and postgresapp simultaneously floating around my system, I can testify it is a nightmare to keep all the drivers in sync among them.
Just to prove that this is not a completely fantastical scenario, I will mention that a few years ago I built a Shiny app on a Linux server that had something like this for the file export:
output$downloadData<-downloadHandler(
filename=function() {
"output.zip"
},
content = function(file) {
Output<-switch(input$format,
"ESRI File Geodatabase" = getGDB(collection_id()),
"GeoJSON" = writeLayers(collection_id(),tempdir(),"geojson"),
"KML_COLOR" = writeLIBKML(collection_id(),tempdir()),
"ESRI Shapefile" = writeLayers(collection_id(),tempdir(),"shp")
)
zip(file,Output,flags = "-r -j -m")
},
contentType = "application/zip"
)You'll notice that in the switch() function for KML I use a different function from the others. In the case of geojson and shapefiles, the function writeLayers() is just a wrapper for sf::st_write(). In the case of the KML, however, I wanted to add vector styling information to the KML, which required using Google's long abandoned libkml driver as opposed to the kml driver that is provided with regular gdal binaries. Setting this up was a nightmare, and I think for a while I ended up bypassing sf entirely for the libkml case and just calling down to ogr2ogr directly through system(), which is not the greatest architecture.
I have no idea how hard it would be (or if it is even possible!), but a dream case for me would be something within the RConsole that works conceptually like setwd() for defining library paths.
rsbivand
commented
3 years ago @aazaff Thanks - thinking aloud about user needs is helpful. Only a few of us follow (digested) PROJ/GDAL/GEOS developer mailing lists, and we try to adapt key R packages to avoid problems. We have no idea how users of R packages might be able to feed back use cases like yours, which is all the more valuable. @Robinlovelace and others - how might the nascent community survey user practices? We only see needs when breakages cause issues, and when the issues (or R-sig-geo posts or SO ...) get filed.
Robinlovelace
commented
3 years ago @Robinlovelace, if you haven't seen, here is the answer from Jody Garnett: https://lists.osgeo.org/pipermail/incubator/2021-May/004331.html
Many thanks for that @kadyb, totally missed that but see it's clearly there and in my inbox. Plan to draft a reply and get feedback here: https://github.com/r-spatial/discuss/issues/44
We have no idea how users of R packages might be able to feed back use cases like yours, which is all the more valuable. @Robinlovelace and others - how might the nascent community survey user practices?
I think surveying users on their needs is a key role for more-or-less formalised groups like r-spatial. I don't have much experience doing this but am sure there is knowhow. I recall @Nowosad put out a request for feedback on a package. If you or or anyone could share info and suggestions (e.g. Microsoft Forms, Google Forms, other best?) that could be a way to get that going, although this is again a separate issue and one that should probably be opened here: https://github.com/r-spatial/discuss/issues
etiennebr
commented
3 years ago I'm catching up on the issue. Just to clarify, although it might clear by now, but just in case. sf::st_drivers() lists drivers that are available from gdal (from the build). While RPostgreSQL uses libpq, AFAIK it is orthogonal to gdal.
If you want to use a Postgres driver with sf, I would strongly recommend RPostgres instead of the two previous options. Unfortunately, I can't help much more since I have no experience with mac drivers.
aazaff
commented
3 years ago Does anyone know if @s-u fixed/updated the gdal configs as suggested above?
s-u
commented
3 years ago @aazaff not sure what you mean, but you can check the current status of available binaries at https://github.com/R-macos/recipes and installation overrides are at https://svn.r-project.org/R-dev-web/trunk/QA/Simon/packages/flags.inst/ If you need anything, either file a PR (for recipes) or send me an e-mail if you need overrides.
aazaff
commented
2 years ago I don't know how or when, but the PostgreSQL driver now shows up correctly with a fresh install of the CRAN MacOS sf x86_64 binary.
Closing issue!
Contextual Comments
This is probably more support than you want to provide, but I think that if we can figure out how to do this it would be great to update https://github.com/r-spatial/sf#macos for the broader OSx
sfcommunity. Especially since the kyngchaos binaries are increasingly out of date and the postgresapp bundle is endorsed by postgis.net for Mac users.Question
I use the https://postgresapp.com binaries for installing PostgreSQL/PostGIS on my machine, which includes the needed libraries for PostGIS (proj, geos, and gdal). According to the postgresapp installation documentation (https://postgresapp.com/documentation/install.html) the libraries and configs are saved at the default following paths (which I can confirm):
Binaries: /Applications/Postgres.app/Contents/Versions/latest/bin Headers: /Applications/Postgres.app/Contents/Versions/latest/include Libraries: /Applications/Postgres.app/Contents/Versions/latest/lib
Therefore, when combined with the "sf" installation documentation my assumption is that the following should work.
Although I receive no errors or warnings, this does not seem to be working correctly as
sf::st_drivers()still does not list the PostgreSQL driver as an option. Are there additionalconfigure.args=arguments that need to be passed?