search

r-spatial / sf

Simple Features for R
https://r-spatial.github.io/sf/
Other
1.35k stars 299 forks source link

read_sf GeoPackage crashes R #1606

Closed alazarolop closed 3 years ago

alazarolop commented 3 years ago

Hi,

I've just installed sf from GitHub (0.9-8) on macOS Big Sur (11.2.1). There is no error when building the package, but when I try to read a GeoPackage, R crashes.

r$> library(sf)                                                                                                                                                               
Linking to GEOS 3.9.1, GDAL 3.2.1, PROJ 7.2.1

r$> world <- sf::read_sf("gadm36_levels.gpkg", layer = "level0")                                          

 *** caught segfault ***
address 0x100000000, cause 'memory not mapped'

Traceback:
 1: CPL_read_ogr(dsn, layer, query, as.character(options), quiet,     type, fid_column_name, drivers, wkt_filter, promote_to_multi,     int64_as_string, dsn_exists, dsn_isdb)
 2: st_read.character(..., quiet = quiet, stringsAsFactors = stringsAsFactors,     as_tibble = as_tibble)
 3: st_read(..., quiet = quiet, stringsAsFactors = stringsAsFactors,     as_tibble = as_tibble)
 4: sf::read_sf("gadm36_levels.gpkg",     layer = "level0")

I don't get any clue from the Traceback in order to fix it. I'm able to read the GeoPackage with ogrinfo tough, so I guess it's not a problem with GDAL installation.

a@MBP gadm36_levels_gpkg % ogrinfo gadm36_levels.gpkg
INFO: Open of `gadm36_levels.gpkg'
      using driver `GPKG' successful.
1: level0 (Multi Polygon)
2: level1 (Multi Polygon)
3: level2 (Multi Polygon)
4: level3 (Multi Polygon)
5: level4 (Multi Polygon)
6: level5 (Multi Polygon)
a@MBP gadm36_levels_gpkg %

Moreover, I'm able to read a SHP file with sf.

r$> read_sf("pgsql-ogr-fdw/data/pt_two.shp")                                                                                                 
Simple feature collection with 2 features and 4 fields
geometry type:  POINT
dimension:      XY
bbox:           xmin: -0.581035 ymin: 0.2443327 xmax: 0.098329 ymax: 0.7032013
geographic CRS: WGS 84
# A tibble: 2 x 5
  name    age height birthdate               geometry
  <chr> <int>  <dbl> <date>               <POINT [°]>
1 Peter    45   5.6  1965-04-12  (0.098329 0.7032013)
2 Paul     33   5.84 1971-03-25 (-0.581035 0.2443327)

Could it be a problem with my sf installation or perhaps a bug with GeoPackage in sf? I was able to read the file before upgrading to latest version of GDAL, GEOS, Proj and sf.

Thank you in advance.

edzer commented 3 years ago

is this a binary or a source install?

alazarolop commented 3 years ago

From source.

I've also rebuilt from source rgdal and rgeos with no luck.

edzer commented 3 years ago

Have you tried binary installs?

alazarolop commented 3 years ago

No, I built R from source to get OpenBLAS support and so AFAIK I can't install binaries, am I right?

Is there a way I could extract some debug information more than the Traceback?

PS: I've also installed everything from scratch in a new R installation, but I get the same outcome.

edzer commented 3 years ago

Is there a way I could extract some debug information more than the Traceback?

Yes, but I'm not a mac user so can't help you with that.

I guess you're using dynamic linking; a common source of problems is that you have more than one version of GDAL, GEOS and or PROJ installed on your system.

alazarolop commented 3 years ago

I'll check it out and I'll also recompile them from source.

I'll close the ticket in the meantime. Thank you a lot for the support Edzer.

edzer commented 3 years ago

Solved?

alazarolop commented 3 years ago

Not really. I've rebuilt from source GDAL and R packages rgeos, rgdal, sf; but I'm getting the same issue. I closed cause I said I would in the meantime and I can't also give more information on the topic.

alazarolop commented 3 years ago

I've just tried to rebuilt R from source and install the packages from scratch with no luck neither.

edzer commented 3 years ago

Did you make 100% sure there is one and only one of the GEOS, GDAL and PROJ libraries on your system?

alazarolop commented 3 years ago

I apologise for closing and reopening the issue.

I get the same issue using rgdal approach, either trying to metadata or the data:

rgdal::ogrInfo("data/project_sites.gpkg")

 *** caught segfault ***
address 0x100000000, cause 'memory not mapped'

Traceback:
 1: ogrListLayers(dsn = dsn)
 2: rgdal::ogrInfo("data/project_sites.gpkg")
readOGR("data/project_sites.gpkg")

 *** caught segfault ***
address 0x100000000, cause 'memory not mapped'

Traceback:
 1: ogrListLayers(dsn = dsn)
 2: readOGR("data/project_sites.gpkg")

So I guess it's exceed sf and can be related to GDAL. It's working in the command line tough:

ogrinfo data/project_sites.gpkg
INFO: Open of `data/project_sites.gpkg'
      using driver `GPKG' successful.
1: sites (Point)
2: sites_sum (Point)

My session info: 

R version 4.0.4 (2021-02-15)
Platform: x86_64-apple-darwin20.3.0 (64-bit)
Running under: macOS Big Sur 11.2.1

Matrix products: default
BLAS/LAPACK: /usr/local/Cellar/openblas/0.3.13/lib/libopenblasp-r0.3.13.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_4.0.4
alazarolop commented 3 years ago

Did you make 100% sure there is one and only one of the GEOS, GDAL and PROJ libraries on your system?

Yes. I only install them using Homebrew and there is only one version for each. I don't have any installation from a third repository.

edzer commented 3 years ago

Did you compile the command line tool ogrinfo from source too?

alazarolop commented 3 years ago

I first used a binary downloaded from Homebrew and then I compiled it from source, in both cases I could read the GeoPackage without issues.

edzer commented 3 years ago

Since this happens both with rgdal and sf, it seems like a problem with your GDAL setup. I suggest to close here.

alazarolop commented 3 years ago

Thank you for the support.

alazarolop commented 3 years ago

@edzer I solved it with a workaround: creating a connection as a SQLite file and then reading it. For future references:

x <- dbConnect(RSQLite::SQLite(), "data/project_sites.gpkg") %>%
  sf::st_read(layer = "sites", as_tibble = TRUE)
pat-s commented 3 years ago

Facing the same issue

macOS 11.2.1 R 4.0.4 sf 0.9-7

library(sf) 
st_read(system.file("gpkg/nc.gpkg", package="sf"), quiet = TRUE)                      
Linking to GEOS 3.9.1, GDAL 3.2.1, PROJ 7.2.1

 *** caught segfault ***
address 0x100000000, cause 'memory not mapped'

Traceback:
 1: CPL_read_ogr(dsn, layer, query, as.character(options), quiet,     type, fid_column_name, drivers, wkt_filter, promote_to_multi,     int64_as_string, dsn_exists, dsn_isdb)
 2: st_read.character(system.file("gpkg/nc.gpkg", package = "sf"),     quiet = TRUE)
 3: st_read(system.file("gpkg/nc.gpkg", package = "sf"), quiet = TRUE)
rsbivand commented 3 years ago

Please consider running R -d gdb and reporting the output.

alazarolop commented 3 years ago

Hi @rsbivand , this is my output:

a@MBP ~ % R -d gdb
/usr/local/bin/R: line 298: exec: gdb: not found
rsbivand commented 3 years ago

Try (untried) R -d lldb.

alazarolop commented 3 years ago

Try (untried) R -d lldb.

a@MBP ~ % R -d lldb
(lldb) target create "/usr/local/Cellar/r/4.0.4/lib/R/bin/exec/R"
Current executable set to '/usr/local/Cellar/r/4.0.4/lib/R/bin/exec/R' (x86_64).
(lldb)

Then, I could type text but not R commands.

edzer commented 3 years ago

(if this works like gdb, type r to run the R process, then the commands that lead to the crash)

alazarolop commented 3 years ago

Thanks for the clue @edzer . Now, it works like a charm. 

> library(sf)
Linking to GEOS 3.9.1, GDAL 3.2.1, PROJ 7.2.1
> st_read(system.file("gpkg/nc.gpkg", package="sf"), quiet = TRUE)
Simple feature collection with 100 features and 14 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: -84.32385 ymin: 33.88199 xmax: -75.45698 ymax: 36.58965
Geodetic CRS:  NAD27
> sessionInfo()
R version 4.0.4 (2021-02-15)
Platform: x86_64-apple-darwin20.3.0 (64-bit)
Running under: macOS Big Sur 11.2.2

Matrix products: default
BLAS/LAPACK: /usr/local/Cellar/openblas/0.3.13/lib/libopenblasp-r0.3.13.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/POSIX/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] sf_0.9-8

loaded via a namespace (and not attached):
 [1] compiler_4.0.4     magrittr_2.0.1     class_7.3-18       tools_4.0.4
 [5] DBI_1.1.1          units_0.7-0        Rcpp_1.0.6         KernSmooth_2.23-18
 [9] grid_4.0.4         e1071_1.7-4        classInt_0.4-3

If I run it like usual without the flag R, I get the error. How could I add this fixed into my R workflow / loading process.

edzer commented 3 years ago

Oh, that is nasty! Could you try running the script with valgrind, i.e. within R -d valgrind ?

alazarolop commented 3 years ago

Ok! Unfortunately, I get the same output than with R -d gdb:

a@MBP ~ % R -d valgrind
/usr/local/bin/R: line 298: exec: valgrind: not found

I'm not sure if I needed to add something at the end to run it or similar:

R -d valgrind -e 'sf::st_read(system.file("gpkg/nc.gpkg", package="sf"), quiet = TRUE)'

But it outputs the same error.

edzer commented 3 years ago

You'd need to install valgrind.

alazarolop commented 3 years ago

It will take a bit of time, cause it's not available in Homebrew for macOS. I'll let you know as soon as I get it.

alazarolop commented 3 years ago

Unfortunately, it's not available for macOS 11.x

checking for a supported OS... ok (darwin20.3.0)
checking for the kernel version... unsupported (20.3.0)
configure: error: Valgrind works on Darwin 10.x, 11.x, 12.x, 13.x, 14.x, 15.x, 16.x and 17.x (Mac OS X 10.6/7/8/9/10/11 and macOS 10.12/13)

Is there another alternative that we could try?

ManuelSpinola commented 3 years ago

I am having the same problem.

Reading geopackage files crushes R

I am using macOS Big Sur 11.2.3

library(sf) Linking to GEOS 3.9.0, GDAL 3.2.1, PROJ 7.2.1

sessionInfo() R version 4.0.4 (2021-02-15) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16

Matrix products: default LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] sp_1.4-5

loaded via a namespace (and not attached): [1] pillar_1.5.1 compiler_4.0.4 tools_4.0.4 downlit_0.2.1
[5] digest_0.6.27 lattice_0.20-41 viridisLite_0.3.0 jsonlite_1.7.2
[9] evaluate_0.14 lifecycle_1.0.0 tibble_3.1.0 gtable_0.3.0
[13] pkgconfig_2.0.3 rlang_0.4.10 cli_2.3.1 DBI_1.1.1
[17] distill_1.2 yaml_2.2.1 xfun_0.22 dplyr_1.0.5
[21] httr_1.4.2 knitr_1.31 generics_0.1.0 vctrs_0.3.6
[25] htmlwidgets_1.5.3 grid_4.0.4 tidyselect_1.1.0 glue_1.4.2
[29] data.table_1.14.0 R6_2.5.0 fansi_0.4.2 plotly_4.9.3
[33] rmarkdown_2.7 tidyr_1.1.3 ggplot2_3.3.3 purrr_0.3.4
[37] magrittr_2.0.1 scales_1.1.1 ellipsis_0.3.1 htmltools_0.5.1.1 [41] assertthat_0.2.1 colorspace_2.0-0 utf8_1.2.1 tinytex_0.30
[45] lazyeval_0.2.2 munsell_0.5.0 crayon_1.4.1

rsbivand commented 3 years ago

Same question on https://stat.ethz.ch/pipermail/r-sig-geo/2021-March/028632.html. I relied there:

Please refer to the associated sf issue on github: https://github.com/r-spatial/sf/issues/1606 for completeness. Please confirm that you are not using the MacOS sf CRAN binary package, but have installed GDAL, PROJ and GEOS yourself. Please also confirm that you see the same error with every GPKG file.

rsbivand commented 3 years ago

Further, with R 4.0.4 on Rosetta but with CRAN binary packages, there is no problem. It is most likely that the problem is caused by user-installed upstream software components, and sf or rgdal installed from source, possibly with libsqlite3 also installed from source. Please report back if any other drivers fail - RSQLite does not use local binaries, but bundles the software library itself.

alazarolop commented 3 years ago

GPKG read and write work again flawlessly with the latest commits of sf (370f934acdd3ba08bc61f052ff1cd34a20ef3e5f) and the latest release of GDAL (3.2.2_1) in Homebrew. Thank you a lot for your support!

rsbivand commented 3 years ago

OK, for your installation, the GDAL 3.2.1 binary was broken but 3.2.2 is not, or your source GDAL 3.2.1 was broken for reasons unknown, but 3.2.2 is not. These systems are deterministic, so it would have helped (others, later on) to establish and reproduce what was broken. However, good to hear that 3.2.2 appears to work.

ManuelSpinola commented 3 years ago

Thank you very much Roger and Alberto.

It works after updating GDAL 3.2.2

Manuel

El sáb, 3 abr 2021 a las 7:27, Roger Bivand @.***>) escribió:

OK, for your installation, the GDAL 3.2.1 binary was broken but 3.2.2 is not, or your source GDAL 3.2.1 was broken for reasons unknown, but 3.2.2 is not. These systems are deterministic, so it would have helped (others, later on) to establish and reproduce what was broken. However, good to hear that 3.2.2 appears to work.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r-spatial/sf/issues/1606#issuecomment-812865564, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFI3FB6MKTSPKQENYS4QKMTTG4JUHANCNFSM4X45562A .

-- Manuel Spínola, Ph.D. Instituto Internacional en Conservación y Manejo de Vida Silvestre Universidad Nacional Apartado 1350-3000 Heredia COSTA RICA @. @.> @.*** Teléfono: (506) 8706 - 4662 Personal website: Lobito de río https://sites.google.com/site/lobitoderio/ Institutional website: ICOMVIS http://www.icomvis.una.ac.cr/

alazarolop commented 3 years ago

@rsbivand I wouldn't say GDAL was broken at that point, since it was working properly and it was reading GPKG files in version 3.2.1. Many things have been upgraded since then, including GDAL version (not the way it was built in Homebrew) but also the sf R package (I haven't checked all the commits since the issue was opened). That's way I couldn't determine the reason, although I agree with you that it would have been interested to know it. Thank you for your support.

JimShady commented 2 years ago

I'm sporadically getting this issue (caught segfault, memory not mapped) on Ubuntu, with GDAL 3.0.2. Seems to normally happen with larger geopackages.

It seems the solution is to update GDAL to > 3.2.2 ? Or is it more complex than that?

Thanks

edzer commented 2 years ago

It seems the solution is to update GDAL to > 3.2.2 ? Or is it more complex than that?

reinstall sf after updating GDAL.

JimShady commented 2 years ago

It seems the solution is to update GDAL to > 3.2.2 ? Or is it more complex than that?

reinstall sf after updating GDAL.

Cheers