fix #431 - projection sf version >1.0.0

[mvarewyck]

Fix for createShapeData() with sf version >= 1.0.0

[mvarewyck]

I checked sf::sf_use_s2(FALSE) also works for sf <= 1.0.0, so removed the restriction in DESCRIPTION. Dockerfile build works again.

[SanderDevisscher]

@mvarewyck I've uploaded the spatialdata with createShapeData() again. But after succesfully building the app the issue persists.

Also I think something else might be going on. Because when I look at the private page, these spatial files seem to work just fine. So between spatialDataWBE_sf.RData & the seperate wbe files might lay the solution.

[mvarewyck]

So between spatialDataWBE_sf.RData & the seperate wbe files might lay the solution.

I don't think so.

The maps on the public page (using spatialData_sf.RData) do NOT look fine, while the ones on the private page (using either spatialDataWBE_sf.RData for 'admin' or the seperate wbe files) do look fine. So, the problem is in spatialData_sf.RData. These shape files are indeed handled differently wrt projection: https://github.com/inbo/reporting-rshiny-grofwildjacht/blob/master/reporting-grofwild/R/data_create.R#L56-L61 I just checked and this step is needed, given the provinces.geojson input file.

I tried to mimick the creation of shape data as you do, and it seems to work

> packageVersion("sf")
[1] ‘1.0.14’
## My local version writes only to 'spatialData_mv.RData'
> createShapeData()
Spherical geometry (s2) switched off
Linking to GEOS 3.8.0, GDAL 3.0.4, PROJ 6.3.1; sf_use_s2() is FALSE
although coordinates are longitude/latitude, st_union assumes that they are
planar
[1] TRUE
> readS3(file = "spatialData_mv.RData")
> leaflet(spatialData$provinces) %>% addTiles() %>% addPolygons()
> sf_use_s2()
[1] FALSE

@SanderDevisscher Do you also get this notification sf_use_s2() is FALSE ? You only get it once, ie. the first time you run createShapeData() in a fresh R session. Later, you can ask for it using sf_use_s2()

[SanderDevisscher]

@mvarewyck at the start of createShapeData() I get the following notification:

Spherical geometry (s2) switched off
Linking to GEOS 3.11.2, GDAL 3.6.2, PROJ 9.2.0; sf_use_s2() is FALSE

and this notification at the end:

although coordinates are longitude/latitude, st_union assumes that they are planar
Warning message:
package ‘sf’ was built under R version 4.3.1 

and running sf_use_s2() after the execution yields: [1] FALSE

[mvarewyck]

I don't see why it's going wrong then.

Can you test this?

packageVersion("sf")
## [1] ‘1.0.14’
jsonDir <- "~/git/reporting-rshiny-grofwildjacht/data"
iLevel <- "provinces"
sf::sf_use_s2(FALSE)
file <- file.path(jsonDir, paste0(iLevel, ".geojson"))
shapeData <- geojsonsf::geojson_sf(file)
## This should give an ugly plot
library(leaflet)
leaflet(shapeData) %>% addTiles %>% addPolygons()
## This should give nice plot
suppressWarnings(sf::st_crs(shapeData) <- 31370)
shapeData <- sf::st_transform(shapeData, crs = "+proj=longlat +datum=WGS84")
leaflet(shapeData) %>% addTiles %>% addPolygons()

I can replace the spatialData_sf.RData for now if it's urgent.

[SanderDevisscher]

I don't see why it's going wrong then.

Can you test this?

packageVersion("sf")
## [1] ‘1.0.14’
jsonDir <- "~/git/reporting-rshiny-grofwildjacht/data"
had to change this to "~/github/reporting-rshiny-grofwildjacht/data"

iLevel <- "provinces" sf::sf_use_s2(FALSE) file <- file.path(jsonDir, paste0(iLevel, ".geojson")) shapeData <- geojsonsf::geojson_sf(file)

This should give an ugly plot

library(leaflet) leaflet(shapeData) %>% addTiles %>% addPolygons() Ugly indeed

## This should give nice plot
suppressWarnings(sf::st_crs(shapeData) <- 31370)
shapeData <- sf::st_transform(shapeData, crs = "+proj=longlat +datum=WGS84")
leaflet(shapeData) %>% addTiles %>% addPolygons()

Like it should be

I can replace the spatialData_sf.RData for now if it's urgent.

Would be nice but not sustainable! This needs a fix. Can you include this file? Jachtter_2023-2024_dis.zip

I'll disable the spatialData update for now to prevent accidental overwriting.

[SanderDevisscher]

maybe its a difference between linux & windows. I'll try to run createShapeData() in a docker container.

[SanderDevisscher]

I'll try this command

docker run --name test -p 4000:3838 inbo/grofwild R -e "Sys.setenv('AWS_DEFAULT_REGION'='eu-west-1', 'AWS_ACCESS_KEY_ID'='AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY'='AWS_SECRET_ACCESS_KEY');reportingGrofwild::createShapeData(jsonDir = 'C:\\Users\\SANDER~1\\AppData\\Local\\Temp\\RtmpkDqYQn\\spatialData\\')"

[SanderDevisscher]

vermoedelijk doe ik nog iets mis maar ik krijg deze error als ik het via docker probeer:

Error in rcpp_geojson_to_sf(geojson, expand_geometries) : Invalid JSON
Calls: <Anonymous> ... geojson_to_sf -> geojson_to_sf.character -> rcpp_geojson_to_sf
Execution halted

[mvarewyck]

I'll try this command

I think you need to specify a docker volume, to point your local files to some folder that's accessible within docker. I don't know how the slashes work in Windows :(

docker run --name test -v C:\\Users\\SANDER~1\\AppData\\Local\\Temp\\RtmpkDqYQn\\spatialData\\:\\jsonDir-p 4000:3838 inbo/grofwild R -e "Sys.setenv('AWS_DEFAULT_REGION'='eu-west-1', 'AWS_ACCESS_KEY_ID'='AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY'='AWS_SECRET_ACCESS_KEY');reportingGrofwild::createShapeData(jsonDir = '/jsonDir')"

[SanderDevisscher]

if I use your command verbatim I get this error:

Unable to find image '4000:3838' locally
docker: Error response from daemon: pull access denied for 4000, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.

[SanderDevisscher]

after modding the command to single slashes I get this error:

Error in rcpp_geojson_to_sf(geojson, expand_geometries) : Invalid JSON
Calls: <Anonymous> ... geojson_to_sf -> geojson_to_sf.character -> rcpp_geojson_to_sf
Execution halted

[mvarewyck]

I would try to list the files in jsonDir.

docker run --name test -v C:\\Users\\SANDER~1\\AppData\\Local\\Temp\\RtmpkDqYQn\\spatialData\\:\\jsonDir -p 4000:3838 inbo/grofwild R -e "Sys.setenv('AWS_DEFAULT_REGION'='eu-west-1', 'AWS_ACCESS_KEY_ID'='AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY'='AWS_SECRET_ACCESS_KEY');list.files('/jsonDir')"

Anyhow, it's most efficient if we do a quick debug during the meeting tomorrow :)

[SanderDevisscher]

using docker cp I was able copy the files to "/jsonDir". rerunning the container after yields the same result

[SanderDevisscher]

list.files yields the following:

2023-10-25 17:31:35  [1] "communes.geojson"               "faunabeheerzones.geojson"      
2023-10-25 17:31:35  [3] "fbz_gemeentes.geojson"          "flanders.geojson"              
2023-10-25 17:31:35  [5] "Jachtter_2016-2017_dis.geojson" "Jachtter_2017-2018_dis.geojson"
2023-10-25 17:31:35  [7] "Jachtter_2018-2019_dis.geojson" "Jachtter_2019-2020_dis.geojson"
2023-10-25 17:31:35  [9] "Jachtter_2020-2021_dis.geojson" "Jachtter_2021-2022_dis.geojson"
2023-10-25 17:31:35 [11] "Jachtter_2022-2023_dis.geojson" "Jachtter_2023-2024_dis.geojson"
2023-10-25 17:31:35 [13] "provinces.geojson"              "utm5.geojson"                  
2023-10-25 17:31:35 [15] "WBE_binnengrenzen_2014.geojson" "WBE_binnengrenzen_2015.geojson"
2023-10-25 17:31:35 [17] "WBE_binnengrenzen_2016.geojson" "WBE_binnengrenzen_2017.geojson"
2023-10-25 17:31:35 [19] "WBE_binnengrenzen_2018.geojson" "WBE_binnengrenzen_2019.geojson"
2023-10-25 17:31:35 [21] "WBE_binnengrenzen_2020.geojson" "WBE_binnengrenzen_2021.geojson"
2023-10-25 17:31:35 [23] "WBE_binnengrenzen_2022.geojson" "WBE_binnengrenzen_2023.geojson"

[mvarewyck]

Would be nice but not sustainable! This needs a fix. Can you include this file? Jachtter_2023-2024_dis.zip

Is it fine if I just overwrite the spatialData_sf.RData? Or does the shape data on the private app also needs an update? I don't have Jachtter_2022-2023

[SanderDevisscher]

apparently I was reading from an empty folder. For https://github.com/inbo/reporting-rshiny-grofwildjacht/pull/435#issuecomment-1779532267 I used ;reportingGrofwild::createShapeData(jsonDir = '\\jsonDir')" while for the list.files() which worked I used ;list.files('/jsonDir')"

I've changed the command to

docker run --name test -p 4000:3838 inbo/grofwild R -e "Sys.setenv('AWS_DEFAULT_REGION'='eu-west-1', 'AWS_ACCESS_KEY_ID'='AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY'='AWS_SECRET_ACCESS_KEY');reportingGrofwild::createShapeData(jsonDir = '/jsonDir')"

now I get this error:

Error in st_area.sfc(st_geometry(x), ...) : 
package lwgeom required, please install it first
Calls: <Anonymous> ... <Anonymous> -> st_area.sf -> st_area -> st_area.sfc
Execution halted

My local lwgeom version:

packageVersion("lwgeom")
[1] ‘0.2.13’

[SanderDevisscher]

Would be nice but not sustainable! This needs a fix. Can you include this file? Jachtter_2023-2024_dis.zip

Is it fine if I just overwrite the spatialData_sf.RData? Or does the shape data on the private app also needs an update? I don't have Jachtter_2022-2023

An update of the data for the private app is the main reason I ran createShapeData(). The public app spatialData is mostly unchanged.
Jachtter_2022-2023_dis.zip

[mvarewyck]

now I get this error:

Error in st_area.sfc(st_geometry(x), ...) : 
package lwgeom required, please install it first
Calls: <Anonymous> ... <Anonymous> -> st_area.sf -> st_area -> st_area.sfc
Execution halted

Getting closer. I pushed a new dockerfile which should fix this issue.

[SanderDevisscher]

createShapeData() completed without throwing an error in docker but it seems the error persists :-(

[mvarewyck]

createShapeData() completed without throwing an error in docker but it seems the error persists :-(

I can now reproduce in docker with 'sf' version > 1.0.0

[SanderDevisscher]

createShapeData() completed without throwing an error in docker but it seems the error persists :-(

I can now reproduce in docker with 'sf' version > 1.0.0

The error you mean ? ifso jippie ??

[mvarewyck]

createShapeData() completed without throwing an error in docker but it seems the error persists :-(

I can now reproduce in docker with 'sf' version > 1.0.0

The error you mean ? ifso jippie ??

No, it was because there was still a geojson.zip file in my folder. Creating the shape data with docker works correctly for me (even with --no-cache and using the latest sf). All data is now up to date on UAT.

[SanderDevisscher]

damn

[mvarewyck]

@SanderDevisscher Can you test your script again? It works for me, but I still have the old geojson files for the provinces etc.

[SanderDevisscher]

@mvarewyck can you resolve the last remaining conflict?

[mvarewyck]

@SanderDevisscher Docker builds and runs fine for me now.