search

SymbolixAU / geojsonsf

Conversion between sf and geojson
Other
82 stars 7 forks source link

string limitation for very large geojson #64

Closed sericson0 closed 5 years ago

sericson0 commented 5 years ago

I am trying to load the Microsoft building footprint data for Texas (https://github.com/Microsoft/USBuildingFootprints) which is 2.1 GB. However, when i run the code:

footprint_data = "(path to folder containing data)./Texas.geojson" texas_buildings = geojson_sf(footprint_data)

I get the following error

Error in rcpp_read_sf_file(normalizePath(geojson), expand_geometries) : 
  R character strings are limited to 2^31-1 bytes

The same code works for smaller files in the Microsoft data, so it seems like the error is due to the size of the file. Any suggestions would be greatly appreciated :-)

TODO

dcooley commented 5 years ago

Indeed - the solution may be to implement FileReadStream rather than, or as well as the buffer - to - StringVector implementation

SymbolixAU commented 5 years ago

working prototype

sf <- geojsonsf:::rcpp_read_sf_file_stream(normalizePath("~/Downloads/Texas.geojson"), FALSE)
sf
# Simple feature collection with 9891540 features and 0 fields
# geometry type:  POLYGON
# dimension:      XY
# bbox:           xmin: -106.646 ymin: 25.85305 xmax: -93.52176 ymax: 36.50031
# epsg (SRID):    4326
# proj4string:    +proj=longlat +datum=WGS84 +no_defs
# First 10 features:
#                          geometry
# 1  POLYGON ((-95.61533 30.7471...
# 2  POLYGON ((-95.59383 30.7382...
# 3  POLYGON ((-95.56662 30.7327...
# 4  POLYGON ((-95.55648 30.7391...
# 5  POLYGON ((-95.57037 30.7253...
# 6  POLYGON ((-95.57031 30.7267...
# 7  POLYGON ((-95.57383 30.7276...
# 8  POLYGON ((-95.5707 30.72964...
# 9  POLYGON ((-95.57335 30.7297...
# 10 POLYGON ((-95.569 30.72455,...
dcooley commented 5 years ago

timings

system.time({
  sf <- geojsonsf::geojson_sf("~/Downloads/Texas.geojson", buffer_size = 1024) ## default
})

# user  system elapsed 
# 126.185  16.056 189.778

system.time({
  sf <- geojsonsf::geojson_sf("~/Downloads/Texas.geojson", buffer_size = 2048)
})
rm(list=ls());gc()

# user  system elapsed 
# 91.989  20.256 127.668

system.time({
  sf <- geojsonsf::geojson_sf("~/Downloads/Texas.geojson", buffer_size = 65536)
})
rm(list=ls());gc()

# user  system elapsed 
# 132.908  13.697 191.704
DarioBoh commented 4 years ago

working prototype

sf <- geojsonsf:::rcpp_read_sf_file_stream(normalizePath("~/Downloads/Texas.geojson"), FALSE)
sf
# Simple feature collection with 9891540 features and 0 fields
# geometry type:  POLYGON
# dimension:      XY
# bbox:           xmin: -106.646 ymin: 25.85305 xmax: -93.52176 ymax: 36.50031
# epsg (SRID):    4326
# proj4string:    +proj=longlat +datum=WGS84 +no_defs
# First 10 features:
#                          geometry
# 1  POLYGON ((-95.61533 30.7471...
# 2  POLYGON ((-95.59383 30.7382...
# 3  POLYGON ((-95.56662 30.7327...
# 4  POLYGON ((-95.55648 30.7391...
# 5  POLYGON ((-95.57037 30.7253...
# 6  POLYGON ((-95.57031 30.7267...
# 7  POLYGON ((-95.57383 30.7276...
# 8  POLYGON ((-95.5707 30.72964...
# 9  POLYGON ((-95.57335 30.7297...
# 10 POLYGON ((-95.569 30.72455,...
geojsonsf:::rcpp_read_sf_file_stream
Error in get(name, envir = asNamespace(pkg), inherits = FALSE) : 
  object 'rcpp_read_sf_file_stream' not found

sessionInfo

─ Session info  
setting  value                       
 version  R version 3.6.3 (2020-02-29)
 os       macOS Mojave 10.14.6        
 system   x86_64, darwin15.6.0        
 ui       RStudio                     
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       America/New_York            
 date     2020-03-24      
 geojsonsf     1.3.3   2020-03-18 [1] CRAN (R 3.6.0)
SymbolixAU commented 4 years ago

The rcpp_read_sf_file_stream was my working prototype I made during development. You should now be using the R code geojsonsf::geojson_sf()