Errors when setting up large .pbf network

[friedtea15]

Hi - I'm using r5r to run a detailed itinerary process for ~50 U.S. metropolitan areas. I downloaded the US osm.pbf file from Geofabrik, then used osmconvert to clip the file based on island bounding polygons (metropolitan area extents in one .poly). The outputted .pbf file is ~4gb (available here as us_01.pbf): https://drive.google.com/drive/folders/1gNW88nVDUhKxArF-KRWGe3j1RvwC0Knr?usp=sharing

When running setup_r5(), it appears I get several errors after ~4 hours of runtime:

INFO: dataFileCache open start
Error in setup_r5(data_path = "~/Documents/logistics crunch/national_analysis/spatial_data/osm_network") : 
  java.lang.NullPointerException
Mar 17, 2023 2:29:34 AM org.geotools.referencing.factory.DeferredAuthorityFactory disposeBackingStore
INFO: Disposing class org.geotools.referencing.factory.epsg.hsql.ThreadedHsqlEpsgFactory backing store
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.hsqldb.persist.RAFileNIO (file:/Library/Frameworks/R.framework/Versions/4.2/Resources/library/r5r/jar/r5-v6.9-all.jar) to method java.nio.DirectByteBuffer.cleaner()
WARNING: Please consider reporting this to the maintainers of org.hsqldb.persist.RAFileNIO
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Mar 17, 2023 2:29:34 AM org.hsqldb.persist.Logger logInfoEvent
INFO: Database closed

Below is my r5r::r5r_sitrep():

$r5r_package_version
[1] ‘1.0.1’

$r5_jar_version
[1] "6.9"

$java_version
[1] "11.0.18"

$set_memory
[1] "-Xmx20G"

$session_info
R version 4.2.2 (2022-10-31)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 11.5.2

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] dplyr_1.1.0       ggplot2_3.4.1     data.table_1.14.8 sf_1.0-9          r5r_1.0.1        

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.10        spdep_1.2-7        compiler_4.2.2     pillar_1.8.1       LearnBayes_2.15.1 
 [6] class_7.3-20       tools_4.2.2        boot_1.3-28        checkmate_2.1.0    gtable_0.3.1      
[11] nlme_3.1-160       lifecycle_1.0.3    tibble_3.1.8       lattice_0.20-45    pkgconfig_2.0.3   
[16] rlang_1.0.6        Matrix_1.5-1       DBI_1.1.3          cli_3.6.0          rstudioapi_0.14   
[21] expm_0.999-7       rJava_1.0-6        coda_0.19-4        spData_2.2.2       e1071_1.7-13      
[26] withr_2.5.0        s2_1.1.2           generics_0.1.3     vctrs_0.5.2        classInt_0.4-8    
[31] grid_4.2.2         tidyselect_1.2.0   glue_1.6.2         R6_2.5.1           spatialreg_1.2-6  
[36] fansi_1.0.4        sp_1.6-0           deldir_1.0-6       magrittr_2.0.3     backports_1.4.1   
[41] scales_1.2.1       splines_4.2.2      MASS_7.3-58.1      units_0.8-1        colorspace_2.1-0  
[46] KernSmooth_2.23-20 utf8_1.2.3         proxy_0.4-27       wk_0.7.1           munsell_0.5.0   

I ran set up "extdata/poa" and it did not reproduce the error.

Here's the full console text. error.txt

Thank you for your work!

[friedtea15]

To be clear - the only thing present in my osm_network folder is the us_01.pbf

No GTFS files.

[friedtea15]

Ran it with a smaller extract from the input .pbf (just one MSA; stl.pbf) and produced the same error:

Mar 17, 2023 9:49:52 AM org.hsqldb.persist.Logger logInfoEvent
INFO: dataFileCache open start
Error in setup_r5(data_path = "~/Documents/logistics crunch/national_analysis/spatial_data/osm_network") : 
  java.lang.NullPointerException

Closes an hour later with illegal reflective error. Same as above.

"extdata/poa" still works successfully.

[dhersz]

Ran it with a smaller extract from the input .pbf (just one MSA; stl.pbf) and produced the same error:

Mar 17, 2023 9:49:52 AM org.hsqldb.persist.Logger logInfoEvent
INFO: dataFileCache open start
Error in setup_r5(data_path = "~/Documents/logistics crunch/national_analysis/spatial_data/osm_network") : 
  java.lang.NullPointerException

Closes an hour later with illegal reflective error. Same as above.

"extdata/poa" still works successfully.

Hi @friedtea15, could you please run setup_r5() with verbose = TRUE?

[rafapereirabr]

Hi @friedtea15 . Thanks for sharing the files. I've tried building the network with the stl.pbf file and encountered the same error message. Note that r5r works fine with OSM data downloaded from Geofabrik, though. So I assume the error emerges because the OSM is being cropped. Unfortunately, the Java error messages are often not very informative, so I cannot really spot the cause of the problem. I would suggest though you have a look at these instructions from Conveyal on how to prepare OSM data.

ps. If you do not need the geometries of the trips, I would strongly suggest you do not use do not use detailed itinerary, and instead you should use the travel time matrix function, which is way way more efficient.

[friedtea15]

Thank you @rafapereirabr and @dhersz

I was able to remove the error using this workaround: https://github.com/conveyal/r5/issues/276

It had to do with the clipping with osmconvert. Switched to osmosis and kept the ways and relations.

However, I ran into an extent error:

Error in setup_r5(data_path = "~/Documents/logistics crunch/national_analysis/spatial_data/osm_test") : 
  java.lang.IllegalArgumentException: Geographic extent of street layer (36683282 km2) exceeds limit of 975000 km2.

From the closed issues, there doesn't appear to be a workaround for this. Odd given that the the extent used covered just one MSA and most certainly not 3 million sq km. I've uploaded the updated stl.osm.pbf to the drive.

Also alas, the route geographies are all I need from the tool :)

[rafapereirabr]

Hi @friedtea15 , I'm glad to hear there is a quick fix for the problem of clipping OSM data with osmosis. Regarding the extent limit of 975000 km2, this is hardcoded upstream in R5 there is nothing we can do from the r5r side.

I guess we can close this issue now. Right?

[mvpsaraiva]

Hi @friedtea15. I've opened the stl.osm.pbf in QGIS and it looks like it still covers the entirety of the US. Please, check the clipping process to see if there's something missing. Also, I think 20GB of RAM maybe not enough for running r5r in such large areas.

[friedtea15]

Hi - sorry for the delay. I was able to complete the process. Basically, I had to iterate the process for each MSA separately to workaround the extent and processing constraints. Worked like a charm. Thanks again for the replies, consider this closed!