starzu-sentio
commented
2 years ago After another 24h I've got more logs with OOM:
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /ors-core/data/java_pid132.hprof ...
OpenJDK 64-Bit Server VM warning: cannot dump array of type object[] with length 798381090; truncating to length 536870908
Heap dump file created [207702913082 bytes in 1905.527 secs]
Exception in thread "AsyncFileHandlerWriter-653687670" 2022-01-27T00:05:18.044902217Z java.lang.OutOfMemoryError: Java heap space
27-Jan-2022 00:05:18.044 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run Unexpected death of background thread [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
java.lang.OutOfMemoryError: Java heap space
at java.base/java.net.URI.create(URI.java:881)
at java.base/java.net.URI.resolve(URI.java:1066)
at org.apache.tomcat.util.file.ConfigFileLoader.getURI(ConfigFileLoader.java:110)
at org.apache.catalina.users.MemoryUserDatabase.backgroundProcess(MemoryUserDatabase.java:673)
at org.apache.catalina.realm.UserDatabaseRealm.backgroundProcess(UserDatabaseRealm.java:138)
at org.apache.catalina.realm.CombinedRealm.backgroundProcess(CombinedRealm.java:308)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1158)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1398)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1370)
at java.base/java.lang.Thread.run(Thread.java:829)
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" 2022-01-27T00:05:18.045890091Z java.lang.OutOfMemoryError: Java heap space
at java.base/java.net.URI.create(URI.java:881)
at java.base/java.net.URI.resolve(URI.java:1066)
at org.apache.tomcat.util.file.ConfigFileLoader.getURI(ConfigFileLoader.java:110)
at org.apache.catalina.users.MemoryUserDatabase.backgroundProcess(MemoryUserDatabase.java:673)
at org.apache.catalina.realm.UserDatabaseRealm.backgroundProcess(UserDatabaseRealm.java:138)
at org.apache.catalina.realm.CombinedRealm.backgroundProcess(CombinedRealm.java:308)2022-01-27T00:05:18.045932862Z
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1158)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1398)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1370)2022-01-27T00:05:18.045958771Z
at java.base/java.lang.Thread.run(Thread.java:829)
27 Jan 00:05:18 ERROR [routing.RoutingProfileManager] - java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
27 Jan 00:05:18 ERROR [routing.RoutingProfileManager] - Failed to initialize RoutingProfileManager instance.
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at org.heigit.ors.routing.RoutingProfileManager.initialize(RoutingProfileManager.java:178) ~[classes/:6.7.0]
at org.heigit.ors.routing.RoutingProfileManager.getInstance(RoutingProfileManager.java:72) ~[classes/:6.7.0]
at org.heigit.ors.servlet.listeners.ORSInitContextListener.lambda$contextInitialized$0(ORSInitContextListener.java:41) ~[classes/:6.7.0]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.lang.Integer.valueOf(Integer.java:1059) ~[?:?]
at org.opensphere.geometry.algorithm.ConcaveHullOpenSphere.concaveHull(ConcaveHullOpenSphere.java:260) ~[classes/:6.7.0]
at org.opensphere.geometry.algorithm.ConcaveHullOpenSphere.getConcaveHull(ConcaveHullOpenSphere.java:169) ~[classes/:6.7.0]
at org.heigit.ors.fastisochrones.Contour.concHullOfNodes(Contour.java:212) ~[classes/:6.7.0]
at org.heigit.ors.fastisochrones.Contour.createContour(Contour.java:404) ~[classes/:6.7.0]
at org.heigit.ors.fastisochrones.Contour.handleBaseCells(Contour.java:103) ~[classes/:6.7.0]
at org.heigit.ors.fastisochrones.Contour.calculateContour(Contour.java:87) ~[classes/:6.7.0]
at org.heigit.ors.routing.graphhopper.extensions.ORSGraphHopper.calculateContours(ORSGraphHopper.java:851) ~[classes/:6.7.0]
at org.heigit.ors.routing.graphhopper.extensions.ORSGraphHopper.postProcessing(ORSGraphHopper.java:675) ~[classes/:6.7.0]
at com.graphhopper.GraphHopper.process(GraphHopper.java:703) ~[graphhopper-core-v0.13.21.jar:?]
at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:680) ~[graphhopper-core-v0.13.21.jar:?]
at org.heigit.ors.routing.graphhopper.extensions.ORSGraphHopper.importOrLoad(ORSGraphHopper.java:163) ~[classes/:6.7.0]
at org.heigit.ors.routing.RoutingProfile.initGraphHopper(RoutingProfile.java:184) ~[classes/:6.7.0]
at org.heigit.ors.routing.RoutingProfile.<init>(RoutingProfile.java:128) ~[classes/:6.7.0]
at org.heigit.ors.routing.RoutingProfileLoader.call(RoutingProfileLoader.java:35) ~[classes/:6.7.0]
at org.heigit.ors.routing.RoutingProfileLoader.call(RoutingProfileLoader.java:21) ~[classes/:6.7.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
... 1 more
27 Jan 00:05:18 INFO [routing.RoutingProfileManager] - ====> Recycling garbage...
27 Jan 00:05:18 INFO [routing.RoutingProfileManager] - Before: Total - 180 GB, Free - 78.01 GB, Max: 180 GB, Used - 101.99 GB
27 Jan 00:05:20 INFO [routing.RoutingProfileManager] - After: Total - 113.75 GB, Free - 80.73 GB, Max: 180 GB, Used - 33.02 GB
27 Jan 00:05:20 INFO [routing.RoutingProfileManager] - ========================================================================
27 Jan 00:05:20 INFO [routing.RoutingProfileManager] - ====> Memory usage by profiles:
27 Jan 00:05:20 INFO [routing.RoutingProfileManager] - [1] 16.16 GB (48.9%)
27 Jan 00:05:20 INFO [routing.RoutingProfileManager] - Total: 16.16 GB (48.9%)
27 Jan 00:05:20 INFO [routing.RoutingProfileManager] - ========================================================================It seems that memory limit is at least one of the problems.
HendrikLeuschner
tflabs-nl
Here's what I did
I'm trying to generate data for Europe on k8s with pod limits set to 8 vCPU and 220GB RAM.
Docker image version:
v6.7.0Source file:http://download.geofabrik.de/europe-latest.osm.pbfEnv variables:
ORS config:
Here's what I got
ArrayIndexOutOfBoundsException (probably) during isochrones generation.
Content of
/var/log/ors/ors.log:Here's what I was expecting
I was expecting successful end of the process. I also tried the same config on smaller map:
http://download.geofabrik.de/europe/poland-latest.osm.pbf. Do you think I should retry with more RAM? If so, could you help me with proper required resources estimation?