search

opentripplanner / OpenTripPlanner

An open source multi-modal trip planner
http://www.opentripplanner.org
Other
2.2k stars 1.03k forks source link

otp-2.2.0-SNAPSHOT-shaded-latest.jar does not load graph due to java.lang.OutOfMemoryError: Java heap space error #4339

Closed bennytaccardi closed 1 year ago

bennytaccardi commented 2 years ago

I have a problem using OpenTripPlanner: after building the graph (using the following command: java -Xmx24G -jar otp-2.2.0-SNAPSHOT-shaded-latest.jar --build --save ./graphs/ ), server does not load the newly created graph. The command used for loading graph into OTP is the following: java -Xmx24G -jar otp-2.2.0-SNAPSHOT-shaded-latest.jar --load ./graphs/.

I expect OTP to index the graph and start the server.

In contrast, the resulting behavior is the following.

java -Xmx28G -jar otp-2.2.0-SNAPSHOT-shaded-latest.jar --load ./graphs/
07:32:20.711 INFO (OtpStartupInfo.java:39) OTP STARTING UP (version: 2.2.0-SNAPSHOT, [ser.ver.id](http://ser.ver.id/): 49, commit: ee0859bf29a6863ecfee583bfb649c44815fad9b, branch: dev-2.x)
07:32:20.729 INFO (OtpStartupInfo.java:45) 
  ___                 _____     _       ____  _                             
 / _ \ _ __   ___ _ _|_   _| __(_)_ __ |  _ \| | __ _ _ __  _ __   ___ _ __ 
| | | | '_ \ / _ \ '_ \| || '__| | '_ \| |_) | |/ _` | '_ \| '_ \ / _ \ '__|
| |_| | |_) |  __/ | | | || |  | | |_) |  __/| | (_| | | | | | | |  __/ |   
 \___/| .__/ \___|_| |_|_||_|  |_| .__/|_|   |_|\__,_|_| |_|_| |_|\___|_| 
      |_|                        |_| 
Version:     2.2.0-SNAPSHOT
[Ser.ver.id](http://ser.ver.id/):  49
Commit:      ee0859bf29a6863ecfee583bfb649c44815fad9b
Branch:      dev-2.x
Build:       2022-08-01T09:35:16+0000

07:32:20.729 INFO (OTPMain.java:106) Searching for configuration and input files in /home/ubuntu/otp/./graphs
07:32:21.030 INFO (ConfigLoader.java:224) File './graphs/otp-config.json' is not present. Using default configuration.
07:32:21.160 INFO (ConfigLoader.java:220) Load JSON configuration file './graphs/build-config.json'
07:32:21.170 INFO (ConfigLoader.java:221) Summarizing './graphs/build-config.json': {
  "timeZone" : "Europe/Rome"
}
07:32:21.228 INFO (ConfigLoader.java:224) File './graphs/router-config.json' is not present. Using default configuration.
07:32:21.288 INFO (OTPFeature.java:68) Features turned on: 
APIBikeRental
APIServerInfo
APIGraphInspectorTile
APIUpdaterStatus
OptimizeTransfers
TransferConstraints
FloatingBike
07:32:21.289 INFO (OTPFeature.java:69) Features turned off: 
MinimumTransferTimeIsDefinitive
ParallelRouting
ActuatorAPI
DataOverlay
FlexRouting
GoogleCloudStorage
ReportApi
SandboxAPIGeocoder
SandboxAPILegacyGraphQLApi
SandboxAPIMapboxVectorTilesApi
SandboxAPITransmodelApi
SandboxAPITravelTime
SandboxAPIParkAndRideApi
TransferAnalyzer
VehicleToStopHeuristics
07:32:21.308 INFO (GraphBuilderDataSources.java:116) Data source location(s): ./graphs
07:32:21.308 INFO (GraphBuilderDataSources.java:119) Existing files expected to be read or written:
07:32:21.313 INFO (GraphBuilderDataSources.java:122) - ⚙️ build-config.json  ./graphs  2022-08-01 16:14:28  34 bytes
07:32:21.316 INFO (GraphBuilderDataSources.java:122) - 🌐 graph.obj  ./graphs  2022-08-02 06:52:51  498.0 MB
07:32:21.316 INFO (GraphBuilderDataSources.java:127) Files excluded due to command line switches or unknown type:
07:32:21.317 INFO (GraphBuilderDataSources.java:130) - 🚌 gtfs-ete-22.zip  ./graphs  2022-08-01 15:47:15  415 kb
07:32:21.317 INFO (GraphBuilderDataSources.java:130) - 🚌 car-region-haute-savoie.gtfs.zip  ./graphs  2022-08-01 15:50:18  125 kb
07:32:21.318 INFO (GraphBuilderDataSources.java:130) - 🚌 gtfs_fp2022_2022-07-27_04-15.zip  ./graphs  2022-08-01 15:46:08  110.1 MB
07:32:21.319 INFO (SerializedGraphObject.java:144) Reading graph from './graphs/graph.obj'
07:32:55.659 INFO (SerializedGraphObject.java:161) Graph read. |V|=39412 |E|=0
07:32:55.660 INFO (ConfigModel.java:73) Using the graph embedded JSON router configuration.
07:32:55.724 INFO (TransitModel.java:126) Index transit model...
07:32:55.724 INFO (StopModel.java:64) Index stop model...
07:32:55.738 INFO (StopModelIndex.java:44) StopModelIndex init...
07:32:55.833 INFO (StopModelIndex.java:79) StopModelIndex init complete.
07:32:55.833 INFO (StopModel.java:66) Index stop model complete.
07:32:55.833 INFO (TransitModelIndex.java:65) Transit model index init...
07:32:59.689 INFO (TransitModelIndex.java:120) Transit Model index init complete.
07:32:59.689 INFO (TransitModel.java:130) Index transit model complete.
07:32:59.689 INFO (Graph.java:335) Index street model...
07:33:05.617 INFO (null:-1) dataFileCache open start
07:33:05.897 INFO (StreetVertexIndex.java:388) Index street vertex progress tracking started.
07:33:05.935 INFO (StreetVertexIndex.java:410) Index street vertex progress tracking complete. 39,412 done in 0s (1,065,189 per second). 
07:33:05.935 INFO (Graph.java:337) Index street model complete.
07:33:05.944 INFO (OTPAppConstruction.java:159) Wiring up and configuring server.
07:33:06.372 INFO (OTPAppConstruction.java:213) Creating transit layer for Raptor routing.
07:33:06.373 INFO (TransitLayerMapper.java:78) Mapping transitLayer from Graph...
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.opentripplanner.routing.algorithm.raptoradapter.transit.constrainedtransfer.TransferIndexGenerator.findTPoints(TransferIndexGenerator.java:186)
at org.opentripplanner.routing.algorithm.raptoradapter.transit.constrainedtransfer.TransferIndexGenerator.findTPoints(TransferIndexGenerator.java:139)
at org.opentripplanner.routing.algorithm.raptoradapter.transit.constrainedtransfer.TransferIndexGenerator.lambda$generateTransfers$0(TransferIndexGenerator.java:66)
at org.opentripplanner.routing.algorithm.raptoradapter.transit.constrainedtransfer.TransferIndexGenerator$$Lambda$314/0x0000000800d68478.accept(Unknown Source)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEachOrdered(ReferencePipeline.java:601)
at org.opentripplanner.routing.algorithm.raptoradapter.transit.constrainedtransfer.TransferIndexGenerator.generateTransfers(TransferIndexGenerator.java:65)
at org.opentripplanner.routing.algorithm.raptoradapter.transit.mappers.TransitLayerMapper.map(TransitLayerMapper.java:99)
at org.opentripplanner.routing.algorithm.raptoradapter.transit.mappers.TransitLayerMapper.map(TransitLayerMapper.java:59)
at org.opentripplanner.standalone.configure.OTPAppConstruction.creatTransitLayerForRaptor(OTPAppConstruction.java:215)
at org.opentripplanner.standalone.configure.OTPAppConstruction.setupTransitRoutingServer(OTPAppConstruction.java:180)
at org.opentripplanner.standalone.configure.OTPAppConstruction.createApplication(OTPAppConstruction.java:160)
at org.opentripplanner.standalone.configure.OTPAppConstruction.createGrizzlyServer(OTPAppConstruction.java:115)
at org.opentripplanner.standalone.OTPMain.startOtpWebServer(OTPMain.java:193)
at org.opentripplanner.standalone.OTPMain.startOTPServer(OTPMain.java:169)
at org.opentripplanner.standalone.OTPMain.main(OTPMain.java:55)
07:43:11.187 INFO (OtpStartupInfo.java:43) OTP SHUTTING DOWN (version: 2.2.0-SNAPSHOT, [ser.ver.id](http://ser.ver.id/): 49, commit: ee0859bf29a6863ecfee583bfb649c44815fad9b, branch: dev-2.x)

Currently, I'm using the following OTP's version: Version: 2.2.0-SNAPSHOT Ser.ver.id: 49 Commit: ee0859bf29a6863ecfee583bfb649c44815fad9b Branch: dev-2.x Build: 2022-08-01T09:35:16+0000

As dataset, I'm using three different GTFS file which could be retrieved using these links:

I've also configured a build-config.json file in order to avoiding multiple timezones problem. build-config.json: { "timeZone": "Europe/Rome" }

In order to reproduce the problem, is sufficient to download the dataset as whole and build the graph using otp command: java -Xmx24G -jar otp-2.2.0-SNAPSHOT-shaded-latest.jar --build --save ./graphs/

After that, the server should be started using the command: java -Xmx24G -jar otp-2.2.0-SNAPSHOT-shaded-latest.jar --load ./graphs/

Which fails generating the described problem.

leonardehrenfried commented 2 years ago

java.lang.OutOfMemoryError means you don't assign enough memory to your JVM. What happens if you increase Xmx even more?

bennytaccardi commented 2 years ago

I am using a machine with 32 G of RAM and 32 might not be enough? How much ram should I use, in order to start the server with the second gtfs files (https://opentransportdata.swiss/it/dataset/timetable-2022-gtfs2020) (of 110Mb) which among the previous files is the one generating the out of memory

leonardehrenfried commented 2 years ago

It's hard to say but usually the street data (OSM) takes up most of the memory.

I run an instance for all of Germany and for that I allocated 96GB of memory, but I think I can squeeze it in 60GB.

Although it is a bit strange that you can build the graph but not run it. Usually graph building takes more memory.

I would try to increase Xmx to 28GB and perhaps even more by adding swap.

leonardehrenfried commented 2 years ago

If you're a Java developer you could also do some memory profiling to find out what's using all the memory.

bennytaccardi commented 2 years ago

Ok, I've instantiated a 64 Gb Ram machine. Starting server with 60gb works but now another strange behavior appeared: graph built with |V| = 39k and |E|=0. Does this mean that there are no links between nodes? In addition, OTP GUI does not suggest any stop. It sounds not so good.

leonardehrenfried commented 2 years ago

Yes, there is something strange happening with your OSM data. Where do you get your extract from?

bennytaccardi commented 2 years ago

As reported in initial issue, data sources are:

leonardehrenfried commented 2 years ago

Your list does not contain osm data. If you don't add any, you will not have street routing.

bennytaccardi commented 2 years ago

Exactly, my dataset is made up by GTFS files (files linked previously) and OSM. After some trials, I realized that GTFS files are the problem and particularly, this one: https://opentransportdata.swiss/it/dataset/timetable-2022-gtfs2020 causes the error

alessiocamillo commented 2 years ago

@leonardehrenfried Is Open Trip Planner also able to track the routes provided by gtfs for example of buses?

leonardehrenfried commented 2 years ago

Yes, the GTFS bus mode is supported but I think this is not what you're asking, isn't it? Are you talking about the shapes of the routes?

Could you perhaps rephrase the question and give more details?

bennytaccardi commented 2 years ago

No, is not about route's shapes. My initial ask is about retrieving itineraries from a mixed dataset (GTFS and OSM) using OTP. The initial problem was about the overcoming of heap space during loading phase of the OTP. Then, the problem is turned about graph without edges (I mean, |E| = 0. This happens with just GTFS data as datasource, so withouth OSM) even though build and loading phases finished successfully. I realized that version 1.4.0 and 1.5.0, run on the same dataset, highlights an error which is: According to the GTFS specification there are route types 1700 (Miscellaneous Service) with subtypes 1701 (Cable Car) and 1702 (Horse-drawn Carriage). On the other hand, version 2.2.0-SNAPSHOT highlights the error of building a graph without edges.

leonardehrenfried commented 2 years ago

I'm not sure I'm following along. Is there a question in your last post?

bennytaccardi commented 2 years ago

Currently, after building the graph (using --build option) and after starting the OTP server (using --load option), it isn't able to suggest stops and routes. I also tried to build the graph using different GTFS files (in order to exclude data errors of a single GTFS). The result is always the same: if I write a stop name in the input text of OTP GUI, it doesn't suggest any stops (and no routes will be shown consequentely).

leonardehrenfried commented 2 years ago

There is no autocomplete search in the debug client. You need to click into the map to trigger a routing request.

t2gran commented 2 years ago

My guess is that the street graph contains (|V| = 39k and |E|=0) stops only - not OSM street data. The transit model is not in the street graph, so it might be ok even if your street graph only have stops.

My suggestion is to take the data from a smaller area and set it up - so it works, then add more areas gradually. You need a OSM (xyz.osm.pbf file) to include street information to use the OTP Debug client. You can start with a OSM file, then add a GTFS file for the same area - the areas do not need to be the same, but test routing where both is overlapping.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days