Required memory, RAM, for large network, java heap space

matsim-org / pt2matsim

Package to create a multi-modal MATSim network and schedule from public transit data (GTFS or HAFAS) and an OSM map of the area.

http://www.ivt.ethz.ch/publikationen/studenten/530.html

GNU General Public License v2.0

48 stars 66 forks source link

Required memory, RAM, for large network, java heap space #194

Closed Areza622 closed 2 weeks ago

Areza622 commented 1 year ago

Hi, I want to build up a network for a large area, baden-württemberg, and I get "Java heap space" error. Currently I have 32 GB of RAM and I want to know how much memory is needed for the area that I mentioned? or maybe are there other ways to process large areas?

Thank you in advance.

j-minster commented 1 year ago

How are you calling pt2matsim? You can specify the amount of RAM the JVM can use:

java -Xmx16384m -cp ../target/pt2matsim-22.11-shaded.jar:. org.matsim.pt2matsim.run.Osm2MultimodalNetwork ./config.xml;

In this case, -Xmx16384m is saying "you can use 16Gb of my RAM". You can also do this if you're trying to run a project in an IDE.

And what do you mean by 'build up a network'? Convert OSM to XML? or perform PT routing?

Areza622 commented 1 year ago

@j-minster Hi, I use IntelliJ and earlier I tried setting the max heap size to 32000 MiB but it didn't help! By building up I meant covert OSM map to xml including puclic transportation data.

j-minster commented 1 year ago

I have trouble with IntelliJ too. It's possible you've set the wrong parameter. Can you try to edit the configuration like this:

https://www.jetbrains.com/idea/guide/tutorials/hello-world/creating-a-run-configuration/

Try to set the RAM like I have here, for example, using CLI args:

(Note that this is for a different project, don't copy these settings verbatim for this project)

I can recommend trying to run from the command line like in my previous comment though. Of course, you have to generate a config and edit that first. There is instructions for that in the wiki for this repository.

Areza622 commented 1 year ago

@j-minster Thank you for your answer, I will try that and I hope it solves the issue. Can you please tell me what is the name of the box that you have written Xmx8000m?(because in IntelliJ in my pc, I don't see that specific box!)

j-minster commented 1 year ago

I checked and it's called a 'VM option'. You can add one by pressing the text (visible in my screenshot) that says 'Modify options'.

Areza622 commented 1 year ago

@j-minster Ok, thank you so much!

ikaddoura commented 4 months ago

Maybe also try to reduce your OSM file in a previous step, e.g. reduce, filter, (...) the osm file to what you actually need by using osmosis.