arup-group / genet

Manipulate MATSim networks via a Python API.
MIT License
44 stars 9 forks source link

Fix runtime increase #201

Closed KasiaKoz closed 1 year ago

KasiaKoz commented 1 year ago

Resolves #199 by changing pandas version, see issue for more details. Some small changes to readme and setup to match with https://github.com/arup-group/genet/pull/192

Some tests running intermodal-access-egress-network which surfaced the problem: PT2MATSim is our small test network in genet Londinium and Londinium in docker is a smaller Londinium network Londinium chonk is a denser Londinium (with detailed walk and bike networks)

Pre python 3.11:

PT2MATSim: 5.68s user 1.36s system 31% cpu 22.555 total
Londinium: 14.06s user 0.57s system 91% cpu 15.927 total
Londinium in docker: 0.01s user 0.03s system 0% cpu 29.674 total
Londinium chonk: 38.46s user 1.58s system 78% cpu 51.189 total

Post python 3.11:

PT2MATSim: 6.75s user 2.32s system 128% cpu 7.085 total
Londinium: 34.45s user 3.49s system 106% cpu 35.707 total
Londinium in docker:  0.01s user 0.01s system 0% cpu 52.099 total
Londinium chonk: 1483.67s user 19.35s system 98% cpu 25:26.03 total

After fix:

PT2MATSim: 9.19s user 3.28s system 33% cpu 37.537 total
Londinium: 9.31s user 2.81s system 114% cpu 10.598 total
Londinium in docker: 0.01s user 0.03s system 0% cpu 14.583 total
Londinium chonk: 21.48s user 2.51s system 103% cpu 23.143 total

The situation on AWS was even worse, the intermodal-access-egress-network process using Londinium chonk network was hanging for hours in a batch job. These links need authentication to TCM dev AWS account: