search

arup-group / elara

Command line utility for processing MATSim events output files.
MIT License
14 stars 4 forks source link

fixes index loss for vkt #210

Closed fredshone closed 2 years ago

fredshone commented 2 years ago

hopefully fixes the mystery of the missing link ids found in the VKT outputs.

My rough diagnosis was that geopandas was replacing index name with "id".

I've simply moved the required indexes into regular fields called "link_id" and (for example) "subpopulation" as required for grouping. Needs testing on a real dataset.

You can ignore the changes to pam.main.

fredshone commented 2 years ago

@alex-kaye pretty sure the link id "loss" for groupby tables was fixed for VehicleLinkCounts as part of this PR.

For example, running tests/test_2_event_handlers.py::test_volume_count_finalise_car gives two tables:

link_vehicle_counts_car_subpopulation

   link_id from to   length  lanes  capacity    modes                                           geometry subpopulation    0    1  ...   14   15       16       17   18   19   20   21   22   23    total
0      1-2    1  2    100.0    1.0    1000.0  bus,car  LINESTRING (-7.55716 49.76681, -7.55578 49.76687)          poor  0.0  0.0  ...  0.0  0.0      0.0  10000.0  0.0  0.0  0.0  0.0  0.0  0.0  20000.0
1      1-2    1  2    100.0    1.0    1000.0  bus,car  LINESTRING (-7.55716 49.76681, -7.55578 49.76687)          rich  0.0  0.0  ...  0.0  0.0  10000.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  20000.0
2      1-2    1  2    100.0    1.0    1000.0  bus,car  LINESTRING (-7.55716 49.76681, -7.55578 49.76687)           NaN  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
3      1-5    1  5  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.55716 49.76681, -7.56745 49.85632)          poor  0.0  0.0  ...  0.0  0.0      0.0  10000.0  0.0  0.0  0.0  0.0  0.0  0.0  20000.0
4      1-5    1  5  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.55716 49.76681, -7.56745 49.85632)          rich  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
5      1-5    1  5  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.55716 49.76681, -7.56745 49.85632)           NaN  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
6      2-1    2  1    100.0    1.0    1000.0  bus,car  LINESTRING (-7.55578 49.76687, -7.55716 49.76681)          poor  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  10000.0
7      2-1    2  1    100.0    1.0    1000.0  bus,car  LINESTRING (-7.55578 49.76687, -7.55716 49.76681)          rich  0.0  0.0  ...  0.0  0.0  10000.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  10000.0
8      2-1    2  1    100.0    1.0    1000.0  bus,car  LINESTRING (-7.55578 49.76687, -7.55716 49.76681)           NaN  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
9      2-3    2  3  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.55578 49.76687, -7.41893 49.77337)          poor  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
10     2-3    2  3  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.55578 49.76687, -7.41893 49.77337)          rich  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  10000.0
11     2-3    2  3  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.55578 49.76687, -7.41893 49.77337)           NaN  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
12     3-2    3  2  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.41893 49.77337, -7.55578 49.76687)          poor  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
13     3-2    3  2  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.41893 49.77337, -7.55578 49.76687)          rich  0.0  0.0  ...  0.0  0.0  10000.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  10000.0
14     3-2    3  2  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.41893 49.77337, -7.55578 49.76687)           NaN  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
15     3-4    3  4    100.0    1.0    1000.0  bus,car  LINESTRING (-7.41893 49.77337, -7.41755 49.77344)          poor  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
16     3-4    3  4    100.0    1.0    1000.0  bus,car  LINESTRING (-7.41893 49.77337, -7.41755 49.77344)          rich  0.0  0.0  ...  0.0  0.0  10000.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  20000.0
17     3-4    3  4    100.0    1.0    1000.0  bus,car  LINESTRING (-7.41893 49.77337, -7.41755 49.77344)           NaN  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
18     4-3    4  3    100.0    1.0    1000.0  bus,car  LINESTRING (-7.41755 49.77344, -7.41893 49.77337)          poor  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
19     4-3    4  3    100.0    1.0    1000.0  bus,car  LINESTRING (-7.41755 49.77344, -7.41893 49.77337)          rich  0.0  0.0  ...  0.0  0.0  10000.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  10000.0
20     4-3    4  3    100.0    1.0    1000.0  bus,car  LINESTRING (-7.41755 49.77344, -7.41893 49.77337)           NaN  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
21     5-1    5  1  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.56745 49.85632, -7.55716 49.76681)          poor  0.0  0.0  ...  0.0  0.0      0.0  10000.0  0.0  0.0  0.0  0.0  0.0  0.0  10000.0
22     5-1    5  1  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.56745 49.85632, -7.55716 49.76681)          rich  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0
23     5-1    5  1  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.56745 49.85632, -7.55716 49.76681)           NaN  0.0  0.0  ...  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0      0.0

and:

  link_id from to   length  lanes  capacity    modes                                           geometry    0    1    2    3  ...   13   14   15       16       17   18   19   20   21   22   23    total
0     1-5    1  5  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.55716 49.76681, -7.56745 49.85632)  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0      0.0  10000.0  0.0  0.0  0.0  0.0  0.0  0.0  20000.0
1     3-2    3  2  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.41893 49.77337, -7.55578 49.76687)  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0  10000.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  10000.0
2     2-3    2  3  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.55578 49.76687, -7.41893 49.77337)  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0      0.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  10000.0
3     3-4    3  4    100.0    1.0    1000.0  bus,car  LINESTRING (-7.41893 49.77337, -7.41755 49.77344)  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0  10000.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  20000.0
4     1-2    1  2    100.0    1.0    1000.0  bus,car  LINESTRING (-7.55716 49.76681, -7.55578 49.76687)  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0  10000.0  10000.0  0.0  0.0  0.0  0.0  0.0  0.0  40000.0
5     2-1    2  1    100.0    1.0    1000.0  bus,car  LINESTRING (-7.55578 49.76687, -7.55716 49.76681)  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0  10000.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  20000.0
6     5-1    5  1  10000.0    1.0    1000.0  bus,car  LINESTRING (-7.56745 49.85632, -7.55716 49.76681)  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0      0.0  10000.0  0.0  0.0  0.0  0.0  0.0  0.0  10000.0
7     4-3    4  3    100.0    1.0    1000.0  bus,car  LINESTRING (-7.41755 49.77344, -7.41893 49.77337)  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0  10000.0      0.0  0.0  0.0  0.0  0.0  0.0  0.0  10000.0

Both have a column called link_id which looks correct.

fredshone commented 2 years ago

...did your results come from a bitsim run or your own elara job - is it possible you used an old version of elara? Changes are quite recent.

If it's the former case can you point me at the offending files on EFS please.

alex-kaye commented 2 years ago

I took the file outputs from the TE AWS account:

/mnt/efs/10pc_swarming/best3/100/elara/link_vehicle_counts_car_sub_subpopulation.csv
/mnt/efs/10pc/2019_baseline/200/elara/link_vehicle_counts_car_sub_subpopulation.csv

Downloaded to SharePoint saved here and here, respectively

alex-kaye commented 2 years ago

For example, this gives the link_id's shown below. I would expect the link_id to either be from_to , or the simplified id generated from genet

image