Changes requirement for income values in population, they need to be numeric now. This is to simplify how the person table is being read.
Changes the way legs table gets cost information. Before, there was a join on two columns, which turned out to be very slow. Changed it to a single column join, there is a lot of extra data that is then removed. It sacrifices memory a bit, but makes it a lot faster.
before:
2024-03-01T00:01:55,603 INFO TablesawKpiCalculator:996 Creating Population Mode Scoring Table
2024-03-01T00:02:20,946 INFO MemoryObserver:42 used RAM: 6007 MB free: 1528 MB total: 7536 MB
2024-03-01T00:03:20,953 INFO MemoryObserver:42 used RAM: 5408 MB free: 2127 MB total: 7536 MB
...
2024-03-01T01:58:21,641 INFO MemoryObserver:42 used RAM: 6432 MB free: 1103 MB total: 7536 MB
2024-03-01T01:58:49,924 INFO TablesawKpiCalculator:774 Creating Facilities Table
after:
2024-03-01T14:55:40,563 INFO TablesawKpiCalculator:1007 Creating Population Mode Scoring Table
2024-03-01T14:55:40,563 INFO TablesawKpiCalculator:1013 Reading persons file into a Table
2024-03-01T14:55:40,563 INFO TablesawKpiCalculator:1333 Reading CSV input stream into a table
2024-03-01T14:55:45,309 INFO TablesawKpiCalculator:1015 Crceated a persons table with 416794 rows
2024-03-01T14:55:45,310 INFO TablesawKpiCalculator:1018 Found no `income` column in the persons table - creating one
2024-03-01T14:55:45,326 INFO TablesawKpiCalculator:1022 Found no `subpopulation` column in the persons table - creating one
2024-03-01T14:55:45,363 INFO TablesawKpiCalculator:1033 Adding people to the person mode scores table
2024-03-01T14:55:46,381 INFO TablesawKpiCalculator:1045 Joining person mode scores table to a new temp table
2024-03-01T14:55:52,939 INFO TablesawKpiCalculator:1057 Finished populating all person-related tables
2024-03-01T14:55:52,939 INFO TablesawKpiCalculator:112 Reading legs file from stream
2024-03-01T14:55:52,939 INFO TablesawKpiCalculator:1333 Reading CSV input stream into a table
2024-03-01T14:56:12,993 INFO TablesawKpiCalculator:860 Adding costs to legs table
2024-03-01T14:56:37,389 INFO MemoryObserver:42 used RAM: 8198 MB free: 3065 MB total: 11264 MB
2024-03-01T14:56:54,534 INFO TablesawKpiCalculator:873 Compute monetary cost for each leg from scoring params
2024-03-01T14:56:54,692 INFO TablesawKpiCalculator:882 Adding contribution from person money events
2024-03-01T14:56:54,692 INFO TablesawKpiCalculator:883 Create a time columns in seconds
2024-03-01T14:56:54,693 INFO TablesawKpiCalculator:886 Adding dep_time column
2024-03-01T14:56:55,684 INFO TablesawKpiCalculator:890 Adding trav_time column
2024-03-01T14:56:56,292 INFO TablesawKpiCalculator:896 Iterating over the money log
2024-03-01T14:56:56,292 INFO TablesawKpiCalculator:909 Finished iterating over the money log
2024-03-01T14:56:56,292 INFO TablesawKpiCalculator:911 Finished adding costs to legs table
2024-03-01T14:56:56,292 INFO TablesawKpiCalculator:115 Finished reading legs file
The memory observer stats are from two different computers. Before is excerpt from a log from you @mfitz and after is from a workbox.
When I ran the before version on the same workbox, it showed me 10720 MB. So it's 10720 MB -> 11264 MB (30pc sim).
Speed improvements:
before:
after:
The memory observer stats are from two different computers. Before is excerpt from a log from you @mfitz and after is from a workbox. When I ran the before version on the same workbox, it showed me
10720 MB. So it's10720 MB->11264 MB(30pc sim).