Scoring of negative activity durations

gac55 commented 3 years ago

Consider an agent who has busy plans and struggles to fulfil their plans. They may end up with plans that feature trips where they end up late to their activity.

In this case they may end up with a negative duration for this activity. There is a lateArrival param for penalising, but what does MATSim do in these extreme cases? We of course want to score these plans badly.

We are seeing some strange walking plans scoring better than driving, for very long/complex plans.

gac55 commented 3 years ago

Upon exploration we have discovered this

kainagel commented 3 years ago

The issue that you reference has to do with the "wrap-around" activity, which assumes that the last and first activity of the day are the same, and for that "stitched together" activity assumes the evening arrival time as activity start time, and the morning departure time as activity end time. (Except of the two activities are of different type, in which case "something else" happens, which I would need to look up.). There is a config switch usingOldScoringBelowZeroUtilityDuration which should be set to false (the default).

For all other activities, persons need to arrive before they depart, so negative durations cannot happen.

Could you please check your setting of usingOldScoringBelowZeroUtilityDuration before I start thinking about other possibilities? Thanks

JWJoubert commented 3 years ago

@gac55 maybe just a question for clarity. You mention

We are seeing some strange walking plans scoring better than driving, for very long/complex plans.

Are you seeing this in the plans.xml.gz file or the experienced_plans.xml.gz? Because we also saw some odd behaviour in s scenario, but that was in the agents' intended plan, not the experienced plan.

gac55 commented 3 years ago

Thank you kindly @kainagel

Good to know, I have checked my config - <param name="usingOldScoringBelowZeroUtilityDuration" value="false" />.

Thanks @JWJoubert . These are in the final experienced plans.

Let me give a concrete (but extreme) example. below is the selected plan (featuring very long walks, with a total score of -3726 and then the unselected (but reasonable!) plan (featuring driving, with a total score of -7315).

plan score="-3726.870869544291" selected="yes">
            <activity type="home" link="" x="" y="" end_time="10:04:07" >
            </activity>
            <leg mode="walk" dep_time="10:04:07" trav_time="09:52:49">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">walk</attribute>
                </attributes>
                <route type="generic" start_link="" end_link="" trav_time="09:52:49" distance="29640.874401379406"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="08:47:24" >
            </activity>
            <leg mode="walk" dep_time="08:47:24" trav_time="07:08:26">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">ferry</attribute>
                </attributes>
                <route type="generic" start_link="" end_link="" trav_time="07:08:26" distance="21422.11969409629"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="08:26:54" >
            </activity>
            <leg mode="walk" dep_time="08:26:54" trav_time="07:08:26">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">ferry</attribute>
                </attributes>
                <route type="generic" start_link="" end_link="" trav_time="07:08:26" distance="21422.11969409629"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="10:19:40" >
            </activity>
            <leg mode="walk" dep_time="10:19:40" trav_time="05:30:53">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">walk</attribute>
                </attributes>
                <route type="generic" start_link="" end_link="182745" trav_time="05:30:53" distance="16544.943043716183"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="12:19:01" >
            </activity>
            <leg mode="walk" dep_time="12:19:01" trav_time="04:25:03">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">walk</attribute>
                </attributes>
                <route type="generic" start_link="182745" end_link="" trav_time="04:25:03" distance="13252.76924694734"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="12:49:11" >
            </activity>
            <leg mode="walk" dep_time="12:49:11" trav_time="15:20:41">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">walk</attribute>
                </attributes>
                <route type="generic" start_link="" end_link="" trav_time="15:20:41" distance="46034.99906114046"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="13:17:46" >
            </activity>
            <leg mode="walk" dep_time="13:17:46" trav_time="16:03:03">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">walk</attribute>
                </attributes>
                <route type="generic" start_link="" end_link="" trav_time="16:03:03" distance="48152.564143100375"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="15:31:25" >
            </activity>
            <leg mode="walk" dep_time="15:31:25" trav_time="20:59:30">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">walk</attribute>
                </attributes>
                <route type="generic" start_link="242923" end_link="" trav_time="20:59:30" distance="62975.12340852869"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="16:00:58" >
            </activity>
            <leg mode="walk" dep_time="16:00:58" trav_time="20:59:30">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">walk</attribute>
                </attributes>
                <route type="generic" start_link="" end_link="242923" trav_time="20:59:30" distance="62975.12340852869"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="17:00:39" >
            </activity>
            <leg mode="walk" dep_time="17:00:39" trav_time="30:17:54">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">walk</attribute>
                </attributes>
                <route type="generic" start_link="" end_link="" trav_time="30:17:54" distance="90895.70921412861"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="16:37:33" >
            </activity>
            <leg mode="walk" dep_time="16:37:33" trav_time="09:52:49">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">walk</attribute>
                </attributes>
                <route type="generic" start_link="" end_link="" trav_time="09:52:49" distance="29640.874401379406"></route>
            </leg>
            <activity type="home" link="" x="" y="" >
            </activity>
        </plan>

        <plan score="-7315.8224944899" selected="no">
            <activity type="home" link="" x="" y="5644901.804574376" end_time="07:06:00" >
            </activity>
            <leg mode="car" dep_time="07:06:00" trav_time="00:21:07">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="" end_link="" trav_time="00:21:07" distance="26028.583165393335" vehicleRefId="15-1880-1-1_0"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="07:37:00" >
            </activity>
            <leg mode="car" dep_time="07:37:00" trav_time="00:28:31">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="" end_link="262490" trav_time="00:28:31" distance="27778.07422585695" vehicleRefId="15-1880-1-1_0"></route>
            </leg>
            <activity type="work" link="262490" x="" y="" end_time="09:01:00" >
            </activity>
            <leg mode="car" dep_time="09:01:00" trav_time="00:26:01">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="262490" end_link="" trav_time="00:26:01" distance="19986.04503023014" vehicleRefId="15-1880-1-1_0"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="11:29:00" >
            </activity>
            <leg mode="car" dep_time="11:29:00" trav_time="00:11:02">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="" end_link="182745" trav_time="00:11:02" distance="14173.939929061144" vehicleRefId="15-1880-1-1_0">5</route>
            </leg>
            <activity type="work" link="182745" x="" y="" end_time="12:38:00" >
            </activity>
            <leg mode="car" dep_time="12:38:00" trav_time="00:08:38">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="182745" end_link="" trav_time="00:08:38" distance="11001.780490013083" vehicleRefId="15-1880-1-1_0"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="12:53:00" >
            </activity>
            <leg mode="car" dep_time="12:53:00" trav_time="00:36:35">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="" end_link="" trav_time="00:36:35" distance="42515.63769648336" vehicleRefId="15-1880-1-1_0"></route>
            </leg>
            <activity type="work" link="509562" x="" y="" end_time="13:33:00" >
            </activity>
            <leg mode="car" dep_time="13:33:00" trav_time="00:31:36">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="" end_link="" trav_time="00:31:36" distance="45151.11801200648" vehicleRefId="15-1880-1-1_0"></route>
            </leg>
            <activity type="work" link="242923" x="" y="" end_time="14:06:00" >
            </activity>
            <leg mode="car" dep_time="14:06:00" trav_time="00:58:51">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="" end_link="" trav_time="00:58:51" distance="75851.25994286442" vehicleRefId="15-1880-1-1_0"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="15:13:00" >
            </activity>
            <leg mode="car" dep_time="15:13:00" trav_time="00:54:37">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="" end_link="" trav_time="00:54:37" distance="75866.48178271756" vehicleRefId="15-1880-1-1_0"></route>
            </leg>
            <activity type="work" link="242923" x="" y="" end_time="16:08:00" >
            </activity>
            <leg mode="car" dep_time="16:08:00" trav_time="01:16:40">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="" end_link="" trav_time="01:16:40" distance="99843.19524434919" vehicleRefId="15-1880-1-1_0"></route>
            </leg>
            <activity type="work" link="" x="" y="" end_time="17:20:00" >
            </activity>
            <leg mode="car" dep_time="17:20:00" trav_time="00:26:23">
                <attributes>
                    <attribute name="routingMode" class="java.lang.String">car</attribute>
                </attributes>
                <route type="links" start_link="" end_link="" trav_time="00:26:23" distance="25566.852658668402" vehicleRefId="15-1880-1-1_0">7</route>
            </leg>
            <activity type="home" link="" x="" y="" >
            </activity>
        </plan>

    </person>

I should caveat that this is an extreme example, with a very long trip, but it is strange how these extreme walks seem to make it through.

Our modeParams for context:

         <parameterset type="modeParams">
            <param name="constant" value="0.0" />
            <param name="marginalUtilityOfDistance_util_m" value="-0.002" />
            <param name="marginalUtilityOfTraveling_util_hr" value="0" />
            <param name="mode" value="walk" />
            <param name="monetaryDistanceRate" value="0.0" />
         </parameterset>
         <parameterset type="modeParams">
            <param name="constant" value="-5.0" />
            <param name="marginalUtilityOfDistance_util_m" value="0.0" />
            <param name="marginalUtilityOfTraveling_util_hr" value="-0.0" />
            <param name="mode" value="car" />
            <param name="monetaryDistanceRate" value="-0.00009" />
         </parameterset>

gac55 commented 3 years ago

Mode params in context:

kainagel commented 3 years ago

Are these truly experienced plans? They look rather like "planned" plans.

One thing that could happen: Your simulations cuts at least the walk plan short, e.g. if you terminate the mobsim after 30:00:00 hours. There is a "stuck" penalty, but it may not be large enough to overcome the situation you have here. It is defined in ScoringParameters as follows:

            abortedPlanScore = Math.min(
                    Math.min(marginalUtilityOfLateArrival_s, marginalUtilityOfEarlyDeparture_s),
                    Math.min(worstMarginalUtilityOfTraveling_s-marginalUtilityOfPerforming_s, marginalUtilityOfWaiting_s-marginalUtilityOfPerforming_s)
            ) * 3600.0 * 24.0; // SCENARIO_DURATION
            // TODO 24 has to be replaced by a variable like scenario_dur (see also other places below)
            // This rather complicated definition has to do with the fact that exp(some_large_number) relatively quickly becomes Inf.
            // In consequence, the abortedPlanScore needs to be more strongly negative than anything else, but not much more.
            // kai, feb'12

The code would allow to override it, but there is no config setting for it. It is also, as the comment says, not totally easy to set it to a useful value. (Also, if in fact both plans are aborted, it will not make a difference.)

As a first attempt, I would try to set up a a special agent with exactly the two plans you have given above, no innovation module, random plans choice, mobsimEndTime set to infty (probably have to set it to zero for that), and see if you then get reasonable scores.

gac55 commented 3 years ago

Very helpful, thanks @kainagel.

The stuck penalty sounds a lot like the theory we considered.

I like the idea of turning innovation off, I will try this.

Forgive the simple question - how do I distinguish between planned plans and experienced plans in the outputs? The original driving plan may indeed be a plan plan, I had to mine for it in the ITERS as it was being washed out and not making it to the final plans.

kainagel commented 3 years ago

There is a setting writeExperiencedPlans in planCalcScore config group. Set this to "yes", and you should get "experienced" plans in each iterations directory, meaning that they should contain the actual times, not the "planned" times.

JWJoubert commented 3 years ago

how do I distinguish between planned plans and experienced plans in the outputs?

Not a silly question, @gac55. You have to actually switch it on in the config (or programmatically). It will then write a file that is (literally) called experienced_plans.xml.gz ... ah, @kainagel beat me to it ;-)

gac55 commented 3 years ago

Thank you both. You were ahead of the curve @JWJoubert, I thought I knew what you meant ;)

I will do some more digging and run some sims over the week - happy weekend!

gac55 commented 3 years ago

Thank you both again. We have some new results:

I ran a simulation without innovation and sure enough we got reasonable plans for those initially allocated driving (but for whom we saw walking scoring highest). Although the experienced plan was sometimes subtly different (i.e. as a result of congestion) it scored well. There is nothing untoward happening with their "reasonable" alternative to explain why they are taking the very poor option. Reassuring.
We analysed in more detail the walking plans which had scored well compared to the more reasonable car option. 100% of these plans had negative activity durations. Take this example below, we see a growing delay through the day, with negative activity durations becoming ever more extreme. Despite these negative durations, the plan was selected and a positive score achieved.

We are now running a 72hour day simulation in order to confirm the theory. We expect this to help remove many of these cases. Of course, this isn't a very practical solution as it requires more compute and run time. If we confirm our theory I think we will have to adjust the ScoringParameters to ensure such an event is sufficiently penalised.

A final query - Why is the experienced plans not the default output?

gac55 commented 3 years ago

I wonder if the discrete mode choice implementation would reduce the occurrences of this. Walking trips would be very unlikely to be suggested as an innovation strategy as their estimated utility would be deemed very poor and would, in theory, be unlikely to be ever simulated.

kainagel commented 3 years ago

This looks odd. Could you please send (i.e. add here into github):

the plans file
the experienced plans file
the events file Optimally, these would be filtered for that particular agent.

fredshone commented 3 years ago

What are your thoughts about the difference between the "planned" and "experienced" plans? Our current working assumption is that times and durations can be different due to the difference between expected and simulated travel times. But we also see changes in mode-share, particulalry from PT modes into car, which i was not expecting.

gac55 commented 3 years ago

This looks odd. Could you please send (i.e. add here into github):

the plans file

the experienced plans file

the events file Optimally, these would be filtered for that particular agent.

Thanks Kai. We're trying to re-create in a test case. If that doesn't work we'll anonymise our current simulation and share here.

kainagel commented 3 years ago

Thanks. In the end, we don't need the full files. What we would need:

The "planned" plan of the concerned agent, and the name of the file from where it comes (I just want to make sure that it is the iteration input file).
The "experienced" plan of the concerned agent.
All events of that concerned agent. This will not include linkEnter/Leave events since they are linked to the vehicle, but I do not think that they are neede.

fredshone commented 3 years ago

I've not been able to reproduce the above using toy data, but in the process have got a little more familiar with the fact that we have been using planned rather than experienced plans 🤦🏻

We uncovered that the significant difference in mode-share between planned and experienced plans was due to a systematic scaling error causing some buses to get stuck, therefore curtailing some experienced plans using bus (🤦🏻🤦🏻).

I think at this point we are seeing less/no weird scoring/choices - presumably because plans are not getting 'stuck' as often.

matsim-org / matsim-libs

Scoring of negative activity durations #1595