ActivitySim / activitysim

An Open Platform for Activity-Based Travel Modeling
https://activitysim.github.io
BSD 3-Clause "New" or "Revised" License
189 stars 96 forks source link

Non-mandatory tour frequency occasionally removes row from choosers table #841

Closed JoeJimFlood closed 2 months ago

JoeJimFlood commented 3 months ago

We've had a couple of ABM3 runs fail during non-mandatory tour frequency due to a record being removed from the choosers table when doing some of the initial data processing. In both cases, it was resolved simply by starting over the full run from scratch. This is the error message that appears:

Traceback (most recent call last):
  File "\\jupiter\abm\dev\activitysim\activitysim\core\mp_tasks.py", line 930, in mp_run_simulation
    run_simulation(queue, step_info, resume_after, shared_data_buffer)
  File "\\jupiter\abm\dev\activitysim\activitysim\core\mp_tasks.py", line 882, in run_simulation
    raise e
  File "\\jupiter\abm\dev\activitysim\activitysim\core\mp_tasks.py", line 879, in run_simulation
    pipeline.run_model(model)
  File "\\jupiter\abm\dev\activitysim\activitysim\core\pipeline.py", line 529, in run_model
    orca.run([step_name])
  File "C:\ProgramData\Anaconda3\envs\asim_baydag\lib\site-packages\orca\orca.py", line 2177, in run
    step()
  File "C:\ProgramData\Anaconda3\envs\asim_baydag\lib\site-packages\orca\orca.py", line 973, in __call__
    return self._func(**kwargs)
  File "\\jupiter\abm\dev\activitysim\activitysim\abm\models\non_mandatory_tour_frequency.py", line 174, in non_mandatory_tour_frequency
    expressions.assign_columns(
  File "\\jupiter\abm\dev\activitysim\activitysim\core\expressions.py", line 138, in assign_columns
    results = compute_columns(df, model_settings, locals_dict, trace_label)
  File "\\jupiter\abm\dev\activitysim\activitysim\core\expressions.py", line 112, in compute_columns
    results, trace_results, trace_assigned_locals = assign.assign_variables(
  File "\\jupiter\abm\dev\activitysim\activitysim\core\assign.py", line 381, in assign_variables
    raise err
  File "\\jupiter\abm\dev\activitysim\activitysim\core\assign.py", line 355, in assign_variables
    expr_values = to_series(eval(expression, globals_dict, _locals_dict))
  File "<string>", line 1, in <module>
  File "\\jupiter\abm\dev\activitysim\activitysim\abm\models\util\overlap.py", line 344, in person_available_periods
    availability = pd.Series(availability, index=persons.index)
  File "C:\ProgramData\Anaconda3\envs\asim_baydag\lib\site-packages\pandas\core\series.py", line 461, in __init__
    com.require_length_match(data, index)
  File "C:\ProgramData\Anaconda3\envs\asim_baydag\lib\site-packages\pandas\core\common.py", line 571, in require_length_match
    raise ValueError(
ValueError: Length of values (67125) does not match length of index (67126)

The expression that is causing the error is this one. I noticed that the expression before is identical except for continuous=False in the call of person_available_periods(), leading me to believe that the issue is occurring in this function, as the only additional step that is added when the continuous variable is set to True is that function being called.

dhensle commented 2 months ago

The issue seems to be with the line c = np.bincount(starts[:, 0]). The np.bincount function counts the number of occurrences of each value in the input array. If the maximum value in the input array is less than the length of the array, the output of np.bincount will be shorter than the input array.

To fix this I added a minlength parameter to np.bincount to ensure that the output is of the desired length, as see in #854.

It is still a little unclear exactly why this is happening. The issue can often go away when just re-running. I believe @JoeJimFlood is the only one to hit this error so far and it is just one case in a full model run that get's lost. It seems like perhaps some person has an extremely unlikely daily pattern where the starts / stops happen to cause the (diffs == 1) to never be true.

Regardless of the root cause, the fix seems to prevent the crash.