SANDAG / ABM

Sandag ABM
https://github.com/SANDAG/ABM/wiki
20 stars 21 forks source link

MemoryError when Running Parking Location #72

Closed JoeJimFlood closed 11 months ago

JoeJimFlood commented 1 year ago

@anneku and I both have gotten a memory error in the parking location model while running the resident model. The error message I'm getting is as follows:

Traceback (most recent call last):
  File "c:\abm_runs\jflo\abm3_0822\activitysim\activitysim\core\mp_tasks.py", line 930, in mp_run_simulation
    run_simulation(queue, step_info, resume_after, shared_data_buffer)
  File "c:\abm_runs\jflo\abm3_0822\activitysim\activitysim\core\mp_tasks.py", line 882, in run_simulation
    raise e
  File "c:\abm_runs\jflo\abm3_0822\activitysim\activitysim\core\mp_tasks.py", line 879, in run_simulation
    pipeline.run_model(model)
  File "c:\abm_runs\jflo\abm3_0822\activitysim\activitysim\core\pipeline.py", line 529, in run_model
    orca.run([step_name])
  File "C:\Users\jflo\.conda\envs\abm3_0822\lib\site-packages\orca\orca.py", line 2177, in run
    step()
  File "C:\Users\jflo\.conda\envs\abm3_0822\lib\site-packages\orca\orca.py", line 973, in __call__
    return self._func(**kwargs)
  File "c:\abm_runs\jflo\abm3_0822\activitysim\activitysim\abm\models\parking_location_choice.py", line 330, in parking_location
    parking_locations, save_sample_df = run_parking_destination(
  File "c:\abm_runs\jflo\abm3_0822\activitysim\activitysim\abm\models\parking_location_choice.py", line 246, in run_parking_destination
    choices, destination_sample = choose_parking_location(
  File "c:\abm_runs\jflo\abm3_0822\activitysim\activitysim\abm\models\parking_location_choice.py", line 174, in choose_parking_location
    destination_sample = logit.interaction_dataset(
  File "c:\abm_runs\jflo\abm3_0822\activitysim\activitysim\core\logit.py", line 318, in interaction_dataset
    alts_sample = alternatives.take(sample).copy()
  File "C:\Users\jflo\.conda\envs\abm3_0822\lib\site-packages\pandas\core\generic.py", line 3871, in take
    return self._take(indices, axis)
  File "C:\Users\jflo\.conda\envs\abm3_0822\lib\site-packages\pandas\core\generic.py", line 3886, in _take
    new_data = self._mgr.take(
  File "C:\Users\jflo\.conda\envs\abm3_0822\lib\site-packages\pandas\core\internals\managers.py", line 978, in take
    return self.reindex_indexer(
  File "C:\Users\jflo\.conda\envs\abm3_0822\lib\site-packages\pandas\core\internals\managers.py", line 751, in reindex_indexer
    new_blocks = [
  File "C:\Users\jflo\.conda\envs\abm3_0822\lib\site-packages\pandas\core\internals\managers.py", line 752, in <listcomp>
    blk.take_nd(
  File "C:\Users\jflo\.conda\envs\abm3_0822\lib\site-packages\pandas\core\internals\blocks.py", line 880, in take_nd
    new_values = algos.take_nd(
  File "C:\Users\jflo\.conda\envs\abm3_0822\lib\site-packages\pandas\core\array_algos\take.py", line 117, in take_nd
    return _take_nd_ndarray(arr, indexer, axis, fill_value, allow_fill)
  File "C:\Users\jflo\.conda\envs\abm3_0822\lib\site-packages\pandas\core\array_algos\take.py", line 158, in _take_nd_ndarray
    out = np.empty(out_shape, dtype=dtype)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.75 TiB for an array with shape (70, 3428117913) and data type int64

The 3.4 billion number is the product of the number of choosers on that thread and the number of alternatives. Do we need to reduce those numbers, or is it a bug in the ActivitySim code itself?

JoeJimFlood commented 11 months ago

Fixed by only running the parking location choice model for trips ending in parking constrained areas