Processes mp_accessibility_1 failed with exitcode -9

[gbrawn]

Running exmple_psrc_full breaks with the following traceback: I've tried to adjust the number of processes, running with and without accessibility.

WARNING - process mp_accessibility_1 failed with exitcode -9
WARNING - fail_fast terminating remaining running processes
INFO - terminating process mp_accessibility_0
INFO - terminating process mp_accessibility_2
INFO - terminating process mp_accessibility_3
INFO - terminating process mp_accessibility_4
INFO - terminating process mp_accessibility_5
INFO - terminating process mp_accessibility_6
INFO - terminating process mp_accessibility_7
INFO - terminating process mp_accessibility_8
INFO - terminating process mp_accessibility_9
INFO - terminating process mp_accessibility_10
INFO - terminating process mp_accessibility_11
INFO - terminating process mp_accessibility_12
INFO - terminating process mp_accessibility_13
INFO - terminating process mp_accessibility_14
INFO - terminating process mp_accessibility_15
INFO - terminating process mp_accessibility_16
INFO - terminating process mp_accessibility_17
INFO - terminating process mp_accessibility_18
INFO - terminating process mp_accessibility_19
INFO - terminating process mp_accessibility_20
INFO - terminating process mp_accessibility_21
INFO - terminating process mp_accessibility_22
INFO - terminating process mp_accessibility_23
INFO - terminating process mp_accessibility_24
INFO - terminating process mp_accessibility_25
INFO - terminating process mp_accessibility_26
INFO - terminating process mp_accessibility_27
INFO - Time to execute all models until this error : 256.75 seconds (4.3 minutes)
ERROR - activitysim run encountered an unrecoverable error
Traceback (most recent call last):
  File "/home/geoffrey_brawn/anaconda3/envs/asim/lib/python3.9/site-packages/activitysim/cli/run.py", line 251, in run
    mp_tasks.run_multiprocess(injectables)
  File "/home/geoffrey_brawn/anaconda3/envs/asim/lib/python3.9/site-packages/activitysim/core/mp_tasks.py", line 1364, in run_multiprocess
    completed = run_sub_simulations(injectables,
  File "/home/geoffrey_brawn/anaconda3/envs/asim/lib/python3.9/site-packages/activitysim/core/mp_tasks.py", line 1171, in run_sub_simulations
    check_proc_status()
  File "/home/geoffrey_brawn/anaconda3/envs/asim/lib/python3.9/site-packages/activitysim/core/mp_tasks.py", line 1085, in check_proc_status
    raise RuntimeError("Process %s failed" % (p.name,))
RuntimeError: Process mp_accessibility_1 failed
Traceback (most recent call last):
  File "/home/geoffrey_brawn/anaconda3/envs/asim/bin/activitysim", line 10, in <module>
    sys.exit(main())
  File "/home/geoffrey_brawn/anaconda3/envs/asim/lib/python3.9/site-packages/activitysim/cli/main.py", line 21, in main
    sys.exit(asim.execute())
  File "/home/geoffrey_brawn/anaconda3/envs/asim/lib/python3.9/site-packages/activitysim/cli/cli.py", line 27, in execute
    args.func(args)
  File "/home/geoffrey_brawn/anaconda3/envs/asim/lib/python3.9/site-packages/activitysim/cli/run.py", line 251, in run
    mp_tasks.run_multiprocess(injectables)
  File "/home/geoffrey_brawn/anaconda3/envs/asim/lib/python3.9/site-packages/activitysim/core/mp_tasks.py", line 1364, in run_multiprocess
    completed = run_sub_simulations(injectables,
  File "/home/geoffrey_brawn/anaconda3/envs/asim/lib/python3.9/site-packages/activitysim/core/mp_tasks.py", line 1171, in run_sub_simulations
    check_proc_status()
  File "/home/geoffrey_brawn/anaconda3/envs/asim/lib/python3.9/site-packages/activitysim/core/mp_tasks.py", line 1085, in check_proc_status
    raise RuntimeError("Process %s failed" % (p.name,))
RuntimeError: Process mp_accessibility_1 failed

[gbrawn]

I run the simulation as per the example_manifest.yaml activitysim run -c configs -o output -d data -s settings_mp.yaml

[jpn--]

How much RAM is on the machine where you are trying to run this model? The PSRC Full model is quite large

[gbrawn]

64GB, I thought it would be plenty. I looked for some hardware requirements but I wasn't able to find any?

[jpn--]

I don't think we've published hardware requirements for individual example models; they vary substantially across implementations. But I believe 64GB is nowhere near enough for the PSRC full model, at least using the currently published version 1.0.4. Upcoming releases will permit a smaller RAM footprint, but for now you need at least enough RAM to load the entire set of skims into memory, or you'll need to use the experimental / undocumented memory mapped skims, by setting this flag in your network_los.yaml file:

skim_dict_factory: MemMapSkimFactory

[gbrawn]

What would you suggest as the minimum RAM requirement?

[stefancoe]

@gbrawn We are able to run it on a machine with 128 GB of RAM, which requires an initial run using chunk training.

Also, just to be clear, the PSRC example was put together to test the mechanics of ActivitySim as a two zone implementation using PSRC data from our current ABM platform, Soundcast, which uses Daysim. We have not updated these inputs and many of them are not correct or basically made up, such as our Park and Ride skims. This example should not be used for anything other than as a reference to a large, regional mechanically working two-zone model. Please contact me directly if you would like to discuss further- scoe@psrc.org. Thanks!