aletzdy
commented
1 month ago Run with full sample, sharrow on, and single process (1 TB memory, Intel Xeon Gold 6342 @ 2.8GHz machine).
Runtime is 1428.2 mins (compared to 1,433.1 mins for a similar run with NO chunking) Chunking is set to explicit in settings_mp.yaml, and explicit chunking is enabled for a number of models whose memory usage was higher than 300Gb in a previous sharrow run with same specs but with NO chunking. These models are:
compute_accessbility mandatory_tour_scheduling non-mandatory_tour_scheduling trip_destination
For these models, the argument explicit_chunk was set to 0.5 in each respective model's yaml file, indicating the need for 2 (=1/0.5) chunks. Further chunks may be added in other tests and as required.
timing_log.csv activitysim.log memory_profile.csv
Below is the run without chunking:
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">
event | full_rss_gb_chunking | full_rss_gb_No_chunking | Difference | % Diff accounting for skims -- | -- | -- | -- | -- preload_injectables | 2.167452 | 2.527322 | -0.35987 | NA initialize_proto_population | 173.9227 | 174.1077 | -0.18509 | NA compute_disaggregate_accessibility | 206.3356 | 201.5561 | 4.779454 | 17.4% initialize_landuse | 180.1346 | 175.3758 | 4.75886 | 375.3% initialize_households | 198.2141 | 193.3762 | 4.837839 | 25.1% compute_accessibility | 276.3907 | 367.4861 | -91.0954 | -47.1% av_ownership | 184.3127 | 180.3675 | 3.945181 | 63.0% auto_ownership_simulate | 202.3356 | 197.699 | 4.636545 | 19.7% work_from_home | 193.0878 | 192.2186 | 0.869179 | 4.8% external_worker_identification | 194.0765 | 192.6287 | 1.447735 | 7.8% external_workplace_location | 194.758 | 192.3864 | 2.371576 | 13.0% school_location | 292.9196 | 289.0295 | 3.890065 | 3.4% workplace_location | 402.6434 | 397.1514 | 5.49206 | 2.5% transit_pass_subsidy | 194.9363 | 192.3512 | 2.585076 | 14.2% transit_pass_ownership | 194.0668 | 193.3555 | 0.711307 | 3.7% vehicle_type_choice | 270.9691 | 265.4805 | 5.488607 | 6.0% adjust_auto_operating_cost | 183.9534 | 178.5603 | 5.393015 | 121.1% transponder_ownership | 188.1608 | 182.8532 | 5.30756 | 60.7% free_parking | 194.9765 | 193.1514 | 1.825108 | 9.6% telecommute_frequency | 195.4649 | 192.9583 | 2.506637 | 13.3% cdap_simulate | 195.6484 | 193.3964 | 2.252005 | 11.7% mandatory_tour_frequency | 195.0767 | 194.2196 | 0.857059 | 4.3% mandatory_tour_scheduling | 330.5758 | 453.69 | -123.114 | -44.0% school_escorting | 214.1988 | 208.6679 | 5.530911 | 16.0% joint_tour_frequency_composition | 205.0339 | 201.3234 | 3.710493 | 13.6% external_joint_tour_identification | 201.6065 | 201.7927 | -0.18614 | -0.7% joint_tour_participation | 199.2767 | 194.9526 | 4.324151 | 20.7% joint_tour_destination | 207.107 | 202.9316 | 4.175458 | 14.5% external_joint_tour_destination | 196.1608 | 191.8689 | 4.291928 | 24.2% joint_tour_scheduling | 212.2434 | 212.0547 | 0.188678 | 0.5% non_mandatory_tour_frequency | 238.0545 | 233.7386 | 4.315931 | 7.2% external_non_mandatory_identification | 208.9876 | 207.5797 | 1.407865 | 4.2% non_mandatory_tour_destination | 383.1723 | 379.6374 | 3.534909 | 1.7% external_non_mandatory_destination | 197.0708 | 192.72 | 4.350751 | 23.4% non_mandatory_tour_scheduling | 227.4625 | 275.858 | -48.3955 | -47.6% vehicle_allocation | 218.2018 | 217.5814 | 0.620433 | 1.4% tour_mode_choice_simulate | 217.6302 | 212.2314 | 5.398741 | 14.2% atwork_subtour_frequency | 196.0487 | 190.7215 | 5.327159 | 32.1% atwork_subtour_destination | 295.5799 | 290.2229 | 5.357048 | 4.6% atwork_subtour_scheduling | 211.7981 | 206.3961 | 5.401965 | 16.7% atwork_subtour_mode_choice | 198.8897 | 193.8696 | 5.020123 | 25.4% stop_frequency | 222.6066 | 217.8433 | 4.7633 | 10.9% trip_purpose | 199.3885 | 195.7768 | 3.611701 | 16.7% trip_destination | 426.6972 | 536.408 | -109.711 | -30.3% trip_purpose_and_destination | 206.8114 | 201.6604 | 5.150953 | 18.7% trip_scheduling | 208.1987 | 204.3938 | 3.804885 | 12.6% trip_mode_choice | 224.9043 | 219.6366 | 5.267706 | 11.6% parking_location | 292.0136 | 303.9549 | -11.9413 | -9.2% write_data_dictionary | 210.6467 | 207.9639 | 2.682831 | 7.9% track_skim_usage | 204.6225 | 202.1674 | 2.455048 | 8.7% write_trip_matrices | 298.8238 | 296.5611 | 2.262761 | 1.8% write_tables | 257.3621 | 255.2239 | 2.138206 | 2.6% finalizing | 230.174 | 228.311 | 1.862975 | 3.4%
This is the issue to report on memory usage and runtime performance...
data_dir: "data-full" full scale skims (24333 MAZs) households_sample_size: 0 (full scale 100% sample of households) sharrow: "false" multiprocess: false single process chunk_training_mode: explicit