Compute Studio Error

[hdoupe]

• link: https://compute.studio/PSLmodels/Tax-Brain/42731/
• inputs:

{
    "meta_parameters": {
        "year": 2017,
        "data_source": "CPS",
        "use_full_sample": true
    },
    "adjustment": {
        "behavior": {
            "cg": [
                {
                    "value": 0
                }
            ],
            "inc": [
                {
                    "value": 0
                }
            ],
            "sub": [
                {
                    "value": 0
                }
            ]
        },
        "policy": {
            "NIIT_rt": [
                {
                    "year": 2017,
                    "value": 0
                }
            ]
        }
    }
}

• traceback:

Traceback (most recent call last):
  File "/home/distributed/api/celery_app/__init__.py", line 75, in f
    outputs = func(*args, **kwargs)
  File "/home/distributed/api/celery_app/pslmodels_taxbrain_tasks.py", line 49, in sim
    return compconfig.run_model(meta_param_dict, adjustment)
  File "/opt/conda/lib/python3.7/site-packages/compconfig/functions.py", line 177, in run_model
    results = compute(*delayed_list)
  File "/opt/conda/lib/python3.7/site-packages/dask/base.py", line 446, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/dask/threaded.py", line 82, in get
    **kwargs
  File "/opt/conda/lib/python3.7/site-packages/dask/local.py", line 480, in get_async
    key, res_info, failed = queue_get(queue)
  File "/opt/conda/lib/python3.7/site-packages/dask/local.py", line 144, in queue_get
    return q.get()
  File "/opt/conda/lib/python3.7/queue.py", line 170, in get
    self.not_empty.wait()
  File "/opt/conda/lib/python3.7/threading.py", line 296, in wait
    waiter.acquire()
  File "/opt/conda/lib/python3.7/site-packages/billiard/pool.py", line 227, in soft_timeout_sighandler
    raise SoftTimeLimitExceeded()
billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()

[hdoupe]

I think this is a memory issue related to using the CPS and behavior parameters at the same time. behresp makes copies of the Calculator objects which makes the sims using behavior take longer and require significantly more memory.

[hdoupe]

I increased Tax-Brain's server size from 2 CPU and 13 GB RAM to 4 CPU and 26 GB RAM, and bumped the time limit from 600 seconds to 1200 seconds. Sims using the CPS and behavior parameters can be run successfully now.

[andersonfrailey]

Thanks for the report, @hdoupe. Seems like memory usage is going to be a recurring issue with behavior simulations with the CPS. I'm not sure what the long term solution to this is, but for now throwing more computing resources is probably our best bet. I'll see if I can think of something better for the long term

[hdoupe]

Agreed. The bright side of the beefier servers is that the regular sims using the PUF without behavior params are completeing in about 150-180 secs instead of 300. Here's a breakdown of the approximate sim times:

data source	static	behavioral
cps	290	450
puf	160	300

I don't know the full rationale for why behresp makes copies of the Calculator objects, but if we were able to toggle that off, we could save about 4-5 GB on behavior sims.