search

PSLmodels / OG-Core

An overlapping generations model framework for evaluating fiscal policies.
https://pslmodels.github.io/OG-Core/
Creative Commons Zero v1.0 Universal
66 stars 114 forks source link

Update threads_per_worker in example run script #926

Closed rickecon closed 4 months ago

rickecon commented 6 months ago

@talumbau and @jdebacker. I think we need to make the same change in the OG-Core example run script that TJ's PR #102 made in OG-USA. We should limit the number of threads_per_worker to 1 in the dask Client class instantiation.

client = Client(n_workers=num_workers, threads_per_worker=1)
talumbau commented 6 months ago

Thanks for opening this issue. I agree it looks like this is the right call, but I was going to ask you both to try a run with this setting on your Macs to make sure there is no regression on your side. I don't have a good sense of the default features of the Dask LocalCluster and default scheduling choices. Also I noticed that this setting exists on something called cs_config in OG-USA here:

https://github.com/PSLmodels/OG-USA/blob/master/cs-config/cs_config/functions.py#L214

Can you provide some details on what cs-config is? Second point is that we should create a central "client creation" function provided by OG-USA utilities so that we can centralize these features (we might need additional Client customization as time goes on)

jdebacker commented 6 months ago

@talumbau asks:

Can you provide some details on what cs-config is?

the cs-config files are those used for a Compute-Studio (C/S) web application of the OG-USA model. This app has generally been run on Google Cloud Services. @kcreekdev has been doin some work recently in OG-USA related to some memory issues he's encountered on the GCS machines.

Second point is that we should create a central "client creation" function provided by OG-USA utilities so that we can centralize these features (we might need additional Client customization as time goes on)

Ok - let me know what these means. We thought it was pretty useful that the client passes through the OG-Core model and so one can set that up in the "run script" for whatever their configuration is (e.g., running on a local cluster, a cluster with Slurm scheduler, etc.). But anything that would aid users parameterizing the client correctly for their environment would be good.

rickecon commented 5 months ago

The updating of the run script threads-per-worker part of this issue was resolved in PR #928. However, it looks like we still need to resolve how Dask clusters are created in the Compute Studio files.

jdebacker commented 4 months ago

There should be no CS file in this repo.