Permanent income levels are not randomized

[sbenthall]

$ python run_any_simulation.py --dphm 148 --quarters 12 --pop_CRRA 6 --pop_DiscFac 0.9 --pop_aNrmInitMean 0.02677 --seed 123 ../out/20240205b

Gets me this plot of the mean permanent income levels:

Changing the seed:

$ python run_any_simulation.py --dphm 148 --quarters 12 --pop_CRRA 6 --pop_DiscFac 0.9 --pop_aNrmInitMean 0.02677 --seed 299 ../out/20240205b

Gets me this:

Which is the same! The permanent income levels are stochastic, but they seem to be using the same random seed even though the SHARKFin random seed is different.

The mean log income is listed as '5' because it's factoring in the DPHM, which is 148 = e^5

[sbenthall]

@alanlujan91 I'm not sure what's going on with the population seed.

It seems to be trying to something here:

https://github.com/sbenthall/SHARKFin/blob/master/simulate/run_any_simulation.py#L187-L189 https://github.com/sbenthall/SHARKFin/blob/master/sharkfin/population.py#L114-L121

[alanlujan91]

hmm maybe changing the seed is not resetting the RNG?

[alanlujan91]

which notebook has this output?

[sbenthall]

Visualize Sim, when the output from the run_any_simulation.py is opened in it.

https://github.com/sbenthall/SHARKFin/blob/master/simulate/Visualize%20Sim.ipynb

The missing part of the workflow to reproduce is to load the data into that notebook after each execution.

[alanlujan91]

not really sure how to debug this one.. I tried resetting the seed but that wasn't it

[sbenthall]

Ok, I've written a new automated test which exposes the (increasingly weird) issue: https://github.com/sbenthall/SHARKFin/blob/master/sharkfin/tests/test_population_seed_issue.py

This test current fails at line 62: https://github.com/sbenthall/SHARKFin/blob/master/sharkfin/tests/test_population_seed_issue.py#L62

What this test is doing is creating two different SharkfinPopulations with different random seeds, and simulating forward, then comparing the pLvls (a) between agents within the same population, and (b) between agents across the two populations (but with the same index).

The findings are quite strange to me.

With a long T_sim, there are differences between the pLvl processes between agents and populations. [Side note: couldn't figure out how to get it to simulate fewer than 1000 periods -- setting T_sim in the parameters dict didn't seem to work].

BUT for all the agents, whichever population, they have identical permanent income processes for the first 28 simulated periods.

I have no idea why this is the case.

But recall that we've been running experiments with rather few 'quarters' or periods (4). So if we are getting identical processes for all agents, then that is effectively entirely correlated incomes across and within each simulation.

[sbenthall]

I've been able to reduce this to a bug in HARK: https://github.com/econ-ark/HARK/issues/1379

[sbenthall]

Fixing this in HARK probably helped https://github.com/econ-ark/HARK/pull/1380

BUT the tests are still failing. For some reason the SharkfinPopulation agents all have the same seeds for their income distributions. Likely this means that they are not getting new random seeds passed to the AgentTypes.

[sbenthall]

Better test isolating the issue: https://github.com/sbenthall/SHARKFin/blob/master/sharkfin/tests/test_population_seed_issue.py

[sbenthall]

explode_agents() is creating AgentType objects with different random seeds, but with identically seeded income distributions.

https://github.com/sbenthall/SHARKFin/blob/master/sharkfin/population.py#L110-L126

Looks like the income distributions are being copied over with 'deepcopy'. Initializing new agents might have been a better choice here. I'll see what sort of fix I can get in.

[sbenthall]

This is now fixed! We are now getting uncorrelated income shocks, so the mean log income is very close to the log DPHM.

It took some extra lines in SharkPopulation https://github.com/sbenthall/SHARKFin/commit/1d5d961e8597a76136a2f290e227e3809ed53d10

[sbenthall]

No, stlll not working right... --- random seeds are not making it through to the SharkPopulation yet. The shocks are uncorrelated, but the same across runs.

[sbenthall]

Ok, fixed this at last I think.