search

dfm / emcee

The Python ensemble sampling toolkit for affine-invariant MCMC
https://emcee.readthedocs.io
MIT License
1.47k stars 430 forks source link

emcee cannot handle more than 20 parameters? #378

Closed deltasata closed 1 year ago

deltasata commented 3 years ago

I have 24 free parameters, I set nwalkers=100 which should be sufficient. However, I am getting the following error

ValueError: Initial state has a large condition number. Make sure that your walkers are linearly independent for the best performance

which looks more like a warning. Whatever I do I cannot avoid this error. Interestingly, the simpler version of the same code with 18 free parameters works fine.

In the API not much has been written about this error.

Please help me troubleshoot this error.

dfm commented 3 years ago

This error suggests that your initial coordinates for the walkers are (numerically) not linearly independent. This will normally happen if you forget to jitter the initial positions or if that jitter is too small. How are you currently initializing? If you're confident that this is not a problem, then you can silence this error using skip_initial_state_check=True as described here.

deltasata commented 3 years ago

Thanks. Got it.

Jinyin-Hu commented 1 year ago

Hi Dan,

I also have the similar question. How many parameters the emcee can handle? is there any limitation for the number of parameters? I'm using the emcee to do a inversion to determine the Earthquake source parameters. Currently, I have 30 free parameters, 1024 walkers with 10,000 steps for each. The acceptance rate is about 19-20%. It's a little bit low rate, right. Many thanks.

dfm commented 1 year ago

@Jinyin-Hu — There is no formal limit on the number of dimensions, but the efficiency of the sampler will degrade with the number of parameters. I've certainly fit models with >30 parameters, but 10,000 steps probably isn't sufficient. It's not so useful to look at the acceptance rate (20% seems fine or high for such a model), but you'll want to look at the autocorrelation properties of the chains: https://emcee.readthedocs.io/en/stable/tutorials/autocorr/