Maximum number of function evaluations has been exceeded

[embaldridge]

Something is up with the CBC data that is breaking things, I am guessing having to do with large abundances, and it's somewhere in model comparisons. I get an AICc value for the logseries, then I get 'nan' for everything else (logseries untruncated, Poisson lognormal, negative binomial, generalized Yule).

cbc, Site "L105249", S=57, N=7400 Warning: Maximum number of function evaluations has been exceeded. [['"L105249"' '57' '7400' '0.9988657' 'nan' 'nan' 'nan' 'nan' 'nan']] cbc, Site "L105252", S=57, N=3730 Warning: Maximum number of function evaluations has been exceeded. [['"L105252"' '57' '3730' '0.9974472' 'nan' 'nan' 'nan' 'nan' 'nan']]

I'm guessing the problem is wherever that warning message is coming from.

[embaldridge]

BBS and NABA data work, Gentry data gives an output, but it's not right, CBC gives the previous warning message and an output that is nan's, and the FIA and MCDB start on the first site and then lock up.

[embaldridge]

For the Gentry, something goes wrong between calculating the AICc weights, which seems to be going right, and the output, where something goes terribly wrong and I get zeros.

[embaldridge]

Gentry is fixed, something about the way the output bit was working for NABA and BBS, but messing up on Gentry. I changed the bit where the dist_test results are formatted for output, and now it works for all three of those.

[embaldridge]

I added a bunch of print statements, and there seems to be something up with the negative binomial bit and the generalized Yule bit, which is then throwing off the AICc calculations. I suspect that this has something to do with a large N. This is just running the CBC test data.

Dataset imported sucessfully. cbc, Site "L105249", S=57, N=7400 METE predicted Truncated logseries likelihood predicted: -315.187360849 Untruncated logseries likelihood predicted: -315.18736061 Poisson lognormal likelihood predicted: -315.811735304 Warning: Maximum number of function evaluations has been exceeded. Negative binomial likelihood predicted: -inf Generalized Yule likelihood predicted: nan AICc truncated logseries: 634.596943919 AICc untruncated logseries calculated: 632.447448492 AICc Poisson lognormal calculated: 635.84569283 AICc negative binomial calculated: inf AICc generalized Yule calculated: nan AICc values collected AICc weights calculated[nan, nan, nan, nan, nan] Predicted results ready for output. AICc results ready for output: [['"L105249"', 57, 7400, nan, nan, nan, nan, nan]] Results output cbc, Site "L105252", S=57, N=3730 METE predicted Truncated logseries likelihood predicted: -262.599617795 Untruncated logseries likelihood predicted: -262.599617768 Poisson lognormal likelihood predicted: -264.71018752 Warning: Maximum number of function evaluations has been exceeded. Negative binomial likelihood predicted: -inf Generalized Yule likelihood predicted: nan AICc truncated logseries: 529.421457812 AICc untruncated logseries calculated: 527.271962808 AICc Poisson lognormal calculated: 533.642597262 AICc negative binomial calculated: inf AICc generalized Yule calculated: nan AICc values collected AICc weights calculated[nan, nan, nan, nan, nan] Predicted results ready for output. AICc results ready for output: [['"L105252"', 57, 3730, nan, nan, nan, nan, nan]] Results output Dataset analysis complete.

[embaldridge]

FIA and MCDB data get through predicting the likelihood for the negative binomial for the first site (where S > 9), then hang on the generalized Yule.

[embaldridge]

It's something about the generalized Yule solver for MCDB and FIA, and I think it's around line 516 in macroeco_distributions.py.

Observed abundance = 79, 78, 59, 52, 50, 47, 22, 20, 18, 12, 9, 6, 5, 4, 2 (for MCDB, doesn't work) Observed abundance = 12, 12, 8, 5, 4, 3, 2, 1, 1, 1 (for FIA, doesn't work)

Observed abundance = 150, 123, 93, 82, 69, 45, 38, 34, 33, 25, 19, 18, 14, 13, 10, 8, 6, 5, 5, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1 (for BBS, works)

CBC gets parameters a, b, but nan for the likelihood estimate.

[embaldridge]

Something seems to be up with the likelihood estimator for CBC data and the negative binomial, line 391 in macroeco-distributions.py, returns -inf.

[embaldridge]

@ethanwhite: What do we want to do if the generalized Yule distribution fails to converge for a site? I see two possibilities: either skip that distribution, but calculate everything for the rest of the distributions, or skip the site all together and move on to the next one.

[ethanwhite]

If it's failing to converge it's probably a bad fit to the data. One option is to limit the number of iterations and simply use the likelihood of the last iteration. It will be bad, but if it's not converging it's not a good fit anyway. Then keep track of how many sites failed to converge, make sure it's a small fraction, and note that so that folks are aware of it.

[embaldridge]

Xiao tweaked it to limit the number of iterations, print a warning that it failed to converge, and return none for a and b. In my code, if I get a failure to converge, it skips the generalized Yule and leaves the weight blank, so it's only calculating with 5 distributions instead of all 6.

[ethanwhite]

Sounds good to me.