constrained optimization on score tests

[eidlide]

Hello again! We are still experiencing long run times (no resolution after 20 h) for score testing using parallel processing with 30 cores. Our model has a single, continuous variable, 282 taxa, and 138 samples.

When I run the same score test code serially, one time sink seems to be that we get the constrained optimization convergence failure flag upward of 10 times. Are there minimum and maximum values for tau and inner_maxit that you would suggest for initial values? Also, is there a way to find the final tau and inner_maxit for a completed emuFit object, so that you can rerun the code more quickly the next time?

Thank you!!!

[eidlide]

UPDATE: tried the job again b/c our HPC session ran out, and it completed in ~3.5 hours.

[ailurophilia]

Glad to hear! 🎉🥳

[adw96]

Great questions, @eidlide !

@ailurophilia What suggestions do you have for folks experiencing long runtimes? Is tinkering with tau and inner_maxit the best way to go? FYI @paulinetrinh has also been having this challenge on HDW data, I was going to ask you about this anyway.

@ailurophilia Are you also able to explain what's happening when the constrained optimization fails to converge?

@svteichman Would you be able to add an element to the return list that includes the final tau and inner_maxit? I think this is a great request. 🙏

[svteichman]

Will do!

[adw96]

@ailurophilia I'm going to close this since @eidlide resolved their own issue, but know that other people have contacted me about code hanging so likely I'll ping you elsewhere to get suggestions once we have some more examples of this behaviour 🥳