test framework/Optimizers.multitrajlevelWithModelHolding shows prematurely exit

alfoa commented 6 years ago

Issue Description

What did you expect to see happen?

I expected the multilevel_with_multitraj_ensemble.xml test to reach the convergence

What did you see instead?

The multilevel_with_multitraj_ensemble.xml (heavy test) prematurely exits without reaching the optimal point:

(    1.66 sec) SPSA                     : DEBUG           ->    Traj: "0": Process: "       following traj 1       ", Reason: "     removed as redundant     "
(    1.66 sec) SPSA                     : DEBUG           ->    Traj: "1": Process: "  collecting new opt points   ", Reason: "    seeking new opt point     "
(    1.66 sec) SPSA                     : DEBUG           ->    Traj: "2": Process: "  collecting new opt points   ", Reason: "    seeking new opt point     "
(    1.66 sec) SPSA                     : DEBUG           -> Checking readiness for traj "2" ...
(    1.66 sec) SPSA                     : DEBUG           ->     Traj "2": Waiting on collection of new optimization point.
(    1.66 sec) SPSA                     : DEBUG           -> Checking readiness for traj "1" ...
(    1.66 sec) SPSA                     : DEBUG           ->     Traj "1": Waiting on collection of new optimization point.
(    1.66 sec) SPSA                     : DEBUG           -> Not ready to provide a sample yet.
(    1.66 sec) SPSA                     : DEBUG           -> Reviewing status of trajectories:
(    1.66 sec) SPSA                     : DEBUG           ->    Traj: "0": Process: "       following traj 1       ", Reason: "     removed as redundant     "
(    1.66 sec) SPSA                     : DEBUG           ->    Traj: "1": Process: "  collecting new opt points   ", Reason: "    seeking new opt point     "
(    1.66 sec) SPSA                     : DEBUG           ->    Traj: "2": Process: "  collecting new opt points   ", Reason: "    seeking new opt point     "
(    1.66 sec) SPSA                     : DEBUG           -> Checking readiness for traj "2" ...
(    1.66 sec) SPSA                     : DEBUG           ->     Traj "2": Waiting on collection of new optimization point.
(    1.66 sec) SPSA                     : DEBUG           -> Checking readiness for traj "1" ...
(    1.66 sec) SPSA                     : DEBUG           ->     Traj "1": Waiting on collection of new optimization point.
(    1.66 sec) SPSA                     : DEBUG           -> Not ready to provide a sample yet.
(    1.66 sec) STEP MULTIRUN            : DEBUG           -> Finished with 78 runs submitted, 0 jobs running, and 0 completed jobs waiting to be processed.
(    1.66 sec) SPSA                     : DEBUG           -> ===============
(    1.66 sec) SPSA                     : DEBUG           -> | RUN SUMMARY |
(    1.66 sec) SPSA                     : DEBUG           -> ===============

Do you have a suggested fix for the development team?

Please attach the input file(s) that generate this error. The simpler the input, the faster we can find the issue.

For Change Control Board: Issue Review

This review should occur before any development is performed as a response to this issue.

[x] 1. Is it tagged with a type: defect or improvement?
[x] 2. Is it tagged with a priority: critical, normal or minor?
[ ] 3. If it will impact requirements or requirements tests, is it tagged with requirements?
[x] 4. If it is a defect, can it cause wrong results for users? If so an email needs to be sent to the users.
[x] 5. Is a rationale provided? (Such as explaining why the improvement is needed or why current code is wrong.)

For Change Control Board: Issue Closure

This review should occur when the issue is imminently going to be closed.

[ ] 1. If the issue is a defect, is the defect fixed?
[ ] 2. If the issue is a defect, is the defect tested for in the regression test system? (If not explain why not.)
[ ] 3. If the issue can impact users, has an email to the users group been written (the email should specify if the defect impacts stable or master)?
[ ] 4. If the issue is a defect, does it impact the latest stable branch? If yes, is there any issue tagged with stable (create if needed)?
[x] 5. If the issue is being closed without a merge request, has an explanation of why it is being closed been provided?

PaulTalbot-INL commented 6 years ago

I'm not very familiar with the mechanics tested here; it looks like @alfoa was the developer and test writer.

I do see that no new samples are ready but it is clearly not converged when I run it, as well. I'm seeing this in other problems, as I mentioned last week; the implicit readiness checking makes this slightly painful to debug. Hopefully I can look at it soon.

alfoa commented 4 years ago

Closing since new optimizer is the process to be rewritten

idaholab / raven