IllegalStateException: AAAAAH! during SMAC localSearch

[stephanheinemann]

I am using SMAC as planning tuner in a Smart Autoflight Control System PhD project and just observed the following exception after otherwise successful tuning runs:

java.lang.IllegalStateException: AAAAAAH! at ca.ubc.cs.beta.smac.configurator.SequentialModelBasedAlgorithmConfiguration.localSearch (SequentialModelBasedAlgorithmConfiguration.java:658) at ca.ubc.cs.beta.smac.configurator.SequentialModelBasedAlgorithmConfiguration.selectChallengersWithEI (SequentialModelBasedAlgorithmConfiguration.java:427) at ca.ubc.cs.beta.smac.configurator.SequentialModelBasedAlgorithmConfiguration.selectConfigurations (SequentialModelBasedAlgorithmConfiguration.java:277) at ca.ubc.cs.beta.smac.configurator.AbstractAlgorithmFramework.run (AbstractAlgorithmFramework.java:573) at com.cfar.swim.worldwind.managers.smac.SmacAutonomicManager.run (SmacAutonomicManager.java:659) at com.cfar.swim.worldwind.managers.AbstractAutonomicManager.manage (AbstractAutonomicManager.java:1083) at com.cfar.swim.worldwind.ui.world.WorldPresenter$23.run (WorldPresenter.java:1242) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628) at java.lang.Thread.run (Thread.java:834)

Any help with this would be appreciated. Thanks a lot for developing SMAC!

[stephanheinemann]

Apparently this problem is related to the reported quality of my solutions. The quality is minimized by SMAC which led me to report 1d / quality in the ExistingAlgorithmRunResult and if there is no solution, then the quality is 0 and the solution is RunStatus.UNSAT. If I use 1000d - quality instead, I do not see this exception. However, reporting any of Double.POSITIVE_INFINITY or Double.MAX_VALUE leads reproducibly to this exception.

[stephanheinemann]

Thanks for looking at the code. I am aware that software has been developed several years ago but I am very happy to have a Java implementation available. The workaround (using a fix number to subtract from, which cannot be exceeded by the quality) apparently works fine for me. Maybe the issue is related to dividing by quality values smaller than 1 which creates a significant quality spread between neighboring configurations. I could imagine that a large spread could be an issue for local search? Just an assumption really... If you don't mind, I would have a look at the problem in the future and possibly try to find a solution. Would you be ok with me releasing the current version to Maven central for better dependency management in my project (including the aeatk and fastrf artifacts)? I would be releasing under my root com.uvic-cfar.swim (https://search.maven.org/search?q=com.uvic-cfar.swim). Thanks for your help!

[stephanheinemann]

I would also be releasing your modified com.beust.jcommander artifact with the added @ParameterFile annotation. This has unfortunately never been merged into jcommander (https://github.com/cbeust/jcommander/pull/123).

[SJrX]

I spent about 15 minutes looking at the code and it's not clear exactly when this would happen, but it did happen before or was done in reason. If you have a work around I would recommend using it. It's been a while since I (or to the best of my knowledge anyone has been using it)

I think there is also a python implementation that has kind of replaced it: https://github.com/automl/SMAC3

[SJrX]

Maybe the issue is related to dividing by quality values smaller than 1 which creates a significant quality spread between neighboring configurations.

It's some kind of logic error. I still have the data from the ticketing system available in a db backup somewhere but I dunno if I want to dig into it. If you wanted to set a breakpoint on the exception you might be able to get an idea of what is happening. It might be that one of the values is just some odd value in float arithmetic and a comparison doesn't work or something.

Would you be ok with me releasing the current version to Maven central for better dependency management in my project (including the aeatk and fastrf artifacts)? I would be releasing under my root com.uvic-cfar.swim (https://search.maven.org/search?q=com.uvic-cfar.swim). Thanks for your help!

I personally have no objection, although I don't know if I have the moral authority to give it my blessing, especially fastrf which I basically had very little to due with. I suspect no one else really cares though. From a legal standpoint all three are presumably under AGPLv3, for v2.10.03 only, so I think you certainly think legally you can do it and I won't object and I think everyone else has forgotten about it :) Previous versions are only free for academic and non-commercial usage. I think you need to make sure that the licensing is preserved.

would also be releasing your modified com.beust.jcommander artifact with the added @ParameterFile annotation. This has unfortunately never been merged into jcommander

Yup go nuts, and good luck, and you can try and ping me if you are stuck with anything else.

[SJrX]

If you do want to dig into it, instead of setting a break point, you could just log a few values here: https://github.com/SJrX/SMAC/blob/master/src/ca/ubc/cs/beta/smac/configurator/SequentialModelBasedAlgorithmConfiguration.java#L658 and make the exception something useful.

I'd log currentMinEI, epsilon, and min. and maybe Arrays.toString(eiVal)

[frank-hutter]

Hi,

we actually do have a newer version than the latest release, which we’ve still been using internally. @Marius, could you please send Steven the info for the latest version we have run experiments with in AClib, so that we leave the Java version in a consistent, final state?

Best, Frank

On Fri 8. Oct 2021 at 19:14, Steve Ramage @.***> wrote:

If you do want to dig into it, instead of setting a break point, you could just log a few values here: https://github.com/SJrX/SMAC/blob/master/src/ca/ubc/cs/beta/smac/configurator/SequentialModelBasedAlgorithmConfiguration.java#L658 and make the exception something useful.

I'd log currentMinEI, epsilon, and min. and maybe Arrays.toString(eiVal)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/SJrX/SMAC/issues/2#issuecomment-938893651, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLOK7Z46LYP24CHMHJSF6TUF4RI7ANCNFSM5FSY6GSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[stephanheinemann]

Thanks Frank, any more up-to-date version would be greatly appreciated. It was a lucky coincidence that I discovered your paper about SMAC. SMAC in fact turns out to be a very promising option for a tuner in an autonomic manager of a smart flight guidance system that tunes and selects planners according to the extracted autonomic context (planning context) features (environment, weather, traffic, airspaces etc.). Since SMAC supports tuning based on features, it just fit in perfectly.