Cannot constrain on objective metric

[ga92xug]

I am working on a NAS with Ax. I have 2 objectives valid_acc and flops. I want to constrain the flops to be smaller than a max flop value. outcome_constraints=[f"flops <= {self.cfg.objective.max_flops}"] Now Ax tells me that: "Cannot constrain on objective metric". There is an obvious easy work around for that define another metric flops_outcome_constraint and use that in the outcome_constraint. I just don't know how this impacts the performance of the model.

To put this in a larger perspective. Some of the architectures that are proposed are too large/expensive to be evaluated. I want to mark these as infeasible. It is difficult to know before which combinations will result in expensive architectures. For some the flop count can be calculated exactly and if it is above a certain threshold we want to discard these. However, for some architectures, even the building process might be so slow that we have to abandon it. Is there a better way to mark these trials as infeasible?

[Balandat]

So you are doing multi-objective optimization here, i.e. explore the tradeoffs between flops and valid_acc? In that case what you'll want to do is to use an "objective threshold" that restricts the outcome space to a subset that is of interest to you. See the use of ObjectiveThreshold in the "deep dive" section of this tutorial: https://ax.dev/tutorials/multiobjective_optimization.html

[lena-kashtelyan]

@ga92xug it seems to me that your question was answered, so I'm closing this issue. Please feel free to follow up on it, but if you do, please reopen the issue as we might not see your comment on a closed one.