ForbiddenAndConjunction-related issue

[ixeixe]

Hello ConfigSpace package developers. I'm having a problem with ForbiddenInClause and ForbiddenAndConjunction. Here is an example:

import numpy as np
from openbox import space as sp
from ConfigSpace import ConfigurationSpace, Categorical, Float, Integer,Constant
from ConfigSpace import ForbiddenAndConjunction, ForbiddenEqualsClause, ForbiddenInClause

space = sp.Space()

lp2=Categorical('lp2', [1, 2, 8, 64])
A=Categorical('A', [1, 2, 8])
x=Categorical('x', [1, 2, 8])
tmp=Categorical('tmp', [1, 2, 8])
#I hope that
#When lp2=1, A and tmp cannot be in [1,2], and x cannot be in [1,2]
#When lp2=2, A and tmp cannot be in [1,8], and x cannot be in [1,8]
#When lp2=8, A and tmp cannot be in [1,2], and x cannot be in [1,2]
#When lp2=64, A and tmp cannot be in [1,2], and x cannot be in [1,2]
F1=ForbiddenAndConjunction(
    ForbiddenInClause(lp2,[1]),
    ForbiddenInClause(A,[2,8]),
    ForbiddenInClause(tmp,[2,8]),
    ForbiddenInClause(x,[2,8]))
F2=ForbiddenAndConjunction(
    ForbiddenInClause(lp2,[2]),
    ForbiddenInClause(A,[1,8]),
    ForbiddenInClause(tmp,[1,8]),
    ForbiddenInClause(x,[1,8]))
F3=ForbiddenAndConjunction(
    ForbiddenInClause(lp2,[8,64]),
    ForbiddenInClause(A,[1,2]),
    ForbiddenInClause(tmp,[1,2]),
    ForbiddenInClause(x,[1,2]))

F=[F1,F2,F3]
names = [lp2,A ,x ,tmp]
space.add_variables(names)
space.add(F)
space.sample_configuration(1)

The output is:

Configuration space object:
  Hyperparameters:
    A, Type: Categorical, Choices: {1, 2, 8}, Default: 1
    lp2, Type: Categorical, Choices: {1, 2, 8, 64}, Default: 1
    tmp, Type: Categorical, Choices: {1, 2, 8}, Default: 1
    x, Type: Categorical, Choices: {1, 2, 8}, Default: 1
  Forbidden Clauses:
    (Forbidden: lp2 in {1} && Forbidden: A in {2, 8} && Forbidden: tmp in {2, 8} && Forbidden: x in {2, 8})
    (Forbidden: lp2 in {2} && Forbidden: A in {1, 8} && Forbidden: tmp in {1, 8} && Forbidden: x in {1, 8})
    (Forbidden: lp2 in {8, 64} && Forbidden: A in {1, 2} && Forbidden: tmp in {1, 2} && Forbidden: x in {1, 2})

The result of sample space.sample_configuration (1) is:

 Configuration(values={
   'A': 2,
   'lp2': 2,
   'tmp': 8,
   'x': 8,
 })

But this does not conform to the forbidden clause F2.

F2=ForbiddenAndConjunction(
    ForbiddenInClause(lp2,[2]),
    ForbiddenInClause(A,[1,8]),
    ForbiddenInClause(tmp,[1,8]),
    ForbiddenInClause(x,[1,8]))

When lp2= 2, x should not be equal to 8, and y should not be equal to 8. According to the Boolean calculation of your source code, the sampling configuration must trigger 4 clauses of F2 at the same time to achieve the effect I want, if only 3 clauses of F2 and less are triggered at the same time, it will also be successfully sampled, and the effect I want will not be achieved. Please how can I modify it to achieve 【When lp2=2, A and tmp cannot be in [1,8], and x cannot be in [1,8]】？ I look forward to your answers and thank you for your help!

[eddiebergman]

Hi @ixeixe,

This is likely an oversight on my part. We did some optimization work to ensure that we minimize the amount of forbiddens that need to be checked and my guess is that this is what causes the behaviour. I will try to look at it in the coming week but unfortunatly I am at an event and may not get time until the following week.

For reference, this is what is actually used during forbidden checks:

• space._dag.fast_forbidden_checks: https://github.com/automl/ConfigSpace/blob/d5b82ae3767f389d340e7a1b2d24a04ea3669f0f/src/ConfigSpace/_condition_tree.py#L361-L393

You could print these out to see where the inconsisteny lies which would help in finding out the logical error I made.

For reference, this is the logic that takes a list of forbiddens and returns an optimized list of forbiddens.

https://github.com/automl/ConfigSpace/blob/d5b82ae3767f389d340e7a1b2d24a04ea3669f0f/src/ConfigSpace/_condition_tree.py#L821-L876

[ixeixe]

Thank you for taking the time out of your busy schedule to get back to me, you mentioned that I can print space._dag.fast_forbidden_checks, the result of which is:

[(Forbidden: lp2 in {1} && Forbidden: A in {2, 8} && Forbidden: tmp in {2, 8} && Forbidden: x in {2, 8}),
 (Forbidden: lp2 in {2} && Forbidden: A in {1, 8} && Forbidden: tmp in {1, 8} && Forbidden: x in {1, 8}),
 (Forbidden: lp2 in {8, 64} && Forbidden: A in {1, 2} && Forbidden: tmp in {1, 2} && Forbidden: x in {1, 2})]

In line 2, the Forbidden I want to be effective is： prohibit [lp2=2 and (A=1 or A=8) and (tmp=1 or tmp=8) and (x=1 or x=8)] The negation is denoted as [lp2≠2 or (A≠1 and A≠8) or (tmp≠1 and tmp≠8) or (x≠1 and x≠8)] I hope that only [lp2=2 and A=2 and tmp=2 and x=2] can appear. I don’t understand the logic of your source code for the time being, I’m sorry, but I still looking forward to your reply at your convenience, thank you!

[eddiebergman]

No worries, the code is hard to make understandable and it's really just an efficiency thing done under the hood. The extra info you gave will be super helpful! I will take a look at this next week once I am back! Apologies for the inconvenience