generate function not able to assign Terminals when no primitive match

[sabban]

I may doing something wrong so feel free to point me in the right direction. Here's the problem I ran into:

I use strongly typed PrimitiveSet. I built a primitive with a custom-built input type and number of Terminal providing this custom-built type. I did this on purpose to fulfill a constraint of this Primitive filled with one of these Terminals. It may be a weird scenario, but when I look the code of the generate function in gp.py, I noticed that there's no fallback on Terminals when no Primitive is able to fullfil the type constraint. I had to do this because I want to translate in gp already existing tree/individuals that have a lot constraint in their construction.

The gp.generate function tried to add a primitive of type '<class 'custom-built type'>', but there is none available.

I see two possible workarounds to do this:

1. I can have my own generate function, or have the generate function in gp.py updated to fallback on Terminal when no Primitives can comply.
2. I may not using Terminals but Primitive with 0 arity.

Have you some thoughts on this ?

[rmlopes]

I was going to open a new issue, but I think my issue is close enough to this one. I am doing coevolution, and would like to use the result of one of the individuals as Terminal in the other. Assuming that the result of the first individual is stored in a variable called myvar, I would like to use it in the second individual. For that I tried adding a Terminal as a 0-arity function, since the documentation says:

class deap.gp.Terminal(terminal, symbolic, ret)¶ Class that encapsulates terminal primitive in expression. Terminals can be values or 0-arity functions.

With that in mind I try to add the terminal it as such:

mypset.addTerminal(lambda: getattr(__main__, 'myvar'), name='myterm')

When running I always get a type error or something similar because there is a function in there. If I use the Terminal object, I will get invalid syntax, with the Terminal object representation in the individual. A Primitive with 0 arity is not possible without refactoring DEAP as well.

The terminal approach should be possible, but perhaps I am missing something or there is actually a bug. Any more ideas?

[chathika]

I seem to be having the same issue and had to resort to @sabban 's workaround 2.

This involves adding both a primitive of arity 0 and terminal for each terminal whose type is used by another primitive, even though this terminal doesn't require a primitive in the first place.

Any idea if there has been any progress on this issue, since?

[adnanmuttaleb]

I Am also having the same issue, I do have a terminals that match the required type but an error always occur

[doodledood]

I bypassed the issue by rewriting the gp.generate to support this:

def generate_safe(pset, min_, max_, terminal_types, type_=None):
    if type_ is None:
        type_ = pset.ret
    expr = []
    height = random.randint(min_, max_)
    stack = [(0, type_)]
    while len(stack) != 0:
        depth, type_ = stack.pop()

        if type_ in terminal_types:
            try:
                term = random.choice(pset.terminals[type_])
            except IndexError:
                _, _, traceback = sys.exc_info()
                raise IndexError("The gp.generate function tried to add "
                                 "a terminal of type '%s', but there is "
                                 "none available." % (type_,)).with_traceback(traceback)
            if inspect.isclass(term):
                term = term()
            expr.append(term)
        else:
            try:
                # Might not be respected if there is a type without terminal args
                if height <= depth or (depth >= min_ and random.random() < pset.terminalRatio):
                    primitives_with_only_terminal_args = [p for p in pset.primitives[type_] if
                                                          all([arg in terminal_types for arg in p.args])]

                    if len(primitives_with_only_terminal_args) == 0:
                        prim = random.choice(pset.primitives[type_])
                    else:
                        prim = random.choice(primitives_with_only_terminal_args)
                else:
                    prim = random.choice(pset.primitives[type_])
            except IndexError:
                _, _, traceback = sys.exc_info()
                raise IndexError("The gp.generate function tried to add "
                                 "a primitive of type '%s', but there is "
                                 "none available." % (type_,)).with_traceback(traceback)
            expr.append(prim)
            for arg in reversed(prim.args):
                stack.append((depth + 1, arg))
    return expr

Basically, when there are no terminals, it tries to put a primitive with 0 or more terminal only args instead of just a real terminal, and if that is not available, it defaults to putting a regular random primitive. Not sure if it's the correct way to go about things but it works for me and my use case.

Note that using this makes height a recommendation instead of a hard limit, as there could be a case that the tree continues to expand beyond height (that's why I try to match primitives with only terminal args - to cut this possibly infinite branch generation, just like a terminal would).

To use this just pass a terminal_types argument to the generate_safe function like this:

terminal_types = [SomeCustomTerminal, int, float]

toolbox.register("expr", generate_safe, pset=pset, min_=1, max_=10, terminal_types=terminal_types)

[aodhan-domhnaill]

@doodledood Your code fixed my issue. My problem was that I needed to restrict the terminal types to a sublist of all the types.

Can you submit your code as a PR? I'm happy to do that, but I thought you should get credit for it.

[ir0nt0ad]

Facing the same problem, a PR would be great if the code by @doodledood is still relevant