Created test domain, but program cannot solve it.

[Angular-Angel]

I decided to try the thing on some fuzzy math, and I got the domain working - but it fails to solve even the simplest problem, requiring the use of only one primitive. Now, I'm running it with the same command as used in the incr.py script, so that may be the issue - but if so, can you tell me what I should look into changing? Also, is there documentation of what all the runtime arguments do? There are a bunch, and I haven't been able to find much explanation on them.

Test Script:

import os
import random

try:
    import binutil  # required to import from dreamcoder modules
except ModuleNotFoundError:
    import bin.binutil  # alt import if called as module

from dreamcoder.ec import commandlineArguments, ecIterator
from dreamcoder.grammar import Grammar
from dreamcoder.program import Primitive
from dreamcoder.task import Task
from dreamcoder.type import arrow, treal
from dreamcoder.utilities import numberOfCPUs

# Primitives
def _fand(x, y): return lambda x, y : x * y
def _for(x, y): return lambda x, y : x + y - x * y

def fAnd():
    x = random.choice(range(0, 100)) / 100
    y = random.choice(range(0, 100)) / 100
    return {"i": (x, y), "o": x * y}

def get_treal_task(item):
    return Task(
        item["name"],
        arrow(treal, treal, treal),
        [((ex["i"],), ex["o"]) for ex in item["examples"]],
    )

if __name__ == "__main__":

    args = commandlineArguments(
        enumerationTimeout=10, activation='tanh',
        iterations=10, recognitionTimeout=3600,
        a=3, maximumFrontier=10, topK=2, pseudoCounts=30.0,
        helmholtzRatio=0.5, structurePenalty=1.,
        CPUs=numberOfCPUs())

    timestamp = datetime.datetime.now().isoformat()
    outdir = 'experimentOutputs/fuzzy/'
    os.makedirs(outdir, exist_ok=True)
    outprefix = outdir + timestamp
    args.update({"outputPrefix": outprefix})

    primitives = [
        Primitive("fand", arrow(treal, treal, treal), _fand),
        Primitive("for", arrow(treal, treal, treal), _for),
    ]

    grammar = Grammar.uniform(primitives)

    training_examples = [
        {"name": "fand", "examples": [fAnd() for _ in range(5000)]},
        #{"name": "for", "examples": [add2() for _ in range(5000)]},
        #{"name": "fand*for", "examples": [add3() for _ in range(5000)]},
    ]
    training = [get_treal_task(item) for item in training_examples]

    # Testing data

    testing_examples = [
        {"name": "fand2", "examples": [fAnd() for _ in range(500)]},
    ]
    testing = [get_treal_task(item) for item in testing_examples]

    # EC iterate

    generator = ecIterator(grammar,
                           training,
                           testingTasks=testing,
                           **args)
    for i, _ in enumerate(generator):
        print('ecIterator count {}'.format(i))

Run Command:

python bin/fuzzy.py -t 2 --testingTimeout 2

[Angular-Angel]

And I did of course remember to edit program.ml to add the new primitives. The program runs, it just doesn't solve anything for this domain.

[Angular-Angel]

I think I must have some kind of error that prevents it from coming up with viable programs? Maybe my inputs are formatted wrong or something. Does anyone know how to test just the part where it comes up with programs?

[Angular-Angel]

Okay, yeah, I'm pretty sure this is cause I don't have it configured to handle multiple inputs.

[madhavk98]

Hi @Angular-Angel how did you configure it to handle multiple inputs? Thanks!

[Angular-Angel]

Nope, never did, sorry. :/

[sneiman]

Hello all -

I am failing to build the ocaml binaries, so am prevented from building my own domain. I got everything else to work well. I have put up an issue - #96 - describing the problems. Hoping you all have a few minutes to see if you can give me some insight ...

thanks

[DanieleMarchei]

After a lot of trial and error I manage to make this use case work, here is the python script:

import datetime
import os
import random

try:
    import binutil # required to import from dreamcoder modules
except ModuleNotFoundError:
    import bin.binutil # alt import if called as module

from dreamcoder.ec import commandlineArguments, ecIterator
from dreamcoder.grammar import Grammar
from dreamcoder.program import Primitive
from dreamcoder.task import Task
from dreamcoder.type import arrow, tlist, treal
from dreamcoder.utilities import numberOfCPUs

# These four definitions work the same... for some reason

def _fand(x): return lambda y : x * y
def _for(x): return lambda y : x + y - x * y

# def _fand(x, y): return x * y
# def _for(x, y): return x + y - x * y

# def _fand(x, y): return 3
# def _for(x, y): return 5

# def _fand(x, y): return lambda x, y :  x * y
# def _for(x, y): return lambda x, y :  x + y - x * y

def fAnd():
    x = random.choice(range(0, 100)) / 100
    y = random.choice(range(0, 100)) / 100

    # we return the two inputs separately and not in a tuple
    return {"x":x, "y":y, "o": x * y}

def get_treal_task(item):
    return Task(
        item["name"],
        arrow(treal, treal, treal),

        # we have two inputs, so we have to provide something in the form ((input1, input2), output)
        [((ex["x"],ex["y"]), ex["o"]) for ex in item["examples"]],
    )

if __name__ == "__main__":

    args = commandlineArguments(
        enumerationTimeout=10, activation='tanh',
        iterations=10, recognitionTimeout=3600,
        a=3, maximumFrontier=10, topK=2, pseudoCounts=30.0,
        helmholtzRatio=0.5, structurePenalty=1.,
        CPUs=numberOfCPUs())

    timestamp = datetime.datetime.now().isoformat()
    outdir = 'experimentOutputs/fuzzy/'
    os.makedirs(outdir, exist_ok=True)
    outprefix = outdir + timestamp
    args.update({"outputPrefix": outprefix})

    primitives = [
        Primitive("fand", arrow(treal, treal, treal), _fand),
        Primitive("for", arrow(treal, treal, treal), _for),
    ]

    grammar = Grammar.uniform(primitives)

    training_examples = [
        {"name": "fand", "examples": [fAnd() for _ in range(5000)]},
    ]
    training = [get_treal_task(item) for item in training_examples]

    # Testing data

    testing_examples = [
        {"name": "fand2", "examples": [fAnd() for _ in range(500)]},
    ]
    testing = [get_treal_task(item) for item in testing_examples]

    # EC iterate

    generator = ecIterator(grammar,
                        training,
                        testingTasks=testing,
                        **args)
    for i, _ in enumerate(generator):
        print('ecIterator count {}'.format(i))

And here are the two primitives added to the program.ml file:

let primitive_fand = primitive "fand" (treal @> treal @> treal) (fun x y -> x *. y);;
let primitive_for = primitive "for" (treal @> treal @> treal) (fun x y -> x +. y -. x *. y);;

Be careful when operating with floating numbers in OCaml because, for example, if you need to perform addition then you need to use the +. operator instead of the regular +. Same thing for all the other operators.

However this does not solve the fundamental issue here: how do the python primitives have to be defined? As you can see by the above script, there are some very funky primitives and all of them work if we use the ocaml solver. If we switch to the python solver then only the first one works. I think this gives us a clue on how to define primitives with multiple inputs. In fact, the script dreamcoder/domains/list/listPrimitives.py uses this exact semantic.