search

lspector / Clojush

The Push programming language and the PushGP genetic programming system implemented in Clojure.
http://hampshire.edu/lspector/push.html
Eclipse Public License 1.0
331 stars 92 forks source link

`clojush.problems.synthetic.order` (maybe others) shouldn't be simplified by default #117

Open Vaguery opened 9 years ago

Vaguery commented 9 years ago

To be frank, this is more a facet of "what are the individual problems for exactly?"

I'm just running all of them in turn by hand and watching, and I find several in synthetic that are interesting number-ordering tasks which get solved... and then the default "simplify" system comes along at the end and makes the "simpler" answers nonsensical. For instance, the order task is described as "put the positive numbers in front of their negative complements", and that works, and then automatic simplification comes along and deletes all the negative numbers. Not violating the terms of the task at all, of course.

In hindsight, I guess this is "they were written before simplification was a thing", but...?

Practically speaking maybe this one should have simplification turned off?

lspector commented 9 years ago

FWIW I think this mostly just shows how silly the synthetic problems are. Mostly I think they're just a product of a lot of people getting stuck in the same academic cul-de-sac, and I think that mostly they just serve to diagnose unfortunate choices for program representations and genetic operators. They are explicitly syntactic rather than semantic, while GP is really always about getting a program that DOES something that we want done, so they almost completely miss the point by definition.

Vaguery commented 9 years ago

I disagree! Lots of examples fully explained are almost required if this is ever going to be more than bunch of enigma wrapped in a bibliography. :)

Vaguery commented 9 years ago

Besides: many ways to skin the proverbial vertebrate.

lspector commented 9 years ago

Okay, examples (fully explained) are good, even if they're silly. But these are silly.

kephale commented 9 years ago

An extra 2-cents:

Easy, maybe. Silly is contestable. The order/majority style problems are indeed effectively syntactic benchmarks. However, in Push (at least versions close to Push3) where there are effectively no constraints on program representation (but many instructions do require things to happen on stacks before they can have an effect [i.e. before their semantic meaning can even be realized/evaluated]), understanding the ability of an evolutionary algorithm to select for ability to perform syntactic organization is quite important. Order/majority factor out these semantics, allowing an EA to be tested for its ability to organize code. Clearly on real problems instruction semantics can greatly facilitate search, and it would be interesting to know if semantics can drown out the effects of syntactic search abilities. If we are talking about traditional tree-based GP which Order/Majority were developed for, then I would be more likely to agree that they are silly (although I need to think about even that more), but for Push, I think Order/Majority are actually especially pertinent. When syntax is not enforced by a GP system, the ability of that GP system to discover the correct syntax becomes an important question. That is one reason why it was significant that autoconstructive GP could achieve comparable performance to PushGP.

Vaguery commented 9 years ago

See? I like it. I'd like to see a blog-post length entry for all of them, but that's me. :)

Actual unknown: are the problems included in the clojar? In other words (because I'm still learning) if somebody puts in :dependencies [[org.clojure/clojure "1.6.0"]] will they all the problems downloaded along with it?

lspector commented 9 years ago

kephale: you're right -- "silly" was overstating things. But what these problems allow one to do is, as you say, to study the system's ability to produce syntax. This is useful when asking meta-questions about a GP systems representations and genetic operators. But when actually solving a problem one generally cares almost entirely about semantics, not syntax. So the relevance of performance on these "synthetic" problems (I think that "structural" is a better term for them) for predicting performance on "real" problems is indirect at best.

Vaguery: Yes, you will get all of the problems if you include a dependency to Clojush, but that would actually look like [clojush "2.0.18"]. The dependency you've included is for Clojure itself.