Gene pool merging and gene exchange

[GoogleCodeExporter]

Since I stopped running/updating my old pool due to multiple errors and started 
a new one, I was wondering how could I grab a few of my old champions and 
insert them into my new environment. One of my gts.py processes was forgotten 
[and running the whole time] during the switchover, and already passed the 
'old' genes to the new environment. I could manually parse the gene server db 
backup files and copy/paste few of the highest scoring genes from the old to 
the new environment, but I'm sure that it could be done in a much more 
efficient way. Maybe the new genes have much to learn from some of the old ones 
that do not have very high scores ;)

If you think about it for a moment.. What's stopping us from exchanging the 
genes between different people that run ga-bitbot? =) One python script that 
would merge them in a smart way is indeed all we need =]

Anyone interested in exchanging the genes [please, be serious, we're talking 
about the 1s and zeroes ;] please reply to this thread/issue, and maybe we can 
spark Brian's interest =]

Original issue reported on code.google.com by purge...@gmail.com on 24 Feb 2012 at 3:46

[GoogleCodeExporter]

The gene strings don't actually carry all the information required to decode 
them. This was a design decision to eliminate redundant static data from being 
stored in the genes. 

The gene server trusts that the clients are all using the same configuration. 
This will cause problems when reloading the database after the gene_def.json 
file has been modified in any way (which I regularly do on the repo). If an old 
gene database is loaded after a configuration change, the genes will not be 
decoded properly and will result in errors if the length of the gene changes.

In order to correct this issue (and satisfy the enhancement request at the same 
time) the gene_server will need to be modified to require that clients send a 
copy of the gene_conf.json before accepting any gene submissions. The contents 
of the configuration can then be stored into the database along with a MD5 hash 
to fingerprint the population. 

This approach would have a few benefits: 
One, it would allow the gene server to manage multiple gene configurations at 
once. 
Two, it would prevent any cross contamination between different population 
configurations. 
Three, since the database would now carry both the genes and the configuration, 
users could share their files. A simple tool could be created to merge/prune 
multiple databases.

Once implemented, a new gts client option could be created to cycle through the 
available gene configurations instead of using the local configuration file. 
This would allow resources to be split across multiple independent populations.

Ok, so far so good; Except that this change would break the rest of the system:

report_gen.py and bcbookie.py currently assume only one configuration

All of the html reports (gene visualization, charts and order book) are setup 
for only one configuration

This would be a big undertaking. The good news is that it's already on my todo 
list, although for an entirely different reason... By supporting completely 
independent populations, a low scoring low markup population could can survive 
along side of a high scoring high markup population. Think one population 
focused on low markup high frequency trading and another that would focus on 
large price swings and high markups; both actively trading at the same time.

I guess this was a long winded way of saying yes I'm interested in making the 
requested change.

Original comment by brian.mo...@gmail.com on 25 Feb 2012 at 5:22

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Thank you for the clarification and the interest. Can't wait to test out the 
changes =)

In the meantime, maybe you want to add this clarification to the ga-bitbot FAQ 
so other people do not try to update their instances from the repository and 
hope to keep their old populations.

Original comment by purge...@gmail.com on 25 Feb 2012 at 7:09

[GoogleCodeExporter]

The FAQ has been updated.

Work has started on implementation.

Original comment by brian.mo...@gmail.com on 27 Feb 2012 at 3:16

• Changed state: Accepted

[GoogleCodeExporter]

gene_server has been updated to implement the gene_library (db collection). 
Backwards compatibility has been maintained but upon the first db reload, the 
old db archive files will be converted to the new format before being deleted.  
The converted data will be stored in the UNDEFINED library. Before it can be 
used, I'll need to create a separate utility script to allow merging & deleting 
of databases. This will be needed to link the gene_def.json file to the db. 
Only after the converted db has been linked will it become active and shareable.

unregistered gts clients (if somehow the old gts.py script is used in the new 
system) will be captured in the UNDEFINED library. 

gts has been updated to register the local gene_def on startup. gts clients are 
also now registered and linked to a gene_def. This allows gts clients to work 
on different databases in the library. 

A new command line option was added to gts 'get_config'. If set, gts will pull 
a random gene_def from the gene server. If no configurations are available gts 
will fall back to the local gene_def.json file.

gal has been updated to launch all gts clients, except for one, with the 
get_config option. This will ensure any new gene_def.json files will be loaded 
into the db.

gene_def has been updated to add a name, description and version variable. This 
is to allow users to tag their gene_def files.

A MD5 hash of the gene_def file is used to register it. Any changes to the file 
at all, even adding spaces, will change the hash and result in a fork in the 
gene library.

When starting the system, the first client to register with the gene server 
sets the default db. The default db is what is used throughout the rest of the 
system (gene vis,charts,bcbookie). This is only temporary until I decide if and 
how I want to automate the switching of default databases from within 
report_gen.py.

Original comment by brian.mo...@gmail.com on 28 Feb 2012 at 6:55

• Changed state: Started

[GoogleCodeExporter]

Sounds great! =)

Tried updating my current instance [git pull] and it seems to work correctly. 
Also, ran the empty, new, instances for a few hours and it also seemed to work 
as intended. Can't wait for the the opportunity to merge my populations, 
exchange/mate them with other users and see what the results will be =)

The only weird issue I noticed is that bcbookie reports:
main: warning - ignoring invalid target order:
{u'status': u'active', u'buy': 4.69841, u'actual': 0, u'target': 4.87918, 
u'sell_period': 0, u'age': 0, u'stop': 4.646258164388718, u'shares': 
5.238379999999999, u'cost': 24.813316301508454, u'master_index': 2, 
u'trade_pos': 976.220180903862, u'score': 0, u'stop_age': 13020.0, 
u'buy_period': 99999}

...which I presume is only because it does not have a valid buy order at this 
moment due to pretty stagnant market conditions.

Please, feel free to ask for my help if you think I could help in any way.

Original comment by purge...@gmail.com on 28 Feb 2012 at 3:57

[GoogleCodeExporter]

Market moved, "system monitor" reports "Buy Order Trigger" > 0, but bcbookie 
still reports "ignoring invalid target order":

{u'status': u'active', u'buy': 4.7116, u'actual': 0, u'target': 4.8911, 
u'sell_period': 0, u'age': 0, u'stop': 4.22110970233263, u'shares': 
5.238379999999999, u'cost': 24.8828278873523, u'master_index': 97, 
u'trade_pos': 979.1069574997002, u'score': 0, u'stop_age': 491460.0, 
u'buy_period': 99999}

Seems the bcbookie does not want to play nice with the new version after all =(

Original comment by purge...@gmail.com on 28 Feb 2012 at 4:11

[GoogleCodeExporter]

regarding comment #6, the issue was caused by a recent change to genetic.py 
which unintentionally caused integers to be generated as floats. This was 
changed back. 

bcbookie has been changed to validate orders with either type. The issue was 
specifically with the stop_age variable; stop_age is now recast as in int 
durring order verification.

Original comment by brian.mo...@gmail.com on 29 Feb 2012 at 1:20