Got backtesting data to share?

[aweijnitz]

Hi, recently found your repo and it looks great. I am looking to experiment a little myself and you mention that before overwhelming MtGox by downloading historic trading data, one could try to ping you.

So, do you have a file, or similar for download before I unleash python download-mtgox-data.py?

Thanks for making these available!

Anders

P.S.

1. Haven't checked the source yet, but is there some argument I could pass to download-mtgox-data.py to limit historical download by date (say limit to the last 10 days).
2. Will running the download command several times do "catch up" or start from scratch again?

[PeterMalkin]

Anders,

thanks for your interest to my simple bots.

I am uploading a copy of mtgox data for you. In about an hour it should be available here: newgen.spb.su/_/mtgox.sqlite3. This file will contain all the history. To update it ("catch up"), just place the file into "mtgoxdata" folder and run "python download-mtgox-data.py".

I hope you find the backtesting framework useful. Please feel free to share (publicly or privately) if you get any cool strategies to perform well :)

[aweijnitz]

Thanks! Downloading now. From what I have read, it is theoretically possible to get guaranteed profit from automated trading given endless funds. Alas, my funds are limited, so I approach this more as fun challenge and hoping not to lose too much.

I have an idea about using the second derivative (the acceleration of change) as a signal for repeated micro trades, but I still have to verify it. You bots and the dump will be helpful.

Will let you know if I find anything noteworthy.

Thanks again!

Anders

[aweijnitz]

Hi, thanks for the download. Looks like it downloaded just fine, but when running the download script I run in to problems. I know you don't have support on this, but do you have any idea what my be wrong? Wrong version of SQLite?

This is the error I get

$:~/goxtool_lab/mtgoxdata$ python download-mtgox-data.py
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "download-mtgox-data.py", line 27, in run
    cursor.execute("select max(tid) from trades where currency=?",(self.currency,))
DatabaseError: database disk image is malformed

File sizes

$:~/goxtool_lab/mtgoxdata$ ls -lrt
total 593908
-rw-r--r-- 1 goxtool goxtool      2836 Jan  5 21:57 mtgox-plot-prices-to-file.py
-rw-r--r-- 1 goxtool goxtool      3380 Jan  5 21:57 download-mtgox-data.py
-rw-r--r-- 1 goxtool goxtool 608149504 Jan  5 22:05 mtgox.sqlite3

SQLite version: SQLite version 3.7.13 2012-06-11 02:05:22

I am running this on Debian Linux on Raspberry Pi Linux pi2 3.6-trunk-rpi #1 Debian 3.6.9-1~experimental.1+rpi7 armv6l

Any pointer to get me started would would be much appreciated.

Anders

[PeterMalkin]

Oh. I apologize. Let me fix this real quick. I will email you when its done.

Btw, you can use direct email. On Jan 5, 2014 1:33 PM, "Anders Weijnitz" notifications@github.com wrote:

Hi, thanks for the download. Looks like it downloaded just fine, but when running the download script I run in to problems. I know you don't have support on this, but do you have any idea what my be wrong? Wrong version of SQLite?

This is the error I get

$:~/goxtool_lab/mtgoxdata$ python download-mtgox-data.py Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "download-mtgox-data.py", line 27, in run cursor.execute("select max(tid) from trades where currency=?",(self.currency,)) DatabaseError: database disk image is malformed

SQLite version: SQLite version 3.7.13 2012-06-11 02:05:22

I am running this on Debian Linux on Raspberry Pi Linux pi2 3.6-trunk-rpi #1 Debian 3.6.9-1~experimental.1+rpi7 armv6l

Any pointer to get me started would would be much appreciated.

Anders

— Reply to this email directly or view it on GitHubhttps://github.com/PeterMalkin/goxtoolbots/issues/1#issuecomment-31616267 .

[PeterMalkin]

I have thought about using second derivative sign to determine peaks/lows. I think you should try looking at simple_mean_reversion bot. I have started implementing a simple idea that if faster moving averages diverge more and more from slower moving averages, it is a sign that the trend will reverse its direction soon. You can see it works pretty well on the month of december 2013. You can just run "python strategy_logic_simple_mean_reversion.py" to see. Unfortunately the current implementation only pays attention to the span between faster/slower moving averages. If the value exceeds a certain percentage of the price, it treats it as a sell/buy signal. As a consequence, it is not applicable to all situations - for example if there is a peak, but the span is not high enough to trigger the signal.

We should try to plug in the second derivative sign detection. Or may be build a simple state machine that would detect divergence of MAs, then a flat space, then a direction reversion. I believe this may work fairly well. Just need a better idea how to detect such events. I will keep you posted if I make progress.

As for making money, I am pretty sure you do not need infinite funds to win in trading. Just gotta code it up right and backtest carefully.

[PeterMalkin]

Try redownloading the database from the same location. It should work now

[aweijnitz]

Thanks, the image won't download at all now. Did you remove it? (I am in Germany and I was asleep when you posted). Getting Fobidden and a 404 (Not found).

However, following some Googling, I also tried to repair the database and I am having some luck with it. Still running, and it doesn't complain at least.

# Attempt repair
$: echo '.dump'| sqlite3 mtgox.sqlite3 | sqlite3 repaired_mtgox.sqlite3

# Move broken DB away and replace with repaired file
$: mv mtgox.sqlite3 mtgox.sqlite3_broken
$: mv repaired_mtgox.sqlite3 mtgox.sqlite3

#Start download
$: python download-mtgox-data.py
connected USD
0
0

[PeterMalkin]

Ok, I apologize again. Try downloading: newgen.spb.su/_/mtgox.sqlite3.tgz I have verified it worked for me.

[aweijnitz]

Perfect!, downloading now. The repair thing didn't pan out. It managed to save something like 5MB out of the original 500+MB.

Anders

[PeterMalkin]

Just to make sure, here's the right link http://newgen.spb.su/_/mtgox.sqlite3.tgz

On Mon, Jan 6, 2014 at 11:13 AM, Anders Weijnitz notifications@github.comwrote:

Perfect!, downloading now. The repair thing didn't pan out. It managed to save something like 5MB out of the original 500+MB.

Anders

— Reply to this email directly or view it on GitHubhttps://github.com/PeterMalkin/goxtoolbots/issues/1#issuecomment-31677007 .

[aweijnitz]

Looks like it's working this time. Can select from it and the downloading of new data looks like it's working as well.

Thanks!