search

BitBotFactory / MikaLendingBot

Automated lending on Cryptocurrency exchanges Poloniex and Bitfinex
http://poloniexlendingbot.readthedocs.io/en/latest/index.html
MIT License
1.11k stars 344 forks source link

Moving your config to work with MACD #501

Open laxdog opened 7 years ago

laxdog commented 7 years ago

TLDR;

Add this to the bottom of your config to use the new method, comment out method = MACD to method = percentile. if you don't use MarketAnalysis you don't need to do anything. 

[MarketAnalysis]
analyseCurrencies = BTC
lendingStyle = 75
MACD_long_win_seconds = 1800
percentile_seconds = 259200

[Daily_min] 
method = MACD
multiplier = 1.05

Moving your config to work with MACD

There are loads of docs around this, so you can read more then once it's all merged.

This is just a quick note of what's changed between the releases and how to get back up and running as quickly as possibly.

FAQ

Q: What is this? A: A new method of working out the best rate to lend at. It analyses the moving average and calculates the rate accordingly.

Q: Why should I use this? A: To get better lending rates. At certain times I've found this to give vastly superior lending rates than the old percentile method. It uses a method similar to this: http://www.investopedia.com/terms/g/goldencross.asp

Q: Does it work on Poloniex and Bitfinex? A: Yes, though bitfinex still needs some work to get the same volume of data as Polo as we don't have a coach for it yet.

Q: How can I get more information? A: The docs have loads of information, and I'm usually on Slack.

Differences

If you are moving an old config over, you can compare to the default.cfg.example

Things you may notice (Only applicable if using MarketAnalysis): 1) There are no text files saved in market_data anymore, only sqlite3 databases 2) Increased network activity as more data is recorded 3) A small increase in memory usage 4) Decreased disk usage in the market_data folder. The DB is more efficient even though we store more data. 5) You need to have numpy and pandas installed pip install -r requirements.txt

Config changes: 1) There is a new section called [MarketAnalysis] if you don't want to analyse the market for better rates, ignore this issue completely and nothing will change for you. If you do want to, it needs be in your config. 2) There is another new section called [Daily_min] if you want to use the old percentage style analysis you can ignore this section. If you want to use the new MACD style (which generally seems to provide better rates), you need this section. 3) lendingStyle (used for percentile analysis) has been moved to [MarketAnalysis] 4) analyseMaxAge has been renamed keep_history_seconds and moved to [MarketAnalysis] 5) Although analyseMaxAge was rename, in the previous iteration it was used to delete old data as well as server as the time period to calculate from, this time period is now controlled via percentile_seconds 6) analyseUpdateInterval has been removed, the coach is now used to record data as often as possible. 7) analyseCurrencies has move to the [MarketAnalysis] section

Config

Minimum required to use MACD: 1) [MarketAnalysis] section containing:

analyseCurrencies = BTC
lendingStyle = 75
MACD_long_win_seconds = 1800
percentile_seconds = 259200

2) [Daily_min] section containing:

method = MACD
multiplier = 1.05

Further

There are a lot more tweaks you can carry out and recommended configs in the docs. Turning off hidecoins is probably one of the better ones I've found to use.

Hopefully people can play around with the settings and see what works for them. You can start by changing the window sizes. Then changing the multiplier.

I try to aim to keep about 95% - 100% of currency on loan. This way you know you are not lending at too low a rate and always beating the market when you loan.

Special thanks to @kimp on Slack for helping out so much with testing.

utdrmac commented 7 years ago

"analyseUpdateInterval" is still in the market_analysis.rst; says above it was removed.

utdrmac commented 7 years ago

Default value of MACD_short_win_seconds is technically MACD_long_win_seconds / 12, not 150 as the docs say.

utdrmac commented 7 years ago

Odd output:

Short higher: Cur:ETH, MACD:0.000067, Perc:0.000065, Best:0.000064
Short higher: Long  higher: Cur:BCH, MACD:0.000085, Perc:0.000087, Best:0.000064
Long  higher: Long  higher: Cur:BTC, MACD:0.000198, Perc:0.000192, Best:0.000185

Long  higher: Cur:ETH, MACD:0.000067, Perc:0.000065, Best:0.000064
Long  higher: Long  higher: Cur:BCH, MACD:0.000081, Perc:0.000087, Best:0.000076
Long  higher: Long  higher: Cur:BCH, MACD:0.000081, Perc:0.000087, Best:0.000076
utdrmac commented 7 years ago

I don't think this is evaluating correctly: https://github.com/BitBotFactory/poloniexlendingbot/blob/master/modules/MarketAnalysis.py#L346

utdrmac commented 7 years ago

Here's a new one:

Long  higher: Long  higher: Cur:BCH, MACD:0.000090, Perc:0.000087, Best:0.000085
Long  higher: Short higher: Cur:BTC, MACD:0.000214, Perc:0.000202, Best:0.000195
Short higher: Long  higher: Cur:ETH, MACD:0.000056, Perc:0.000056, Best:0.000049
Long  higher: Long  higher: Cur:BCH, MACD:0.000090, Perc:0.000087, Best:0.000085
Long  higher: 2017-09-01 14:18:16 Error list index out of range
Traceback (most recent call last):
  File "lendingbot.py", line 94, in <module>
    Lending.lend_all()
  File "/usr/src/bitfinex/modules/Lending.py", line 238, in lend_all
    usable_currencies += lend_cur(cur, total_lent, lending_balances, ticker)
  File "/usr/src/bitfinex/modules/Lending.py", line 409, in lend_cur
    orders = construct_orders(active_cur, active_bal, active_cur_total_balance, ticker)  # Build all potential orders
  File "/usr/src/bitfinex/modules/Lending.py", line 310, in construct_orders
    top_rate, bottom_rate = get_gap_mode_rates(cur, cur_active_bal, cur_total_balance, ticker)
  File "/usr/src/bitfinex/modules/Lending.py", line 367, in get_gap_mode_rates
    top_rate = get_gap_rate(cur, gap_top, order_book, cur_total_balance, True)
  File "/usr/src/bitfinex/modules/Lending.py", line 297, in get_gap_rate
    return Decimal(order_book['rates'][i])
IndexError: list index out of range

Unhandled error, please open a Github issue so we can fix it!
Long  higher: Cur:ETH, MACD:0.000056, Perc:0.000055, Best:0.000049
Long  higher: Long  higher: Cur:BCH, MACD:0.000090, Perc:0.000087, Best:0.000085
utdrmac commented 7 years ago

again

[root@gasgiant bitfinex]# python lendingbot.py
Starting WebServer at 10.10.10.202 on port 8001 with template www
Started WebServer, lendingbot status available at http://10.10.10.202:8001/lendingbot.html
Welcome to Lending Bot on BITFINEX

ETH : Need more data for analysis, still collecting. I have 211/270 records
Caught exception during MACD analysis, using percentile for now
BCH : Need more data for analysis, still collecting. I have 214/270 records
Caught exception during MACD analysis, using percentile for now
BTC : Need more data for analysis, still collecting. I have 217/270 records
Caught exception during MACD analysis, using percentile for now
ETH : Need more data for analysis, still collecting. I have 211/270 records
Caught exception during MACD analysis, using percentile for now
2017-09-01 19:03:39 Error list index out of range

Traceback (most recent call last):
  File "lendingbot.py", line 94, in <module>
    Lending.lend_all()
  File "/usr/src/bitfinex/modules/Lending.py", line 240, in lend_all
    lend_all()
  File "/usr/src/bitfinex/modules/Lending.py", line 238, in lend_all
    usable_currencies += lend_cur(cur, total_lent, lending_balances, ticker)
  File "/usr/src/bitfinex/modules/Lending.py", line 409, in lend_cur
    orders = construct_orders(active_cur, active_bal, active_cur_total_balance, ticker)  # Build all potential orders
  File "/usr/src/bitfinex/modules/Lending.py", line 310, in construct_orders
    top_rate, bottom_rate = get_gap_mode_rates(cur, cur_active_bal, cur_total_balance, ticker)
  File "/usr/src/bitfinex/modules/Lending.py", line 367, in get_gap_mode_rates
    top_rate = get_gap_rate(cur, gap_top, order_book, cur_total_balance, True)
  File "/usr/src/bitfinex/modules/Lending.py", line 297, in get_gap_rate
    return Decimal(order_book['rates'][i])
IndexError: list index out of range

ExceptionType: 'IndexError'
Unhandled error, please open a Github issue so we can fix it!
utdrmac commented 7 years ago

Came home 2hrs later to a slew of errors: https://ghostbin.com/paste/n7q7j

laxdog commented 7 years ago

1) Fixed 2) Fixed 3) A small bit of extra info being written to stdout. You should only see that stuff if you have ma_debug_log set to True. 4) Removed in the fix for 3, that was debug / dev code. 5, 6, 7) Pretty sure these are all related. They look like the DB has been corrupted or bad Pandas data is making it in somehow. It looks like only the BCH data is corrupted / giving problems.

@utdrmac , Can you ping me on slack when you're about and I can work through it. In the mean time, assuming you're on *nix, could give me output from:

df -lh
pip freeze
ls -lrth $PATH_TO_BOT
ls -lrth $PATH_TO_BOT/market_data

Also, your cfg file with the keys and secrets removed.

I haven't done anywhere near the amount of testing on bitfinex as I have on poloniex, so I'm expecting that could be something to do with it.

utdrmac commented 7 years ago 
[root@gasgiant bitfinex]# df -lh
Filesystem      Size  Used Avail Use% Mounted on
udev             10M  4.0K   10M   1% /dev
/dev/md0        4.0G  2.7G  1.2G  71% /
tmpfs           495M     0  495M   0% /dev/shm
tmpfs           495M  3.7M  491M   1% /run
tmpfs           248M  1.1M  246M   1% /run/lock
tmpfs           495M     0  495M   0% /sys/fs/cgroup
/dev/md127      2.8T  2.1T  715G  75% /data
/dev/md127      2.8T  2.1T  715G  75% /home
/dev/md127      2.8T  2.1T  715G  75% /apps
/dev/md127      2.8T  2.1T  715G  75% /run/nfs4/data/media

pip freeze: http://termbin.com/8g06

[root@gasgiant bitfinex]# ls -lrth /usr/src/bitfinex/
total 44K
-rw-r--r-- 1 root root 1.1K Aug  7 09:54 LICENSE
-rw-r--r-- 1 root root  518 Aug  7 09:54 docker-compose.yaml
-rw-r--r-- 1 root root  538 Aug  9 17:35 Dockerfile
-rw-r--r-- 1 root root 3.2K Aug  9 17:35 README.md
-rw-r--r-- 1 root root 7.8K Sep  1 10:34 default.cfg.example
drwxr-xr-x 1 root root  254 Sep  1 10:34 docs
-rw-r--r-- 1 root root   38 Sep  1 10:34 requirements.txt
drwxr-xr-x 1 root root  200 Sep  1 10:34 www
drwxr-xr-x 1 root root  100 Sep  1 10:34 tests
drwxr-xr-x 1 root root  180 Sep  1 18:03 plugins
-rwxr-xr-x 1 root root 6.0K Sep  1 19:02 lendingbot.py
-rw-r--r-- 1 root root 7.1K Sep  1 21:56 default.cfg
drwxr-xr-x 1 root root  934 Sep  1 21:56 modules
drwxr-xr-x 1 root root  216 Sep  2 11:08 market_data
[root@gasgiant bitfinex]#
[root@gasgiant bitfinex]# ls -lrth /usr/src/bitfinex/market_data/
total 9.2M
-rw-r--r-- 1 root root  212 Aug  7 09:54 readme.txt
-rw-r--r-- 1 root root  36K Sep  1 21:57 loan_history.sqlite3
-rw-r--r-- 1 root root 1.7M Sep  2 11:08 ETH.db
-rw-r--r-- 1 root root 1.1M Sep  2 11:08 BCH.db
-rw-r--r-- 1 root root    0 Sep  2 11:08 BCH.db-wal
-rw-r--r-- 1 root root  32K Sep  2 11:08 BCH.db-shm
-rw-r--r-- 1 root root    0 Sep  2 11:08 ETH.db-wal
-rw-r--r-- 1 root root  32K Sep  2 11:08 ETH.db-shm
-rw-r--r-- 1 root root 2.4M Sep  2 11:44 BTC.db
-rw-r--r-- 1 root root 4.0M Sep  2 12:01 BTC.db-wal
-rw-r--r-- 1 root root  32K Sep  2 12:02 BTC.db-shm

Config: http://termbin.com/leqf

laxdog commented 7 years ago

Starting to think 5 & 6 are not related to these changes actually and are just a problem with the gaps on Bitfinex. I'll need to look closer at it again. Going to focus on 7 for now.

laxdog commented 7 years ago

@utdrmac OK, I think I've found it. I don't have a fix for it yet though.

Can you remove all the DBs, and change the ACTIVE in the analyse_currenices to the currencies you're actually lending and then restart everything.

I want to make sure that's what's wrong before I start working on a fix.

I've realised also I need to prefix all the DBs with the exchange name or we'll never be able to run multiple exchanges on the same bot. That's a side point though.

utdrmac commented 7 years ago 
$ rm -f market_data/*
...
[MarketAnalysis]
analyseCurrencies = BTC,BCH,ETH
lendingStyle = 85
MACD_long_win_seconds = 1800
percentile_seconds = 86400
ma_debug_log = True
keep_history_seconds = 95040

[Daily_min]
method = MACD
multiplier = 1.05

Will report back in a couple hours.

nicjb commented 7 years ago

When trying to git pull from pythonanywhere, I get:

Your configuration specifies to merge with the ref 'master'
from the remote, but no such ref was fetched.

Any help on this one?

laxdog commented 7 years ago

MACD is unlikely to work on pythonanywhere. That said, that sounds like something is messed up with your git checkout.

The easiest thing to do, would be back up your config, delete the whole directory and check it out again.