search

BitBotFactory / MikaLendingBot

Automated lending on Cryptocurrency exchanges Poloniex and Bitfinex
http://poloniexlendingbot.readthedocs.io/en/latest/index.html
MIT License
1.11k stars 344 forks source link

MACD analysis uses large amounts of disk IO #552

Closed utdrmac closed 6 years ago

utdrmac commented 7 years ago

I'm using macd analysis on poloniex and percentile on bittrex. The process running poloniex is responsible for almost 1MB/sec write activity on my server while bittrex around 50-100K.

Is there any way to lessen the IO? I turned off the WAL on sqlite but that doesn't seem to have made a difference.

Thoughts?

  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 3177 be/4 root        0.00 B/s  675.02 K/s  0.00 % 14.31 % python lendingbot.py
utdrmac commented 7 years ago

Added some debug lines to code to print each WRITE operation. About 1 insert/sec. Seems like a lot of the same data over and over. Expected? Needed?

SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
2017-10-02 17:55:41 Error HTTP Error 429: Too Many Requests Requesting returnActiveLoans.  Poloniex reports: 'Please do not make more than 8 API calls per second.'
2017-10-02 17:55:41 Error IP has been banned due to many requests. Sleeping for 130.0 seconds
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012900,0.01653648,0.00013000,0.24341468,0.00016000,0.01247971,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012900,0.01653648,0.00013000,0.24341468,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012900,0.01653648,0.00013000,0.24341468,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012900,0.01653648,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012900,0.01653648,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012900,0.01653648,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012900,0.01653648,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012900,0.01653648,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012900,0.01653648,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012900,0.01653648,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012900,0.01653648,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012900,0.01653648,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012900,0.01653648,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012800,0.01068368,0.00012833,0.01068368,0.00012866,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012790,0.05000000,0.00012800,0.01068368,0.00012833,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012790,0.05000000,0.00012800,0.01068368,0.00012833,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012790,0.05000000,0.00012800,0.01068368,0.00012833,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012790,0.05000000,0.00012800,0.01068368,0.00012833,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
SQL: INSERT INTO loans (rate0, amnt0, rate1, amnt1, rate2, amnt2, percentile) VALUES (0.00012733,0.03557047,0.00012790,0.05000000,0.00012800,0.01068368,0);
laxdog commented 7 years ago

1/s is expected on bitfinex, yes and 6/s on poloniex. They both collect as much data as possible.

Whether or not it's needed depends on your use case. If you want an accurate representation of the market so as to calculate the correct lend rate, then yes, it is needed.

If you're more focused on changing settings frequently keeping an eye on the markets you're lending on, then it's not as hard a requirement.

It's still better to have it that not have it. There are a lot of offers that are taken very quickly and a lot of spikes that happen for short periods of time. Having the bot constantly record what the current market looks like allows these to be factored into the lending rate that MACD works out.

I'm still convinced there's something wrong with your measurement or your server. I'm running 4 bots on one server and I don't see as high disk write as that.

@utdrmac can you paste me the output of these please:

sudo iotop -o -P -d 10 -n 5 -b
python -c "import sys,sqlite3; print(sys.version, sqlite3.version, sqlite3.sqlite_version)"
cat /etc/*release*
uname -a

Are you writing to a local disk or share? What's the file system? When are you taking the measurement, on startup or after it's been running for a while?

If you have fatrace installed, could you run

sudo fatrace -f W -c

Just to make sure that it's actually the MACD db files that are the destination of all the writes.

Also, you should keep WAL on, it should be more efficient.

utdrmac commented 7 years ago

Been running for almost a week. Constant IO. It's only noticeable because when my house is quiet, I can hear the HDDs click with every write. If I stop both bots, maybe I hear a click every 10s or so for normal disk/fs stuff.

root@gasgiant:~# iotop -o -P -d 10 -n 5 -b
Total DISK READ :       0.00 B/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:       0.00 B/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ :     405.21 B/s | Total DISK WRITE :     375.53 K/s
Actual DISK READ:     405.21 B/s | Actual DISK WRITE:     582.59 K/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
30794 be/4 root        0.00 B/s  344.67 K/s  0.00 %  1.78 % python lendingbot.py
18785 be/4 root      405.21 B/s   21.37 K/s  0.00 %  0.11 % python lendingbot.py
30744 be/4 openvpn     0.00 B/s    3.17 K/s  0.00 %  0.00 % openvpn --daemon ovpn-server --writepid /var/run/openvpn/server.pid --status /run/openvpn/server.status 10 --cd /etc/openvpn --config /etc/openvpn/server.conf
19763 be/4 root        0.00 B/s 1620.85 B/s  0.00 %  0.00 % [kworker/u8:1]
19970 be/4 root        0.00 B/s    3.17 K/s  0.00 %  0.00 % [kworker/u8:3]
19642 be/4 root        0.00 B/s 1620.85 B/s  0.00 %  0.00 % [kworker/u8:0]
Total DISK READ :     404.57 B/s | Total DISK WRITE :     534.16 K/s
Actual DISK READ:     404.57 B/s | Actual DISK WRITE:     982.78 K/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
30794 be/4 root        0.00 B/s  419.98 K/s  0.00 %  2.06 % python lendingbot.py
 1186 be/4 root        0.00 B/s   39.51 K/s  0.00 %  0.18 % [btrfs-transacti]
18785 be/4 root      404.57 B/s   37.93 K/s  0.00 %  0.12 % python lendingbot.py
30744 be/4 openvpn     0.00 B/s    6.32 K/s  0.00 %  0.00 % openvpn --daemon ovpn-server --writepid /var/run/openvpn/server.pid --status /run/openvpn/server.status 10 --cd /etc/openvpn --config /etc/openvpn/server.conf
19970 be/4 root        0.00 B/s    9.48 K/s  0.00 %  0.00 % [kworker/u8:3]
19642 be/4 root        0.00 B/s    6.32 K/s  0.00 %  0.00 % [kworker/u8:0]
19763 be/4 root        0.00 B/s   14.22 K/s  0.00 %  0.00 % [kworker/u8:1]
 2707 be/4 root        0.00 B/s  404.57 B/s  0.00 %  0.00 % nscd
Total DISK READ :     809.65 B/s | Total DISK WRITE :     380.71 K/s
Actual DISK READ:     809.65 B/s | Actual DISK WRITE:     528.17 K/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
30794 be/4 root        0.00 B/s  313.11 K/s  0.00 %  2.02 % python lendingbot.py
18785 be/4 root      809.65 B/s   29.65 K/s  0.00 %  0.13 % python lendingbot.py
 2830 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.03 % readynasd -v 3 -t
18224 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/1:0]
11752 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0]
30744 be/4 openvpn     0.00 B/s    3.16 K/s  0.00 %  0.00 % openvpn --daemon ovpn-server --writepid /var/run/openvpn/server.pid --status /run/openvpn/server.status 10 --cd /etc/openvpn --config /etc/openvpn/server.conf
19468 be/4 root        0.00 B/s    3.16 K/s  0.00 %  0.00 % [kworker/u8:4]
20048 be/4 root        0.00 B/s 1619.31 B/s  0.00 %  0.00 % [kworker/u8:2]
19763 be/4 root        0.00 B/s   17.39 K/s  0.00 %  0.00 % [kworker/u8:1]
19970 be/4 root        0.00 B/s    7.91 K/s  0.00 %  0.00 % [kworker/u8:3]
19642 be/4 root        0.00 B/s    4.74 K/s  0.00 %  0.00 % [kworker/u8:0]
Total DISK READ :       0.00 B/s | Total DISK WRITE :     496.09 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     809.91 K/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
30794 be/4 root        0.00 B/s  468.45 K/s  0.00 %  2.20 % python lendingbot.py
18785 be/4 root        0.00 B/s   19.75 K/s  0.00 %  0.09 % python lendingbot.py
30744 be/4 openvpn     0.00 B/s    3.16 K/s  0.00 %  0.00 % openvpn --daemon ovpn-server --writepid /var/run/openvpn/server.pid --status /run/openvpn/server.status 10 --cd /etc/openvpn --config /etc/openvpn/server.conf
20048 be/4 root        0.00 B/s    3.16 K/s  0.00 %  0.00 % [kworker/u8:2]
19763 be/4 root        0.00 B/s 1617.84 B/s  0.00 %  0.00 % [kworker/u8:1]

('2.7.9 (default, Jun 29 2016, 13:08:31) \n[GCC 4.9.2]', '2.6.0', '3.19.3')

root@gasgiant:~# cat /etc/*release*
PRETTY_NAME="ReadyNASOS 6.8.0"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Linux gasgiant 4.4.79.x86_64.1 #1 SMP Mon Jul 31 15:14:40 PDT 2017 x86_64 GNU/Linux

# fatrace -f W -c
fatrace: invalid option -- 'f
utdrmac commented 7 years ago

filesystem is btrfs. local disk.

utdrmac commented 6 years ago

Switched over to a RPi2. Same settings. Disk activity is much lower. I wonder why.

21410 be/4 root        0.00 B/s   18.45 K/s  0.00 %  2.76 % python lendingbot.py -cfg polo.cfg
21445 be/4 root        0.00 B/s    2.64 K/s  0.00 %  0.31 % python lendingbot.py -cfg bit.cfg
utdrmac commented 6 years ago

Weird now that the polo bot is consistently writing more IO than Bitfinex, despite Polo only doing 1 coin and Bitfinex doing 3 coins.

utdrmac commented 6 years ago

Back on NAS, Bitfinex:

 6072 be/4 root        0.00 B/s   49.65 K/s  0.00 %  0.68 % python lendingbot.py
laxdog commented 6 years ago

The output from iotop looks fine @utdrmac . It's only doing 300-400k/s total.

Also, polo is writing more as they allow more requests per second. So we make more.

utdrmac commented 6 years ago

Must be an OS thing. Running both bots on my RPi2 produces far less disk IO than running same config on my NAS.