ctubio / Krypto-trading-bot

Self-hosted crypto trading bot (automated high frequency market making) written in C++
https://127.0.0.1:3000
Other
3.37k stars 823 forks source link

hangs on bitfinex #428

Closed hamedafra closed 6 years ago

hamedafra commented 6 years ago

Bot hangs on bitfinex after few minutes of working and debug mode also shows no error:

screen shot 2017-11-24 at 5 39 20 pm

It will be ok after I restart the bot

donaldforest commented 6 years ago

Same was on another server. I transfer bot to another clean server. Then looks bot works fine 24 hours on new. But I update it 2 days ago and then every mornong I found that bot UI not responsing, bot stop made orders, but completely answer in ssh console. I restart it now every morning. Using: HitBTC. Logl looks like yours. Nothing interested and last message in log is about get some trade data from exchange.. :( Are You use --free-version?

serzhiio commented 6 years ago

Looks like you have big latency. I've expirienced something like that on bitfinex.

donaldforest commented 6 years ago

In my case - no.

~/K$ ping api.hitbtc.com PING api.hitbtc.com (165.227.244.237) 56(84) bytes of data. 64 bytes from 165.227.244.237: icmp_seq=1 ttl=58 time=97.4 ms 64 bytes from 165.227.244.237: icmp_seq=2 ttl=58 time=94.0 ms 64 bytes from 165.227.244.237: icmp_seq=3 ttl=58 time=94.2 ms 64 bytes from 165.227.244.237: icmp_seq=4 ttl=58 time=93.9 ms 64 bytes from 165.227.244.237: icmp_seq=5 ttl=58 time=95.6 ms 64 bytes from 165.227.244.237: icmp_seq=6 ttl=58 time=94.0 ms ^C --- api.hitbtc.com ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 5007ms rtt min/avg/max/mdev = 93.965/94.914/97.436/1.270 ms

hamedafra commented 6 years ago

I tried both modes and there was no difference, I also tried --headless mode and it worked a little more than the UI mode, but eventually it stoped working after a while

donaldforest commented 6 years ago

You using server OS? Clean or with lamp? Linux\Unix\Windows?

hamedafra commented 6 years ago

Clean Debian, on redstation

donaldforest commented 6 years ago

I using Ubuntu 16. If this will continue, I ask my admins to trace error. If get some result, Il reply this ticket.

ctubio commented 6 years ago

what means hangs? the UI is not responsive or it is responsive but with a red background? or is green background but not placing new orders?

donaldforest commented 6 years ago

In my case: No K in linux top PIDs. Console app don't quit by "q" (non naked mode). Just Ctrl+c - exit from console app. Web UI not response.

adirta commented 6 years ago

I have this problem. Since some update a week ago, approximately

lunnh commented 6 years ago

I have the same problem. I kept downgrading back until 610abb6 and it's running normally.

donaldforest commented 6 years ago

Updated yesterday. Switched update and directory rights from root to standard user. 12 hours uptime and continue working ok. Testing..

malt001 commented 6 years ago

Updated today to latest version. Still freezes after a few minutes on Ubuntu and HitBTC

esomore commented 6 years ago

Same here, freezing on ubuntu with hitbtc and bitfinex

SpencerMKSmith commented 6 years ago

I am getting this same issue but with a different exchange. I've tried on multiple versions of Ubuntu and Debian and I get the same freezing after a couple of minutes.

@donaldforest is it still running well for you after making that change? Would you mind giving the exact permissions that you used and changes you made to get it running?

esomore commented 6 years ago

@SpencerMKSmith @ctubio also tried on multiple machines and also with docker, after minute or two the process hangs. I can't tell for sure but it might be related to a recent change in sqlite?

write(4, "SQLite format 3\0\20\0\1\1\0@ \0\0\n\251\0\0\0\30"..., 4096) = 4096 lseek(4, 8192, SEEK_SET) = 8192 write(4, "\r\17\315\0\6\17\305\0\17\305\17\352\17\362\17\332\17\323\17\371\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096 lseek(4, 94208, SEEK_SET) = 94208 write(4, "\r\0\0\0\"\5e\0\17\260\17`\17\20\16\300\16p\16 \r\320\r\200\r0\f\340\f\220\f@"..., 4096) = 4096 fdatasync(4) = 0 close(12) = 0 unlink("/data/db/K.1.BTC.USD.db-journal") = 0 fcntl(4, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741826, l_len=510}) = 0 fcntl(4, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=2}) = 0 fcntl(4, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 clock_gettime(CLOCK_REALTIME, {1511974092, 242370494}) = 0 clock_gettime(CLOCK_REALTIME, {1511974092, 242405174}) = 0 epoll_wait(3, [{EPOLLIN, {u32=3143053952, u64=93925487882880}}], 1024, 999) = 1 clock_gettime(CLOCK_REALTIME, {1511974092, 316097002}) = 0 read(7, "\27\3\3\1|", 5) = 5 read(7, ")\204\2055\277\224\237d\314\22!>3~\270\334\245,<\21\300Se\232d\316\2258%\324\353"..., 380) = 380 setsockopt(7, SOL_TCP, TCP_CORK, [1], 4) = 0 clock_gettime(CLOCK_REALTIME, {1511974092, 316328164}) = 0 setsockopt(7, SOL_TCP, TCP_CORK, [0], 4) = 0 epoll_wait(3, [{EPOLLIN, {u32=3143053952, u64=93925487882880}}], 1024, 999) = 1 clock_gettime(CLOCK_REALTIME, {1511974092, 420166024}) = 0 read(7, "\27\3\3\1\23", 5) = 5 read(7, ")\204\2055\277\224\240.\303:\317\371\370\222\361\266\331\3\210\312\271G$\307\346\33\374u\302J\251"..., 275) = 275 setsockopt(7, SOL_TCP, TCP_CORK, [1], 4) = 0 clock_gettime(CLOCK_REALTIME, {1511974092, 420458524}) = 0 setsockopt(7, SOL_TCP, TCP_CORK, [0], 4) = 0 epoll_wait(3, [{EPOLLIN, {u32=3143053952, u64=93925487882880}}], 1024, 999) = 1 clock_gettime(CLOCKREALTIME, {1511974092, 526261671}) = 0 read(7, "\27\3\3\1\23", 5) = 5 read(7, ")\204\2055*\277\224\241WVX\f#\344\n\212\325\D\35\347tP\200\211.\2022 \334\275"..., 275) = 275 setsockopt(7, SOL_TCP, TCP_CORK, [1], 4) = 0 clock_gettime(CLOCK_REALTIME, {1511974092, 526608457}) = 0 setsockopt(7, SOL_TCP, TCP_CORK, [0], 4) = 0 epoll_wait(3, [{EPOLLIN, {u32=3143053952, u64=93925487882880}}], 1024, 999) = 1 clock_gettime(CLOCK_REALTIME, {1511974092, 625832458}) = 0 read(7, "\27\3\3\1\23", 5) = 5 read(7, ")\204\2055*\277\224\242R\326\300\204\321\274PRX:\t\36\3440\211j\361\251\323z\373\227\327\236"..., 275) = 275 setsockopt(7, SOL_TCP, TCP_CORK, [1], 4) = 0 clock_gettime(CLOCK_REALTIME, {1511974092, 626103707}) = 0 setsockopt(7, SOL_TCP, TCP_CORK, [0], 4) = 0 epoll_wait(3, [{EPOLLIN, {u32=3143038448, u64=93925487867376}}], 1024, 999) = 1 clock_gettime(CLOCK_REALTIME, {1511974092, 642922586}) = 0 read(6, "\27\3\3\1\6", 5) = 5 read(6, "?O\3160\265\274\232\326\362Q\375\21\360\244\222\234\27\2532}\263p\312\37Y\326\312;\314\35\37e"..., 262) = 262 setsockopt(6, SOL_TCP, TCP_CORK, [1], 4) = 0 futex(0x556cb9b062a0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 futex(0x556cb9b062a0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 futex(0x556cb9b062a0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 futex(0x556cb9b062a0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 futex(0x556cb9b062a0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 futex(0x556cb9b062a0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 futex(0x556cb9b062a0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 futex(0x556cb9b062a0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 futex(0x556cb9b062a0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0

SpencerMKSmith commented 6 years ago

@lunnh How did you downgrade? When I checkout that commit and make install I get errors.

@ctubio This issue is reproducible as many of us have encountered it, please consider changing the tag on the issue.

@YokiWorks Looks like a SQLite issue to me. @donaldforest mentioned he was able to get it running by playing with some of the permissions to /data/db, but I was unable to get it to work in that way.

lunnh commented 6 years ago

@SpencerMKSmith I started noticing the bot hangs since some updates last week. Tried debug mode but found nothing. Thought it's my machine's own problem so I tried downgrading to older commits. The binary files for that commit was still available at that moment.

esomore commented 6 years ago

@lunnh can you shoot me this binary pls? I would love to switch it in place the current one to see if it takes the problem away

ctubio commented 6 years ago

on Debian im not able to reproduce this yet; but this issue was opened 5 days before the last sqlite change (https://github.com/ctubio/Krypto-trading-bot/commit/1828232796098b358c8f9b944f6df6b74f917799#diff-533b8e08b8f13c06e18b929d8270fe24) so im afraid rolling it back will not help

can you share your $ uname -a? or better a link to an image of your very same distro? cos i would like to be able to reproduce it :dancer:

also can you please confirm that $ make install was executed with the same user as the bot is executed? (that should be a non-root user but with sudoers permissions?)

lunnh commented 6 years ago

@YokiWorks Here you go. build-x86_64-linux-gnu-610abb6.tar.gz

gladstonedigital commented 6 years ago

I'm having the same issue on Debian with gdax. It runs fine for the first few minutes, places a few trades, then the web UI stops updating (I can still click things but there's no new market data coming in) and even Ctrl+C on the command line doesn't kill the program. It says some stuff about chuck norris (???) but doesn't quit until I Ctrl+\ to kill it.

I'm currently trying out the old binary posted by @lunnh and so far no problems, but that doesn't mean much since it hasn't been very long yet.

ctubio commented 6 years ago

im not sure that the problem here was fixed even if latest commit improved bitfinex, feel free to open a new issue just with a link to this if you wanna still improve bitfinex integration (or other exchange integration)

adirta commented 6 years ago

Updated lastest version... On hitbtc with ubuntu, continue still freezing after a few minutes

donaldforest commented 6 years ago

Fixed with full reinstall last clean ubuntu 16 and install only necessary clean build-essential after apt update..