ctubio / Krypto-trading-bot

Self-hosted crypto trading bot (automated high frequency market making) written in C++
https://127.0.0.1:3000
Other
3.32k stars 819 forks source link

GDAX connecting to exchange... #812

Closed JamesBura closed 5 years ago

JamesBura commented 5 years ago

After the latest update, all I get now is connecting to exchange with the red border. I can see everything but I cant start K.

app exchange: COINBASE/LTC/USD
app version: undisclosed
OS distro: undisclosed

300px-spock_vulcan-salute3

ctubio commented 5 years ago

please consider showing some logs/sys specs; i can't imagine what may be wrong in your instance

ctubio commented 5 years ago

does your computer have the file /etc/ssl/certs/ca-certificates.crt? otherwise try:

sudo curl --time-cond /etc/ssl/certs/ca-certificates.crt https://curl.haxx.se/ca/cacert.pem -o /etc/ssl/certs/ca-certificates.crt

hope it helps'' (but i dont think so cos this command is already executed by make install)

JamesBura commented 5 years ago

K-trading-bot x86_64-linux-gnu 2018-09-17 19:19:34. v0.4.10+43 (0day). 09/17 16:31:10.980587 DB loaded OK from :memory:. 09/17 16:31:10.982164 DB loaded OK from /data/db/K.COINBASE.LTC.USD.db. 09/17 16:31:10.982537 UI Warrrrning: Connected web clients will enjoy unsecure SSL encryption.. (because the private key is visible in the source!), consider --ssl-crt and --ssl-key arguments. 09/17 16:31:10.985061 DB loaded last Quoting Parameters OK. 09/17 16:31:10.985912 DB Warrrrning: consider to warm up some EWMA Values. 09/17 16:31:11.180259 GW COINBASE allows client IP. 09/17 16:31:11.180745 GW COINBASE:

ctubio commented 5 years ago

thanks for the logs; saddly is not useful; does your computer have the file /etc/ssl/certs/ca-certificates.crt?

JamesBura commented 5 years ago

thanks for the logs; saddly is not useful; does your computer have the file /etc/ssl/certs/ca-certificates.crt?

I don't see that, I have /etc/sslcert/ but no .crt files.

ctubio commented 5 years ago

maybe you could try to donwload that file? with:

sudo curl https://curl.haxx.se/ca/cacert.pem -o /etc/ssl/certs/ca-certificates.crt

if doesnt helps, i think the issue is just about connectivity not about ssl certs in that case, maybe the error msgs is displayed also whiile calling manually the websocket from this computer?:

curl --include --no-buffer --header "Connection: Upgrade" --header "Upgrade: websocket" --header "Host: ws-feed.pro.coinbase.com" --header "Origin: https://ws-feed.pro.coinbase.com" --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" --header "Sec-WebSocket-Version: 13" https://ws-feed.pro.coinbase.com

the output without errors is:

HTTP/2 200
content-type: text/plain

Please connect via WebSocket

maybe you have some connectivity error instead of this?

but first of everythign make sure you have /etc/ssl/certs/ca-certificates.crt and that cat /etc/ssl/certs/ca-certificates.crt (without sudo) returns a long list of certs please; thanks''

JamesBura commented 5 years ago

Heres what I got

HTTP/2 200 date: Mon, 17 Sep 2018 20:58:43 GMT content-type: text/plain set-cookie: __cfduid=d5cc242536252c13230f057f6e09032881537217923; expires=Tue, 17-Sep-19 20:58:43 GMT; path=/; domain=.pro.coinbase.com; HttpOnly expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" server: cloudflare cf-ray: 45be7e14ed1ac1ae-IAD

Please connect via WebSocket

ctubio commented 5 years ago

did you downloaded the file? what is the output of ls -la /etc/ssl/certs/ca-certificates.crt?

JamesBura commented 5 years ago

did you downloaded the file? what is the output of ls -la /etc/ssl/certs/ca-certificates.crt?

-rw-r--r-- 1 root root 213947 Sep 17 17:11 /etc/ssl/certs/ca-certificates.crt

ctubio commented 5 years ago

thanks for confirm''

i have no idea then (that file is the only requirement of FIX api); can you let me know what is your operative system name and version? i can try with the very same :S (i mean output of $ uname -a)

JamesBura commented 5 years ago

$ uname -a

Linux New-James 4.4.0-17134-Microsoft #285-Microsoft Thu Aug 30 17:31:00 PST 2018 x86_64 GNU/Linux

JamesBura commented 5 years ago

I just ran a fresh install of K, same problem. It was working fine until one of the commits 3 days ago. How can I roll back to that?

JamesBura commented 5 years ago

Ctubio,

I ran the latest update, let it sit for a few minutes and it finally connected to coinbase. As of now, its working as it should. Thank you for everything.

JamesBura commented 5 years ago

Ctubio,

I ran the latest update, let it sit for a few minutes and it finally connected to coinbase. As of now, its working as it should. Thank you for everything.

I spoke too soon. Same issue, "trying to connect" with the red border on the UI. Here's what confuses me, when I shut K down, it cancels my open orders so it's somehow connected. I cant get it run again, I need help!

ctubio commented 5 years ago

:bulb: the "cancel all before exit" is the only exchange call that uses REST api; thats why it works always no matter what

ctubio commented 5 years ago

sorry can you confirm that you had executed make reinstall? just to discard that "fresh install" does not mean diff things for you and me; thanks''

JamesBura commented 5 years ago

Ive reinstalled Debian, and K, still will not work. I changed from LTC/USD to BTC/USD and thats when I got the three headed monkey error.

Everything except Windows has been reinstalled, no three headed monkey. Still cant get a connection to start trading....

ctubio commented 5 years ago

i have no idea, but if you reinstalled, the issue must be with your account, because K works coOl with coinbase i can tell (and all other users too)

maybe helps creating a new api key on coinbase? you can try it, and if works let me know the address displayed in the bot and i will unlock it, so you dont have to unlock it again

hope for it :S

JamesBura commented 5 years ago

New today, I just let it sit for a while and this started.

james@New-James:~/K$ ./K.sh K-trading-bot x86_64-linux-gnu 2018-10-08 02:10:30. v0.4.10+55 (0day). 10/08 14:36:36.327954 DB loaded OK from /data/db/K.COINBASE.LTC.USD.db. 10/08 14:36:36.328563 UI Warrrrning: Connected web clients will enjoy unsecure SSL encryption.. (because the private key is visible in the source!), consider --ssl-crt and --ssl-key arguments. 10/08 14:36:36.329017 UI ready at https://75.110.138.75:3000. 10/08 14:36:36.330751 DB Warrrrning: using default values for Quoting Parameters. 10/08 14:36:36.331286 DB loaded TBP = 17.795900 LTC. 10/08 14:36:36.352819 DB loaded 95 historical Fair Values. 10/08 14:36:36.353382 DB loaded last EWMA Values OK. 10/08 14:36:36.503056 GW COINBASE allows client IP. 10/08 14:36:36.503913 GW COINBASE:

ctubio commented 5 years ago

maybe you have multiple bots running in the same machine trying to use the same database?

that may be a reason of "SQLite error: unable to open database file"

consider killall K or ps axu | grep K

boogey100 commented 5 years ago

I'm not sure exactly what fixed it, but I did the following and FIX is connecting ok again.

The first bit was just to test that it was using the ca-cert file. I noticed some files owned by 1001 in the 1001 group, so the chown/chgrp is to clean these up.

adam@coinserver:/var/krypto$ sudo rm /etc/ssl/certs/ca-certificates.crt adam@coinserver:/var/krypto$ ./K.sh K-trading-bot x86_64-linux-gnu 2018-10-08 02:10:30. v0.4.10+55 (0day). .... 10/09 18:17:19.885868 GW Errrror: Unable to fetch data from COINBASE for symbol "ETH-EUR", possible error message: {"error":"CURL Error: Problem with the SSL CA cert (path? access rights?)"}. 10/09 18:17:19.885917 GW COINBASE Attempting to cancel all open orders, please wait.. 10/09 18:17:19.899979 GW COINBASE cancel all open orders OK. K exit code 0.

adam@coinserver:/var/krypto$ sudo chgrp root . -R adam@coinserver:/var/krypto$ sudo chown adam . -R

adam@coinserver:/var/krypto$ make reinstall test -d .git && ((test -n "git diff" && (echo && echo !!Local changes will be lost!! press CTRL-C to abort. && echo && sleep 5) || :) \ && git fetch && git merge FETCH_HEAD || (git reset FETCH_HEAD && git checkout .)) || curl https://raw.githubusercontent.com/ctubio/Krypto-trading-bot/master/Makefile > Makefile

!!Local changes will be lost!! press CTRL-C to abort.

Already up to date. rm -rf app ..... List of installed K binaries: -rwxr-xr-x 1 root staff 5.4M Oct 9 18:20 /usr/local/bin/K-hello-world -rwxr-xr-x 1 root staff 39M Oct 9 18:20 /usr/local/bin/K-trading-bot

..done! Please restart any running instance and also refresh the UI if is currently opened in your browser.

adam@coinserver:/var/krypto$ ./K.sh K-trading-bot x86_64-linux-gnu 2018-10-08 02:10:30. v0.4.10+55 (0day). ..... 10/09 18:20:23.957400 GW COINBASE WS Streaming book levels. 10/09 18:20:23.971972 GW COINBASE FIX success Logon. 10/09 18:20:24.515689 QE Warrrrning: Unable to calculate quote, missing market data.

JamesBura commented 5 years ago

i have no idea, but if you reinstalled, the issue must be with your account, because K works coOl with coinbase i can tell (and all other users too)

maybe helps creating a new api key on coinbase? you can try it, and if works let me know the address displayed in the bot and i will unlock it, so you dont have to unlock it again

hope for it :S

The new API did not work, same issue. Also I tried the above and it would not "reinstall". I had to delete the file and install from scratch, still no connection.

boogey100 commented 5 years ago

Can you try again :

sudo rm /etc/ssl/certs/ca-certificates.crt sudo chgrp root . -R sudo chown james . -R make reinstall

And if you get any errors, post them here

JamesBura commented 5 years ago

Can you try again :

sudo rm /etc/ssl/certs/ca-certificates.crt sudo chgrp root . -R sudo chown james . -R make reinstall

And if you get any errors, post them here

Thanks for replying, this is what I got last time I tired it, just tired it again on a fresh install of K.

james@New-James:~$ cd K james@New-James:~ /K$ sudo rm /etc/ssl/certs/ca-certificates.crt [sudo] password for james: james@New-James:~/K$ sudo chgrp root . -R james@New-James:~/K$ sudo chown james . -R james@New-James:~/K$ make reinstall test -d .git && ((test -n "git diff" && (echo && echo !!Local changes will be lost!! press CTRL-C to abort. && echo && sleep 5) || :) \ && git fetch && git merge FETCH_HEAD || (git reset FETCH_HEAD && git checkout .)) || curl https://raw.githubusercontent.com/ctubio/Krypto-trading-bot/master/Makefile > Makefile fatal: unable to access 'https://github.com/ctubio/Krypto-trading-bot.git/': Problem with the SSL CA cert (path? access rights?) fatal: ambiguous argument 'FETCH_HEAD': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git [...] -- [...]' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (77) error setting certificate verify locations: CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs Makefile:199: recipe for target 'reinstall' failed make: *** [reinstall] Error 77 james@New-James:~/K$

boogey100 commented 5 years ago

Ah ok. Try: sudo curl https://curl.haxx.se/ca/cacert.pem -o /etc/ssl/certs/ca-certificates.crt make reinstall

JamesBura commented 5 years ago

Still getting an error:

james@New-James:~ /K$ sudo curl https://curl.haxx.se/ca/cacert.pem -o /etc/ssl/certs/ca-certificates.crt [sudo] password for james: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (77) error setting certificate verify locations: CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs james@New-James:~/K$ make reinstall make: *** No rule to make target 'reinstall'. Stop.

boogey100 commented 5 years ago

Can you download https://curl.haxx.se/ca/cacert.pem in a browser cd to your download dir sudo mv ./caccert.pem /etc/ssl/certs/ca-certificates.crt

Then try make reinstall again in your K folder

JamesBura commented 5 years ago

Can you download https://curl.haxx.se/ca/cacert.pem in a browser cd to your download dir sudo mv ./caccert.pem /etc/ssl/certs/ca-certificates.crt

Then try make reinstall again in your K folder

I ended up doing a fresh install of K, the make file was blank for some reason. As for the ca-certificate, I reinstalled those and its back to the root permissions. Nothing has worked, I have no idea what is different now than before.

boogey100 commented 5 years ago

Have you definitely got a root certificate back? Does it show when you do

ls -la /etc/ssl/certs/ca-certificates.crt

JamesBura commented 5 years ago

ls -la /etc/ssl/certs/ca-certificates.crt

james@New-James:~$ ls -la /etc/ssl/certs/ca-certificates.crt -rw-r--r-- 1 root root 207436 Oct 11 14:22 /etc/ssl/certs/ca-certificates.crt

Im at a loss with what else I can do now.

james@New-James:~/K$ ./K.sh K-trading-bot x86_64-linux-gnu 2018-10-08 02:10:30. v0.4.10+55 (0day). 10/12 09:08:06.475192 DB loaded OK from :memory:. 10/12 09:08:06.476885 DB loaded OK from /data/db/K.COINBASE.BTC.USD.db. 10/12 09:08:06.477328 UI Warrrrning: Connected web clients will enjoy unsecure SSL encryption.. (because the private key is visible in the source!), consider --ssl-crt and --ssl-key arguments. 10/12 09:08:06.478698 UI ready at https://75.110.138.75:3000. 10/12 09:08:06.500138 DB Warrrrning: using default values for Quoting Parameters. 10/12 09:08:06.501152 DB Warrrrning: consider to warm up some EWMA Values. 10/12 09:08:06.632203 GW COINBASE allows client IP. 10/12 09:08:06.632693 GW COINBASE:

boogey100 commented 5 years ago

I just setup debian on a virtual machine, and got exactly what you are getting above. I did the following and all works.

cd ~/K sudo mv /etc/ssl/certs/ca-certificates.crt .
(move the ca certs file so that make reinstall can get a new one) make reinstall ./K.sh

I have no idea why this isn't working for you :/

JamesBura commented 5 years ago

Thank you boogey and Carles for assisting, I still havent had any luck. Ive tried everything listed here with no luck. Im running on windows 10 WSL, Ive tried Debian and Ubuntu installs, same result. K has worked before on both so I dont know what has changed. I ran an inplace windows install thinking it might help but it didnt. Any other ideas or things I can check?

One more thing, my wallet shows normally on the screen, I just start K. If I click the red button, it turns yellow and stays that way.

boogey100 commented 5 years ago

Are you using virtual box? Maybe I can give you the debian image I made.

boogey100 commented 5 years ago

Can you try this first: sudo curl --time-cond /etc/ssl/certs/ca-certificates.crt2 https://curl.haxx.se/ca/cacert.pem -o /etc/ssl/certs/ca-certificates.crt2 sudo mv /etc/ssl/certs/ca-certificates.crt2 /etc/ssl/certs/ca-certificates.crt

boogey100 commented 5 years ago

Hey @ctubio, it looks like the ssl cert isn't being updated, see below....

pi@coinpi1:/var/krypto $ sudo curl --time-cond /etc/ssl/certs/ca-certificates.crt https://curl.haxx.se/ca/cacert.pem -o /etc/ssl/certs/ca-certificates.crt2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 pi@coinpi1:/var/krypto $ ls -al /etc/ssl/certs/ca-certificates.crt -rw-r--r-- 1 root root 233394 Oct 21 20:45 /etc/ssl/certs/ca-certificates.crt pi@coinpi1:/var/krypto $ sudo curl --time-cond /etc/ssl/certs/ca-certificates.crt2 https://curl.haxx.se/ca/cacert.pem -o /etc/ssl/certs/ca-certificates.crt2 Warning: Illegal date format for -z, --time-cond (and not a file name). Warning: Disabling time condition. See curl_getdate(3) for valid date syntax. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 204k 100 204k 0 0 884k 0 --:--:-- --:--:-- --:--:-- 884k pi@coinpi1:/var/krypto $ ls -al /etc/ssl/certs/ca-certificates.crt -rw-r--r-- 1 root root 233394 Oct 21 20:45 /etc/ssl/certs/ca-certificates.crt pi@coinpi1:/var/krypto $ pi@coinpi1:/var/krypto $ ls -al /etc/ssl/certs/ca-certificates.* -rw-r--r-- 1 root root 233394 Oct 21 20:45 /etc/ssl/certs/ca-certificates.crt -rw-r--r-- 1 root root 209309 Oct 22 08:32 /etc/ssl/certs/ca-certificates.crt2

JamesBura commented 5 years ago

I tried virtual box but it kept telling me my API key was wrong so I gave up. I ended up reinstalling windows, resetting bios to factory and loading Ubuntu in wsl and starting all over again, guess what? Same issues, I couldn't believe I just wiped my OS. I deleted my API and tried 5 new APIs to get it to start, it won't connect. I haven't tried on another machine but at this point, it's not my OS or connection since I've tried on a mobile hotspot. I have no idea how I'm the only one here with this issue lol.

boogey100 commented 5 years ago

Did you try my last suggestion?

sudo curl --time-cond /etc/ssl/certs/ca-certificates.crt2 https://curl.haxx.se/ca/cacert.pem -o /etc/ssl/certs/ca-certificates.crt2 sudo mv /etc/ssl/certs/ca-certificates.crt2 /etc/ssl/certs/ca-certificates.crt

JamesBura commented 5 years ago

Did you try my last suggestion?

sudo curl --time-cond /etc/ssl/certs/ca-certificates.crt2 https://curl.haxx.se/ca/cacert.pem -o /etc/ssl/certs/ca-certificates.crt2 sudo mv /etc/ssl/certs/ca-certificates.crt2 /etc/ssl/certs/ca-certificates.crt

Sure did, no change. james@James-laptop:~/K$ ls COPYING K.sh LICENSE Makefile README.md app build-x86_64-linux-gnu doc etc src test james@James-laptop:~/K$ sudo curl --time-cond /etc/ssl/certs/ca-certificates.crt2 https://curl.haxx.se/ca/cacert.pem -o /etc/ssl/certs/ca-certificates.crt2 [sudo] password for james: Warning: Illegal date format for -z, --time-cond (and not a file name). Warning: Disabling time condition. See curl_getdate(3) for valid date syntax. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 204k 100 204k 0 0 414k 0 --:--:-- --:--:-- --:--:-- 416k james@James-laptop:~/K$ sudo mv /etc/ssl/certs/ca-certificates.crt2 /etc/ssl/certs/ca-certificates.crt james@James-laptop:~/K$

capture

boogey100 commented 5 years ago

Noooooooo. Do you have a static IP? Maybe I can log in and see if I can debug it?

JamesBura commented 5 years ago

So, I ran make and got this. The only error I can find.

james@James-laptop:~/K$ make Building hello-world.. make src KSRC=hello-world make[1]: Entering directory '/home/james/K' Building hello-world src x86_64-linux-gnu.. x86_64-linux-gnu-g++ (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

make Linux CHOST=x86_64-linux-gnu make[2]: Entering directory '/home/james/K' make[3]: Entering directory '/home/james/K' x86_64-linux-gnu-g++ -DNDEBUG -o build-x86_64-linux-gnu/local/bin/K-hello-world \ -DHAVE_STD_UNIQUE_PTR -DUWS_THREADSAFE \ -static-libstdc++ -static-libgcc -rdynamic \ src/hello-world/hello-world.cxx -pthread -std=c++17 -O3 -DK_BUILD='"x86_64-linux-gnu"' -DK_SOURCE='"K-hello-world"' -DK_0_GIT='"ce96f700a8f77bb04541710e23ed08024eaaed69"' -DK_STAMP='"2018-10-28 12:04:47"' -DK_0_DAY='"v0.4.10+69"' -I/home/james/K/src/include -Ibuild-x86_64-linux-gnu/local/include build-x86_64-linux-gnu/local/include/uWS/*.cpp build-x86_64-linux-gnu/local/lib/K-x86_64-linux-gnu.a build-x86_64-linux-gnu/local/lib/libquickfix.a build-x86_64-linux-gnu/local/lib/libsqlite3.a build-x86_64-linux-gnu/local/lib/libz.a build-x86_64-linux-gnu/local/lib/libcurl.a build-x86_64-linux-gnu/local/lib/libssl.a build-x86_64-linux-gnu/local/lib/libcrypto.a build-x86_64-linux-gnu/local/lib/libncurses.a build-x86_64-linux-gnu/local/lib/libcares.a -ldl x86_64-linux-gnu-g++: error: build-x86_64-linux-gnu/local/lib/K-x86_64-linux-gnu.a: No such file or directory Makefile:135: recipe for target 'Linux' failed make[3]: [Linux] Error 1 make[3]: Leaving directory '/home/james/K' Makefile:133: recipe for target 'Linux' failed make[2]: [Linux] Error 2 make[2]: Leaving directory '/home/james/K' Makefile:120: recipe for target 'src' failed make[1]: [src] Error 2 make[1]: Leaving directory '/home/james/K' Makefile:98: recipe for target 'hello-world' failed make: [hello-world] Error 2

boogey100 commented 5 years ago

I think make reinstall should download the /local/lib/K stuff, then make to compile

JamesBura commented 5 years ago

I finally installed ubuntu on a virtual box again and it works. I wasn't thrilled because its been working on windows ubuntu and debian for so long. Apparently something has changed on their end Im assuming.

ctubio commented 5 years ago

my god what an adventure, many thanks for your friendly discussions

im not sure if in the IRC channel we have an awesome support like @boogey100 did, but next time you have questions if you agree feel free to join IRC eaSy