askmike
commented
6 years ago how many gekkos where you running (backtest + live gekkos)?
Closed AidasK closed 6 years ago
askmike
commented
6 years ago how many gekkos where you running (backtest + live gekkos)?
AidasK
commented
6 years ago 0 backtests, 1 live paper trader and 2 data importers
askmike
commented
6 years ago Reading up on this:
Also a common problem is something like that client side app crashing and leaving connections open and then opening new ones when it restarts. If this kind of thing happens a lot then you'll run out of connections. Or the app is just configured improperly and opens too many connections
Did you start and stop Gekko a bunch of time? Does the problem go away after restarting postgresql? In that case Gekko needs to better close connections.
AidasK
commented
6 years ago No, I have started gekko twice after starting postgres database. I don't think this is the case.
I have started gekko again with 1 data importer and 1 live paper trader. Now I am having 53 active connections and they are increasing.
SELECT * FROM pg_stat_activity;
datid | datname | pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start
| query_start | state_change | wait_event_type | wait_event | state | backend_xid | backend_xmin | query
-------+----------+------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------
+-------------------------------+-------------------------------+-----------------+------------+--------+-------------+--------------+------------------------------------------
------------------------
16385 | gekko | 2258 | 16384 | gekko | | 127.0.0.1 | | 58370 | 2017-12-14 05:34:16.684687+01 |
| 2017-12-14 05:52:46.426871+01 | 2017-12-14 05:52:46.426991+01 | | | idle | | |
+
| | | | | | | | | |
| | | | | | | | SELECT * from poloniex_candles_usdt_btc
+
| | | | | | | | | |
| | | | | | | | WHERE start <= 74757625499 AND start >=
74757565500 +
| | | | | | | | | |
| | | | | | | | ORDER BY start ASC
+
| | | | | | | | | |
| | | | | | | |
13323 | postgres | 1329 | 10 | postgres | psql | 127.0.0.1 | | 55464 | 2017-12-14 05:25:42.968288+01 | 2017-12-14 05:52:46.428405+01
| 2017-12-14 05:52:46.428405+01 | 2017-12-14 05:52:46.428406+01 | | | active | | 266429 | SELECT * FROM pg_stat_activity;
16385 | gekko | 1442 | 16384 | gekko | | 127.0.0.1 | | 55720 | 2017-12-14 05:26:39.768238+01 |
| 2017-12-14 05:52:43.424138+01 | 2017-12-14 05:52:43.425711+01 | | | idle | | |
+
| | | | | | | | | |
| | | | | | | | INSERT INTO poloniex_candles_usdt_btc
+
| | | | | | | | | |
| | | | | | | | (start, open, high,low, close, vwp, vol
ume, trades) +
| | | | | | | | | |
| | | | | | | | values($1, $2, $3, $4, $5, $6, $7, $8)
ON CONFLICT DO NOTHING;+
| | | | | | | | | |
| | | | | | | |
16385 | gekko | 1588 | 16384 | gekko | | 127.0.0.1 | | 56088 | 2017-12-14 05:27:35.009908+01 |
| 2017-12-14 05:27:35.011122+01 | 2017-12-14 05:27:35.011516+01 | | | idle | | | CREATE TABLE IF NOT EXISTS
+
.........................
16385 | gekko | 1694 | 16384 | gekko | | 127.0.0.1 | | 56454 | 2017-12-14 05:28:35.183628+01 |
| 2017-12-14 05:52:46.426928+01 | 2017-12-14 05:52:46.427038+01 | | | idle | | |
+
| | | | | | | | | |
| | | | | | | | SELECT * from poloniex_candles_usdt_btc
+
| | | | | | | | | |
| | | | | | | | WHERE start <= 187938865499 AND start >
= 187938805500 +
| | | | | | | | | |
| | | | | | | | ORDER BY start ASC
+
| | | | | | | | | |
| | | | | | | |
16385 | gekko | 1706 | 16384 | gekko | | 127.0.0.1 | | 56460 | 2017-12-14 05:28:35.932623+01 |
| 2017-12-14 05:52:46.428012+01 | 2017-12-14 05:52:46.428137+01 | | | idle | | |
+
| | | | | | | | | |
| | | | | | | | SELECT * from poloniex_candles_usdt_btc
+
| | | | | | | | | |
| | | | | | | | WHERE start <= 188203945499 AND start >
= 188203885500 +
| | | | | | | | | |
| | | | | | | | ORDER BY start ASC
+
| | | | | | | | | |
| | | | | | | |
16385 | gekko | 1742 | 16384 | gekko | | 127.0.0.1 | | 56604 | 2017-12-14 05:28:55.65362+01 |
| 2017-12-14 05:52:46.427007+01 | 2017-12-14 05:52:46.427123+01 | | | idle | | |
+
| | | | | | | | | |
| | | | | | | | SELECT * from poloniex_candles_usdt_btc
+
| | | | | | | | | |
.......
+
| | | | | | | | | |
| | | | | | | | WHERE start <= 3697405499 AND start >=
3697345500 +
| | | | | | | | | |
| | | | | | | | ORDER BY start ASC
+
| | | | | | | | | |
| | | | | | | |
16385 | gekko | 4462 | 16384 | gekko | | 127.0.0.1 | | 36258 | 2017-12-14 05:52:19.036362+01 |
| 2017-12-14 05:52:46.428269+01 | 2017-12-14 05:52:46.428389+01 | | | idle | | |
+
| | | | | | | | | |
| | | | | | | | SELECT * from poloniex_candles_usdt_btc
+
| | | | | | | | | |
| | | | | | | | WHERE start <= 2087005499 AND start >=
2086945500 +
| | | | | | | | | |
| | | | | | | | ORDER BY start ASC
+
| | | | | | | | | |
| | | | | | | |
16385 | gekko | 4465 | 16384 | gekko | | 127.0.0.1 | | 36264 | 2017-12-14 05:52:19.16764+01 |
| 2017-12-14 05:52:46.427276+01 | 2017-12-14 05:52:46.427388+01 | | | idle | | |
+
| | | | | | | | | |
| | | | | | | | SELECT * from poloniex_candles_usdt_btc
+
| | | | | | | | | |
| | | | | | | | WHERE start <= 2089285499 AND start >=
2089225500 +
| | | | | | | | | |
| | | | | | | | ORDER BY start ASC
+
| | | | | | | | | |
| | | | | | | |
(56 rows)
https://github.com/askmike/gekko/blob/stable/core/tools/dataStitcher.js#L47
This place might be the problem, but it is really hard to know what this code does. I think seedLocalData function is not always called which has this.reader.close(); at the end
askmike
commented
6 years ago This might definitely be the problem, however the data stitcher is only used once when Gekko tries to match live exchange data (the exchange provides data from 1 hour ago, we have data locally up to 30 minutes ago => we can stitch the 2 different sources into one stream of market data).
Though note that the stitcher is not used when using the UI to start live gekkos at the moment.
Could you let me know exactly what you did to go from 0 connections to 53?
It sounds like the only things you did where:
node gekko --ui)And nothing else, and the number of connections is still going up? Would you be able to restart Gekko and only run an import or a stratrunner? To isolate the problem?
AidasK
commented
6 years ago Ok, I will try. Now I have 293 active connections.
[root@CentOS-72-64-minimal ~]# ps aux | grep gekko
gekko 1251 0.0 0.0 1047816 30492 ? Sl 05:25 0:00 npm
gekko 1261 0.0 0.1 1252404 71232 ? Sl 05:25 0:03 node ./gekko --config config.js --ui
gekko 1431 4.3 0.1 1246228 82004 ? Rl 05:26 5:48 /usr/bin/node /home/gekko/gekko/core/workers/pipeline/child
postgres 1442 0.2 0.0 401080 26976 ? Ss 05:26 0:18 postgres: gekko gekko 127.0.0.1(55720) idle
gekko 1577 0.0 0.0 1201700 51980 ? Sl 05:27 0:04 /usr/bin/node /home/gekko/gekko/core/workers/pipeline/child
postgres 1588 0.0 0.0 401080 6648 ? Ss 05:27 0:00 postgres: gekko gekko 127.0.0.1(56088) idle
gekko 1589 0.0 0.0 1065532 31164 ? Sl 05:27 0:00 /usr/bin/node /home/gekko/gekko/core/workers/pipeline/child
postgres 1600 0.0 0.0 401172 6428 ? Ss 05:27 0:00 postgres: gekko gekko 127.0.0.1(56098) idle
gekko 1683 7.9 0.1 1087172 70936 ? Rl 05:28 10:33 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 1694 6.1 0.0 401180 5816 ? Ss 05:28 8:13 postgres: gekko gekko 127.0.0.1(56454) idle
gekko 1695 7.9 0.1 1088024 69800 ? Rl 05:28 10:33 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 1706 6.1 0.0 401180 5816 ? Ss 05:28 8:10 postgres: gekko gekko 127.0.0.1(56460) idle
gekko 1730 7.9 0.0 1080344 64928 ? Rl 05:28 10:28 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 1742 6.0 0.0 401180 5816 ? Ss 05:28 8:05 postgres: gekko gekko 127.0.0.1(56604) idle
gekko 2110 6.2 0.1 1083408 66108 ? Rl 05:33 8:03 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2121 4.4 0.0 401180 5820 ? Ss 05:33 5:43 postgres: gekko gekko 127.0.0.1(57984) idle
gekko 2141 6.1 0.0 1079832 62696 ? Rl 05:33 7:51 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2153 4.3 0.0 401184 5808 ? Ss 05:33 5:34 postgres: gekko gekko 127.0.0.1(58100) idle
gekko 2171 6.0 0.1 1093144 68192 ? Rl 05:33 7:44 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2182 4.2 0.0 401184 5808 ? Ss 05:33 5:24 postgres: gekko gekko 127.0.0.1(58204) idle
gekko 2235 5.8 0.1 1027100 68320 ? Rl 05:34 7:29 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
gekko 2241 5.8 0.1 1018908 68568 ? Rl 05:34 7:27 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2257 4.1 0.0 401184 5808 ? Ss 05:34 5:14 postgres: gekko gekko 127.0.0.1(58368) idle
postgres 2258 4.0 0.0 401184 5816 ? Ss 05:34 5:07 postgres: gekko gekko 127.0.0.1(58370) idle
gekko 2278 5.7 0.0 1017880 65420 ? Rl 05:34 7:18 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2290 3.9 0.0 401184 5808 ? Ss 05:34 5:01 postgres: gekko gekko 127.0.0.1(58486) idle
gekko 2358 5.6 0.1 1017876 67260 ? Rl 05:35 7:05 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2369 3.8 0.0 401184 5808 ? Ss 05:35 4:49 postgres: gekko gekko 127.0.0.1(58726) idle
gekko 2393 5.4 0.0 1078808 63692 ? Rl 05:35 6:54 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2404 3.6 0.0 401184 5820 ? Ss 05:35 4:38 postgres: gekko gekko 127.0.0.1(58864) idle
gekko 2426 5.4 0.0 1016344 64320 ? Rl 05:35 6:48 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2437 3.6 0.0 401184 5820 ? Ss 05:35 4:31 postgres: gekko gekko 127.0.0.1(58978) idle
gekko 2482 5.2 0.1 1085468 69304 ? Rl 05:36 6:36 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
gekko 2487 5.3 0.1 1017368 67336 ? Rl 05:36 6:38 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2501 3.5 0.0 401184 5816 ? Ss 05:36 4:24 postgres: gekko gekko 127.0.0.1(59114) idle
postgres 2507 3.5 0.0 401184 5816 ? Ss 05:36 4:23 postgres: gekko gekko 127.0.0.1(59118) idle
gekko 2595 5.1 0.0 1026572 64972 ? Rl 05:37 6:20 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2607 3.3 0.0 401184 5812 ? Ss 05:37 4:09 postgres: gekko gekko 127.0.0.1(59454) idle
gekko 2610 5.0 0.0 1013788 64564 ? Rl 05:37 6:17 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2621 3.3 0.0 401184 5820 ? Ss 05:37 4:06 postgres: gekko gekko 127.0.0.1(59482) idle
gekko 2708 4.9 0.0 1013260 61452 ? Rl 05:38 6:04 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
gekko 2713 4.8 0.0 1013784 63752 ? Rl 05:38 6:00 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2730 3.1 0.0 401184 5816 ? Ss 05:38 3:53 postgres: gekko gekko 127.0.0.1(59832) idle
postgres 2731 3.1 0.0 401184 5820 ? Ss 05:38 3:56 postgres: gekko gekko 127.0.0.1(59834) idle
gekko 2821 4.7 0.0 1017368 64632 ? Rl 05:39 5:50 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
gekko 2827 4.7 0.0 1014296 64108 ? Rl 05:39 5:46 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2844 3.0 0.0 401184 5820 ? Ss 05:39 3:47 postgres: gekko gekko 127.0.0.1(60202) idle
postgres 2845 3.0 0.0 401184 5820 ? Ss 05:39 3:44 postgres: gekko gekko 127.0.0.1(60204) idle
gekko 2923 4.5 0.1 1012764 71600 ? Rl 05:40 5:31 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
gekko 2928 4.5 0.0 1013788 61660 ? Rl 05:40 5:34 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2947 2.9 0.0 401184 5820 ? Ss 05:40 3:35 postgres: gekko gekko 127.0.0.1(60502) idle
postgres 2949 2.9 0.0 401184 5824 ? Ss 05:40 3:35 postgres: gekko gekko 127.0.0.1(60508) idle
gekko 2973 4.5 0.0 1012760 65508 ? Rl 05:40 5:28 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 2985 2.9 0.0 401184 5820 ? Ss 05:40 3:33 postgres: gekko gekko 127.0.0.1(60630) idle
gekko 3004 4.4 0.0 1012236 62536 ? Rl 05:40 5:23 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 3015 2.8 0.0 401184 5824 ? Ss 05:40 3:28 postgres: gekko gekko 127.0.0.1(60726) idle
gekko 3056 4.4 0.1 1016344 66660 ? Rl 05:41 5:23 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
gekko 3061 4.4 0.1 1017880 67584 ? Rl 05:41 5:21 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
... more lines like this ...Ah this is great for debugging, thanks. Will look into this ASAP!
On Thu, Dec 14, 2017 at 2:44 PM, Aidas Klimas notifications@github.com wrote:
Ok, I will try. Now I have 293 active connections.
[root@CentOS-72-64-minimal ~]# ps aux | grep gekko gekko 1251 0.0 0.0 1047816 30492 ? Sl 05:25 0:00 npm gekko 1261 0.0 0.1 1252404 71232 ? Sl 05:25 0:03 node ./gekko --config config.js --ui gekko 1431 4.3 0.1 1246228 82004 ? Rl 05:26 5:48 /usr/bin/node /home/gekko/gekko/core/workers/pipeline/child postgres 1442 0.2 0.0 401080 26976 ? Ss 05:26 0:18 postgres: gekko gekko 127.0.0.1(55720) idle gekko 1577 0.0 0.0 1201700 51980 ? Sl 05:27 0:04 /usr/bin/node /home/gekko/gekko/core/workers/pipeline/child postgres 1588 0.0 0.0 401080 6648 ? Ss 05:27 0:00 postgres: gekko gekko 127.0.0.1(56088) idle gekko 1589 0.0 0.0 1065532 31164 ? Sl 05:27 0:00 /usr/bin/node /home/gekko/gekko/core/workers/pipeline/child postgres 1600 0.0 0.0 401172 6428 ? Ss 05:27 0:00 postgres: gekko gekko 127.0.0.1(56098) idle gekko 1683 7.9 0.1 1087172 70936 ? Rl 05:28 10:33 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 1694 6.1 0.0 401180 5816 ? Ss 05:28 8:13 postgres: gekko gekko 127.0.0.1(56454) idle gekko 1695 7.9 0.1 1088024 69800 ? Rl 05:28 10:33 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 1706 6.1 0.0 401180 5816 ? Ss 05:28 8:10 postgres: gekko gekko 127.0.0.1(56460) idle gekko 1730 7.9 0.0 1080344 64928 ? Rl 05:28 10:28 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 1742 6.0 0.0 401180 5816 ? Ss 05:28 8:05 postgres: gekko gekko 127.0.0.1(56604) idle gekko 2110 6.2 0.1 1083408 66108 ? Rl 05:33 8:03 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2121 4.4 0.0 401180 5820 ? Ss 05:33 5:43 postgres: gekko gekko 127.0.0.1(57984) idle gekko 2141 6.1 0.0 1079832 62696 ? Rl 05:33 7:51 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2153 4.3 0.0 401184 5808 ? Ss 05:33 5:34 postgres: gekko gekko 127.0.0.1(58100) idle gekko 2171 6.0 0.1 1093144 68192 ? Rl 05:33 7:44 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2182 4.2 0.0 401184 5808 ? Ss 05:33 5:24 postgres: gekko gekko 127.0.0.1(58204) idle gekko 2235 5.8 0.1 1027100 68320 ? Rl 05:34 7:29 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child gekko 2241 5.8 0.1 1018908 68568 ? Rl 05:34 7:27 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2257 4.1 0.0 401184 5808 ? Ss 05:34 5:14 postgres: gekko gekko 127.0.0.1(58368) idle postgres 2258 4.0 0.0 401184 5816 ? Ss 05:34 5:07 postgres: gekko gekko 127.0.0.1(58370) idle gekko 2278 5.7 0.0 1017880 65420 ? Rl 05:34 7:18 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2290 3.9 0.0 401184 5808 ? Ss 05:34 5:01 postgres: gekko gekko 127.0.0.1(58486) idle gekko 2358 5.6 0.1 1017876 67260 ? Rl 05:35 7:05 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2369 3.8 0.0 401184 5808 ? Ss 05:35 4:49 postgres: gekko gekko 127.0.0.1(58726) idle gekko 2393 5.4 0.0 1078808 63692 ? Rl 05:35 6:54 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2404 3.6 0.0 401184 5820 ? Ss 05:35 4:38 postgres: gekko gekko 127.0.0.1(58864) idle gekko 2426 5.4 0.0 1016344 64320 ? Rl 05:35 6:48 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2437 3.6 0.0 401184 5820 ? Ss 05:35 4:31 postgres: gekko gekko 127.0.0.1(58978) idle gekko 2482 5.2 0.1 1085468 69304 ? Rl 05:36 6:36 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child gekko 2487 5.3 0.1 1017368 67336 ? Rl 05:36 6:38 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2501 3.5 0.0 401184 5816 ? Ss 05:36 4:24 postgres: gekko gekko 127.0.0.1(59114) idle postgres 2507 3.5 0.0 401184 5816 ? Ss 05:36 4:23 postgres: gekko gekko 127.0.0.1(59118) idle gekko 2595 5.1 0.0 1026572 64972 ? Rl 05:37 6:20 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2607 3.3 0.0 401184 5812 ? Ss 05:37 4:09 postgres: gekko gekko 127.0.0.1(59454) idle gekko 2610 5.0 0.0 1013788 64564 ? Rl 05:37 6:17 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2621 3.3 0.0 401184 5820 ? Ss 05:37 4:06 postgres: gekko gekko 127.0.0.1(59482) idle gekko 2708 4.9 0.0 1013260 61452 ? Rl 05:38 6:04 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child gekko 2713 4.8 0.0 1013784 63752 ? Rl 05:38 6:00 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2730 3.1 0.0 401184 5816 ? Ss 05:38 3:53 postgres: gekko gekko 127.0.0.1(59832) idle postgres 2731 3.1 0.0 401184 5820 ? Ss 05:38 3:56 postgres: gekko gekko 127.0.0.1(59834) idle gekko 2821 4.7 0.0 1017368 64632 ? Rl 05:39 5:50 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child gekko 2827 4.7 0.0 1014296 64108 ? Rl 05:39 5:46 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2844 3.0 0.0 401184 5820 ? Ss 05:39 3:47 postgres: gekko gekko 127.0.0.1(60202) idle postgres 2845 3.0 0.0 401184 5820 ? Ss 05:39 3:44 postgres: gekko gekko 127.0.0.1(60204) idle gekko 2923 4.5 0.1 1012764 71600 ? Rl 05:40 5:31 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child gekko 2928 4.5 0.0 1013788 61660 ? Rl 05:40 5:34 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2947 2.9 0.0 401184 5820 ? Ss 05:40 3:35 postgres: gekko gekko 127.0.0.1(60502) idle postgres 2949 2.9 0.0 401184 5824 ? Ss 05:40 3:35 postgres: gekko gekko 127.0.0.1(60508) idle gekko 2973 4.5 0.0 1012760 65508 ? Rl 05:40 5:28 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 2985 2.9 0.0 401184 5820 ? Ss 05:40 3:33 postgres: gekko gekko 127.0.0.1(60630) idle gekko 3004 4.4 0.0 1012236 62536 ? Rl 05:40 5:23 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child postgres 3015 2.8 0.0 401184 5824 ? Ss 05:40 3:28 postgres: gekko gekko 127.0.0.1(60726) idle gekko 3056 4.4 0.1 1016344 66660 ? Rl 05:41 5:23 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child gekko 3061 4.4 0.1 1017880 67584 ? Rl 05:41 5:21 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child ... more lines like this ...
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/askmike/gekko/issues/1476#issuecomment-351623163, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7MD4KGXYr7A_2QOx4VeiAaFtFBt9zcks5tAMPEgaJpZM4RBhPs .
-- PGP key at keybase.io/mikevanrossum https://keybase.io/mikevanrossum/key.asc
AidasK
commented
6 years ago $ systemctl restart postgresql-9.6 $ npm start ui is working and I have no active connections
[gekko@CentOS-72-64-minimal gekko]$ ps aux | grep gekko
root 17837 0.0 0.0 187508 2272 pts/2 S 07:47 0:00 su gekko
gekko 17838 0.0 0.0 115396 2072 pts/2 S 07:47 0:00 bash
gekko 17882 0.1 0.0 1047816 28456 pts/2 Sl 07:48 0:00 npm
gekko 17914 0.8 0.0 1180720 63324 pts/2 Sl 07:48 0:00 node ./gekko --config config.js --ui
gekko 18140 0.0 0.0 151040 1748 pts/2 R+ 07:49 0:00 ps aux
gekko 18141 0.0 0.0 112636 720 pts/2 S+ 07:49 0:00 grep --color=auto gekkoStart a new live Gekko! Poloniex papertrader, MACD, candle size 1 minute(default settings)
After 5 minutes, 2 active connections. Looks normal.
After 8 minutes I have clicked on "Market watchers" which was created. Graph is not loading, connections increased to 13.
In chrome console I can see that websocket is trying to reconnect, something is crashing. But I can't see any errors in the output of npm start
After 13 minutes, 13 active connections and:
[gekko@CentOS-72-64-minimal gekko]$ ps aux | grep gekko
root 17837 0.0 0.0 187508 2272 pts/2 S 07:47 0:00 su gekko
gekko 17838 0.0 0.0 115396 2072 pts/2 S 07:47 0:00 bash
gekko 17882 0.0 0.0 1047816 28456 pts/2 Sl 07:48 0:00 npm
gekko 17914 0.1 0.0 1246768 64756 pts/2 Sl 07:48 0:01 node ./gekko --config config.js --ui
gekko 18248 0.0 0.0 1195268 43740 pts/2 Sl 07:50 0:00 /usr/bin/node /home/gekko/gekko/core/workers/pipeline/child
postgres 18259 0.0 0.0 401080 4540 ? Ss 07:50 0:00 postgres: gekko gekko 127.0.0.1(43060) idle
gekko 18261 0.0 0.0 1056828 35660 pts/2 Sl 07:50 0:00 /usr/bin/node /home/gekko/gekko/core/workers/pipeline/child
postgres 18272 0.0 0.0 401080 5860 ? Ss 07:50 0:00 postgres: gekko gekko 127.0.0.1(43072) idle
gekko 18851 46.0 0.0 1083564 65276 pts/2 Rl 07:56 3:26 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 18862 44.1 0.0 401180 5828 ? Ss 07:56 3:17 postgres: gekko gekko 127.0.0.1(44874) idle
gekko 18886 44.0 0.1 1085120 66460 pts/2 Rl 07:56 3:09 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 18897 41.8 0.0 401180 5828 ? Ss 07:56 3:00 postgres: gekko gekko 127.0.0.1(44986) idle
gekko 18951 44.4 0.0 1082044 63268 pts/2 Sl 07:57 3:02 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
gekko 18956 44.4 0.0 1084600 63820 pts/2 Sl 07:57 3:02 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 18968 41.9 0.0 401180 5828 ? Ss 07:57 2:52 postgres: gekko gekko 127.0.0.1(45126) idle
postgres 18974 41.2 0.0 401180 5828 ? Ss 07:57 2:49 postgres: gekko gekko 127.0.0.1(45130) idle
gekko 19006 42.3 0.1 1086116 66388 pts/2 Rl 07:57 2:42 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 19017 39.1 0.0 401180 5832 ? Ss 07:57 2:30 postgres: gekko gekko 127.0.0.1(45250) idle
gekko 19033 43.4 0.0 1085464 63848 pts/2 Sl 07:57 2:41 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 19045 40.8 0.0 401184 5816 ? Ss 07:57 2:31 postgres: gekko gekko 127.0.0.1(45314) idle
gekko 19128 38.8 0.1 1086488 66872 pts/2 Sl 07:58 2:07 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 19139 35.1 0.0 401184 5820 ? Ss 07:58 1:55 postgres: gekko gekko 127.0.0.1(45592) idle
gekko 19143 43.4 0.0 1016856 64768 pts/2 Rl 07:58 2:20 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 19155 40.1 0.0 401184 5820 ? Rs 07:58 2:10 postgres: gekko gekko 127.0.0.1(45644) SELECT
gekko 19158 40.4 0.0 1016856 63992 pts/2 Rl 07:58 2:11 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 19170 36.5 0.0 401184 5828 ? Rs 07:58 1:58 postgres: gekko gekko 127.0.0.1(45662) SELECT
gekko 19180 39.7 0.0 1018392 64076 pts/2 Rl 07:58 2:03 /usr/bin/node /home/gekko/gekko/core/workers/loadCandles/child
postgres 19191 36.1 0.0 401184 5824 ? Ss 07:58 1:52 postgres: gekko gekko 127.0.0.1(45718) idle
gekko 19735 0.0 0.0 151040 1752 pts/2 R+ 08:04 0:00 ps aux
gekko 19736 0.0 0.0 112640 980 pts/2 S+ 08:04 0:00 grep --color=auto gekkohttps://****.com/api/getCandles returns 502 bad gateway nginx log:
2017/12/14 08:09:01 [error] 31079#31079: *1049465 upstream prematurely closed connection while reading response header from upstream, client: 96.40.18.73, server: gekko****.com, request: "POST /api/getCandles HTTP/1.1", upstream: "http://127.0.0.1:3000/api/getCandles", host: "gekko***.com", referrer: "https://gekko****.com/"It looks like that "/api/getCandles" api endpoint timedout.
askmike
commented
6 years ago I have enough information to debug and fix this, your main issue (will try to asap).
It looks like that "/api/getCandles" api endpoint timedout.
After postgresql not accepting new connections? Regardless do you have stdout (console messages) from the node gekko --ui process?
AidasK
commented
6 years ago
Gekko v0.5.11
I'm gonna make you rich, Bud Fox.
Serving Gekko UI on https://gekko***.com:80/
Something went wrong when trying to open your web browser. UI is running on https://gekko****.com:80/.
<-- GET /favicon.ico
--> GET /favicon.ico 404 1ms -
<-- GET /api/imports
--> GET /api/imports 200 1ms 2b
<-- GET /api/gekkos
--> GET /api/gekkos 200 0ms 2b
<-- GET /api/gekkos
--> GET /api/gekkos 200 0ms 2b
<-- GET /api/apiKeys
--> GET /api/apiKeys 200 0ms 2b
<-- GET /api/exchanges
--> GET /api/exchanges 200 492ms 61.04kb
<-- GET /api/gekkos
--> GET /api/gekkos 200 0ms 2b
<-- GET /api/gekkos
--> GET /api/gekkos 200 1ms 2b
<-- GET /api/imports
--> GET /api/imports 200 0ms 2b
<-- GET /api/apiKeys
--> GET /api/apiKeys 200 0ms 2b
<-- GET /api/exchanges
--> GET /api/exchanges 200 2ms 61.04kb
<-- POST /api/scansets
--> POST /api/scansets 200 637ms 304b
<-- GET /api/configPart/performanceAnalyzer
<-- GET /api/configPart/candleWriter
--> GET /api/configPart/performanceAnalyzer 200 1ms 144b
--> GET /api/configPart/candleWriter 200 1ms 47b
<-- GET /api/strategies
--> GET /api/strategies 200 4ms 2.14kb
<-- GET /api/configPart/paperTrader
--> GET /api/configPart/paperTrader 200 0ms 132b
<-- POST /api/startGekko
Gekko 129792347982484 started
--> POST /api/startGekko 200 6ms 148b
<-- POST /api/startGekko
Gekko 561349696396253 started
--> POST /api/startGekko 200 3ms 403b
<-- GET /api/imports
--> GET /api/imports 200 1ms 2b
<-- GET /api/gekkos
--> GET /api/gekkos 200 1ms 984b
<-- GET /api/gekkos
--> GET /api/gekkos 200 1ms 984b
<-- GET /api/apiKeys
--> GET /api/apiKeys 200 0ms 2b
<-- GET /api/exchanges
--> GET /api/exchanges 200 3ms 61.04kb
<-- GET /api/imports
--> GET /api/imports 200 0ms 2b
<-- GET /api/gekkos
--> GET /api/gekkos 200 0ms 984b
<-- GET /api/gekkos
--> GET /api/gekkos 200 1ms 984b
<-- GET /api/apiKeys
--> GET /api/apiKeys 200 1ms 2b
<-- GET /api/exchanges
--> GET /api/exchanges 200 2ms 61.04kb
<-- POST /api/getCandles
<-- POST /api/getCandles
<-- POST /api/getCandles
<-- POST /api/getCandles
<-- POST /api/getCandles
<-- POST /api/getCandles
<-- POST /api/getCandles
<-- GET /api/imports
--> GET /api/imports 200 0ms 2b
<-- GET /api/gekkos
--> GET /api/gekkos 200 0ms 984b
<-- GET /api/gekkos
--> GET /api/gekkos 200 0ms 984b
<-- GET /api/apiKeys
--> GET /api/apiKeys 200 1ms 2b
<-- GET /api/exchanges
--> GET /api/exchanges 200 3ms 61.04kb
<-- GET /api/imports
--> GET /api/imports 200 0ms 2b
<-- GET /api/gekkos
--> GET /api/gekkos 200 1ms 984b
<-- GET /api/apiKeys
--> GET /api/apiKeys 200 0ms 2b
<-- GET /api/exchanges
<-- GET /api/gekkos
--> GET /api/gekkos 200 0ms 984b
--> GET /api/exchanges 200 3ms 61.04kb
<-- POST /api/getCandles
<-- POST /api/getCandles
<-- POST /api/getCandles30 active connections as for now. Live gekko paper trader is running all good.
Issue found: When I click on market watcher, something crashes and it creates one more idle connection to postgres. "Market graph" loader never stops loading and graph never shows up. As I said before, I am getting https://****.com/api/getCandles returns 502 bad gateway
AidasK
commented
6 years ago Any update? Maybe there is a log file where I could look why "/api/getCandles" endpoint for market watcher has crashed?
askmike
commented
6 years ago postgres connections are now pooled
I'm submitting a ... [x ] bug report [ ] feature request [ ] question about the decisions made in the repository
Action taken (what you did) Running gekko on postgres for 8 hours with 2 data imports and 1 market watcher
Expected result (what you hoped would happen) It does not crash
Actual result (unexpected outcome) it crashes with error: { error: remaining connection slots are reserved for non-replication superuser connections at Connection.parseE (/home/gekko/gekko/node_modules/pg/lib/connection.js:546:11) at Connection.parseMessage (/home/gekko/gekko/node_modules/pg/lib/connection.js:371:19) at Socket. (/home/gekko/gekko/node_modules/pg/lib/connection.js:114:22)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at addChunk (_stream_readable.js:263:12)
at readableAddChunk (_stream_readable.js:250:11)
at Socket.Readable.push (_stream_readable.js:208:10)
at TCP.onread (net.js:594:20)
name: 'error',
length: 140,
severity: 'FATAL',
code: '53300',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'postinit.c',
line: '779',
routine: 'InitPostgres' } '\n\n'
Meta debug info:
Gekko version: v0.5.11 Nodejs version: v8.9.3
No other application is using postgres. max_connections in config file is 100