Integrating Machine Learning AI instead Technical Analysis to Gekko

[dramida]

Predicting stock prices using Technical Analysis is a 16th century Japanese tech. Let's improve it by creating an Trading Advisor based on Machine Learning who can predict trends. Using sentiment analysis from Tweeter help also as a second step, as shown in github projects below. Short technical videos explaining how implement AI in crypto stock prediction:

https://youtu.be/G5Mx7yYdEhE https://youtu.be/SSu00IRRraY https://youtu.be/ftMq5ps503w

Here are a list of github projects who resolve the Machine Learning part.

https://github.com/Avhirup/Stock-Market-Prediction-Challenge

https://github.com/ciurana2016/predict_stock_py

http://machinelearningmastery.com/time-series-prediction-with-deep-learning-in-python-with-keras/

I can support the programmer who takes this challenge, financially, like an investment.

[askmike]

I have been thinking about stuff like this for a while, have done small experiments as well. I hope I will have some time in the summer to work on this (will most likely open source this).

Semi related to #767

[shanehull]

This might be relevant https://github.com/buckyroberts/Ella

[klappy]

Sentiment analysis is definitely a good fit here, and one that I had considered.

Another application of machine learning that I am more convinced of is using Genetic Algorithms to both choose the appropriate strategy and then fine tune settings of each strategy for each unique currency pair. We used this approach for tuning machine translation engine settings to produce better output.

The big value here is that it could be overlaid on top of any strategy and optimize ROI, including sentiment analysis.

Here's a few node based npm modules:

• https://www.npmjs.com/package/genetical
• https://github.com/subprotocol/genetic-js

[klappy]

Just saw the link to #767. Nice!

[musquash]

Hey all,

still after this many links I found an interesting paper from Stanford University. They say that they can predict the daily price change of bitcoin with an accuracy of 98.7% just with random tree forrest. Right now I don't now much about machine learning but i am very interessted into it. I hope I can contribute to this project but my programming skills lacks on many parts, too. So I have to learn a lot till I can help here.

Thanks a lot for this great open source project.

[calin-ciobanu]

Hi all, first of all nice job on gekko - really good stuff!

My take on the AI part is that a solid trading bot should take into consideration all 3 types of analysis. I don't agree that TA is so last century but it may not be enough on it's own, so a combination of:

1. technical analysis (maybe optimized through AI)
2. fundamental analysis (the stanford paper mentioned by @musquash )
3. sentiment analysis (mentioned a few times above)

Would be the best formula using all 3 types of analysis imo. And all 3 need AI to tie them up with historical data and try to make predictions (i.e. getting the twitter feed for the last 6 months, training a NN on the tweets vs price impact and try to predict for future tweets - or perhaps a classification algorithm).

However implementing this is non trivial and will require a lot of testing/tuning. I and some friends are considering working on some of these features so perhaps we can share our results if they are worthy.

@askmike I wonder what do you think would be the best way of building this sort of functionality? Plugins? Independent projects talking via API? Modules of some sort? I am (mildly) familiar with the codebase, still digging through it.

@dramida I like your enthusiasm - you are going for open source with that investment?

[thegamecat]

How effective has sentiment analysis been historically? Any evidence?

[calin-ciobanu]

afaik there are some attempts in this area but no idea about how effective. I figure it's worth a shot and see what comes out

[kurt-o-sys]

I'm starting to implement some self-learning, so it may be worth to try to integrate it with gekko. However, I'm not using js (it would be too hard to do so) and I do use the order book. The idea is that the system learns how to predict the (height of) next peak/drop within a time frame of x times the resolution. For example: if the resolution is 1 minute (I'd prefer to have it lower, but that seems not possible using Gekko right now?), than the system could learn to predict the next peak/drop in the next 12 minutes. Or any other value.

Would it be possible to expose order-candles - very similar to the 'trade'-candles - for the calculation of the metrics (in this case, a metric that 'predicts' the height of next peak/drop within a certain timeframe)?

[askmike]

@kurt-o-sys what do you mean "expose"? It's very easy to run a gekko to simply pipe candles into a database (in semi realtime). If you don't want to use JS you can use whatever you want and pull out of that database (or it's very simple to write a gekko plugin that writes to CSV or whatever you want).

---

Would it be possible to expose order-candles - very similar to the 'trade'-candles - for the calculation of the metrics

So you want candles exposed so you can process them yourself? Candles are limited to 1 minute but you can take some core components from Gekko that emit trades instead of candles (at this point you are only using func that you can do with ~20 lines of python).

[Gebrax]

@klappy i'm agree with you. Genetic algo is more fit to this case. We can see many pump and dump actions in crypto trading, therefore it's should be hard to predict. Genetic algo with objective to tune up better stratetegy is what we are looking for. To get in and out in the right time

[kurt-o-sys]

@askmike well, I guess I'm misunderstanding something, but I actually want to add indicators, which use trade candles and 'order candles' to calculate the indicator value. The only candles we have now, are the trades. I just want to have the added and removed orders within the ticker time frame as well. Anyway, I'll have to get another look into Gekko, I suppose.

[askmike]

@kurt-o-sys ah sorry I misunderstood what you meant:

When I (in comments & Gekko documentation) talk about candles, I mean a price tick, it's an x minute "summary" of what happened in a markets with regards to price, trades (amount + volume).

I just want to have the added and removed orders within the ticker time frame as well.

I am not sure what you mean by this.

[kurt-o-sys]

@askmike OK, let me clarify a little. I expect that Gekko receives 'events', which can be trade or order events. Adding a trade or order does have pretty much the same structure:

type - ADD - market - price (rate) - quantity

Removing an order would be more like:

type - DEL - market - quantity

The candles you're referring to, as mostly done, are, as you say, a summary of what happened in a market, being the summary of the market orders. You could make the same candles for orders: the min/max/open/close/weighted average/quantity. So, one can have OCHL-candles as a summary for the trades (usual market price candles), and one can have OCHL-candles as a summary for the new orders places. (And on top of that, one would need also the orders that are removed.)

Another interesting factor could even be the number of trades (or orders).

My experience right now is that the order candles, and their dynamics, can give a very accurate price prediction, without even using the trade candles. Combining both, together with a few other factors, makes the price prediction even better.

I assume, but still have to test properly, that a well-designed computation graph* are able to detect long-term trends and very short term spikes. I don't use any TA metrics, since these contain less information than 'the dynamics of trades and orders themselves'. The lower the ticker time (I'd prefer something in the range 1-12s **), the better the short term predictions.

Does that make any sense, or does it clarify why I like to have the 'order summary of the last tick'?

---

(*) combined neural network, using separate trajectories for orders and trades using memory cells (LSTM), combining them later and adding a few layers, each with their own memory cells. This allows the network to remember the previously calculated values of 'metrics', which can affect the new prediction (or not, depending on the LSTM parameters).

(**) I wouldn't save all the data, so I don't need a lot of storage on disk. On each tick, two things happen:

1. make new predictions and supply them as a metric to the trading advisor (however, I would need the 'order candles' and 'remove orders' events to calculate these metrics)
2. check if the predictions previously made are close enough. If not, do a new training step by changing the weights of the network - this is the self-learning part (there's no test/validation data: this wouldn't allow for self-learning pretty easily, and in case of market predictions, it makes more sense to allow the network to learn new patterns at any time)

[askmike]

@kurt-o-sys when you talk about orders, you are talking about the orderbook (people offering to trade)? In that case: that's out of scope for Gekko. Gekko right now fully ignores order book data to ease backtesting (and sync backtests with live simulations).

Adding a trade or order does have pretty much the same structure:

They have the same structure in that they both mutate the orderbook, but not in the sense they result in a trade (someone bought BTC vs someone elses USD).

My experience right now is that the order candles, and their dynamics, can give a very accurate price prediction, without even using the trade candles. Combining both, together with a few other factors, makes the price prediction even better.

Great! I agree that by watching the orderbook you get a picture better than watching historical trades, the hard thing here is getting this market data.

I assume, but still have to test properly, that a well-designed computation graph* are able to detect long-term trends and very short term spikes. I don't use any TA metrics, since these contain less information than 'the dynamics of trades and orders themselves'. The lower the ticker time (I'd prefer something in the range 1-12s **), the better the short term predictions.

Does that make any sense, or does it clarify why I like to have the 'order summary of the last tick'?

Yes that makes perfect sense! Though I don't think Gekko is the tool you are looking for, since it is designed to do exactly the opposite:

Gekko is designed to be a simple tool that aggregates all historical trades into trade candles (as defined here) and allows you to build TA strategies on top of this data. What you want is something completely different (you want more granual data derived from a different data source (the orderbook), and then you don't want to do anything with TA. I agree that this is a better approach for AI/ML, but it goes directly in the feature set and scope of Gekko. Please read this document:

https://gekko.wizb.it/docs/introduction/scope.html (especially the section on market data).

---

So in order to do what you want, what I'd suggest what we do (I think you already started in the previous comments) is:

• define data (+gruanality) needed to train a model.
  • see how to fetch in realtime
  • see how to get historical data (I guess you want to train a model on months of data without waiting for months) <- this is the hard part if you want to use OB data.
• compare to what gekko is doing and figure out how/if we can integrate.

[kurt-o-sys]

@askmike Allright... too bad. Just to give you an idea of what I'm talking about:

The predictions of next heights/dips, in 4 different time frames (4 different computation graphs), plotted to real data.

resolution ticker: 1 minute

~38 times the resolution - black ~144 times the resolution - green ~ 500 times the resolution - blue ~ 1728 times the resolution - orange

straight lines: max peak within time frame dashed lines: min dip within time frame

It's somewhat optimized for the dataset itself (which will be the case anyway to some extend in a self-learning system), so in a real situation, I expect it to behave worse.

Anyway, I really like the idea and design concept of Gekko a lot. I don't really want the opposite, but more like an extension: adding a metric, but based on orders (which results in the order book) and actual trades. However, since it's out of scope of Gekko, I'll have to build something else around the prediction model :).

[bitcoinvsalts]

I am working on integrating Machine Learning AI instead Technical Analysis to Gekko. Contact me if you want to help out at contact@jsapp.me

[logicappsource]

I want to join in , and doing my bachelor project in software . To this project , hit me up @jsappme @askmike ..¨ Mail : logicappsource@gmail.com

[Hagalas]

It would be great to integrate some machine learning. @thegamecat I don't really get the point of putting genetic algo into gekko 'cause it uses historical data only to select the best parameters for your strategy. I know results can be impressive, especially when you go with the uptrend but does it really that good to play with real money? Wouldn't be better to stick with some neural network instead.

Anyone heard of ANN strategy from tradingview?

[generalectric]

@oskarm91, I dont really see where machine learning can be integrated into gekko? And I'm not sure there are plans to integrate GA? Integrating either of them just seems beyond the scope of gekko.

As far as GA vs ML vs TA... IMO, GA isn't a replacement for ML, and ML isn't a replacement for TA... but rather all tools that can be used in conjunction to achieve desired results.

[FLYBYME]

Just been playing around with this idea over the weekend.

Im using https://wagenaartje.github.io/neataptic for the neural network

The charts are from bitfinex NEOUSD at 15m/3d time frame

I feeding it bbands, macd, cci, dema and candlestick information.

This is really something ton look into!

training-data

first train iteration 1000 error 0.025380668695955993 rate 0.01

first evolve iteration 1000 fitness -0.028091147094537194 error 0.021191147094537195

4th evolve iteration 1000x4 fitness -0.02543513253507006 error 0.01863513253507006

train iteration 50000 error 0.010022455848223871 rate 0.01

[bitcoinvsalts]

@FLYBYME very nice thanks for sharing, please contact me at contact@jsapp.me

[logicappsource]

@FLYBYME Really cool awesome work . Can u also get in contact with me logicappsource@gmail.com

[salimelakoui]

Very intrested in this subject.

I am currently testing an IA (recurrent Neuronal Network) based on the strategy platforme of Gekko.

On my point of view, we would need more data and a way of handling it within gekko.

• RSS / News / Tweets ?
• PNL ?
• nosql storage

[John1231983]

@FLYBYME Great. Could you contact me via john1231983@gmail.com

[FLYBYME]

@salimelakoui what kind of input data are you using for your network? Do you have a training set that you train the network on?

[augustoclaro]

@FLYBYME Would you share your code or explain to us how you come to those results? I'm very interested

[denigada]

@FLYBYME I am working on a bot to catch pump and dumps and profit mad amounts daily. I couldn't find a contact info. I am thinking of merging my bot with AI trading. Please contact me: deniz@denizyazar.com

[salimelakoui]

Hello,

Inputs are the occurence of indicator's signals (UP DOWN) taken by N candles. For exemple :

• RSI-UP and DEMA-UP for day 1
• RSI-UP and DEMA-UP and MACD-UP for day 2 => It make RSI-UP,DEMA-UP#RSI-UP,DEMA-UP,MACD-UP as an input for 2 candles. Hence, I calculate all the possible comnbinaisons for that matters and apply BP considering the % of evolution between the close values.

That is the first net. The second one (which is not a network) would just identify the best moment to buy considering top and downs between two dates versus the result of the first network

• Eviction of certain values that make non sense
• Calculating result note of first net considering "trust".

I am still stucked

• on identifying "bad moments" where the value is behaving randomly...
• sinification of volumes / trades count ?

Any idea ?

Elakoui Directeur Technique 0628730009 salim.elakoui@nway.org

Paris France [image: Facebook] https://facebook.com/salim.elakoui [image: Twitter] https://twitter.com/salimelakoui [image: LinkedIn] https://linkedin.com/in/salimelakoui [image: Google+] https://plus.google.com/+SalimElakoui [image: YouTube] https://youtube.com/channel/UCFsNUh_6pp3bLXWMBNbiltQ [image: Skype] [image: Flickr] https://www.flickr.com/photos/salimelakouimartin

On Thu, Nov 9, 2017 at 5:19 PM, Tim notifications@github.com wrote:

@salimelakoui https://github.com/salimelakoui what kind of input data are you using for your network? Do you have a training set that you train the network on?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/askmike/gekko/issues/789#issuecomment-343207212, or mute the thread https://github.com/notifications/unsubscribe-auth/ATRgT7b6ofsaPMKqus6xvJBXb_zjE9p1ks5s0yYmgaJpZM4N9gh3 .

[Filoz]

Hi, have you ever try with https://github.com/gekkowarez/gekkoga ? I'm new and I was googling and documenting about gekko and I found it

[marcosaccioly]

Anyone actually developing this for gekko and planning on integrating it as open source? I'm willing to help. I currently have gekko installed on digitalocean running tests but no great strategy.

Hi, have you ever try with https://github.com/gekkowarez/gekkoga ?

@Filoz have you actually put the GA to work?

If anyone is willing to talk / develop / get to good solutions, I'm here.

[erasmuswill]

I'm willing to help too :) I looked at gekkoga briefly but I didn't really get the point of it it seems. I'll look at it again tomorrow morning.

[tmvnk]

I have a lot of crypto twitter historic data available via a rest-api . Contact me if you would like to use this.

[HOllarves]

I was able to use Japonicus. It's a GA implementation for Gekko as well. Running my first tests atm. Also looking for a ML implementation for gekko, willing to help any good idea out there.

[thefieldprojects]

@FLYBYME Any luck pushing that dev forward?

[okonon]

@FLYBYME that is really cool! Can you share more info?

[costaeder]

Hello, people!

I was thinking a few days ago on a strategy to improve the accuracy of gekko. Strategies based purely on graphics, in a market like crypto, will inevitably fail because people are unpredictable.

Unlike the stock market, there is a lot of information about the market for crypto-coins in social networks, especially in high / low moments.

The strategy would first examine the following sources: 1 - The comments on the networks (all of them: telegram, facebook, reddit etc). Collect information and rank, in terms of volume, and opinion (neutral, positive, negative). 2 - The news on official sources: NWT, WallStreet, and major newspapers from major markets. 3 - The volume of the currency market.

As written above, the use of neural networks and genetic algorithms is very important to choose the best strategy for that moment, since, given the market volatility, making a strategy based on the past data does not say NOTHING about the future. One strategy can be good at times of purchase, another at times of sale, another when the market is "walking sideways."

This is just a little of what I've been thinking these days. What do you think about it?

Sorry my poor english.

[lee101]

Shameless plug: I'm founder of https://BitBank.nz a machine learning cryptocurrency forecasting/trading startup, we collect features in the market like orderbook imbalances, weighted slopes in the trades, and use machine learning techniques on these features to come up with a forecast of what the future weighted averages should be in the trades. (where your trade will probably land).

We provide high frequency predictions of the weighted trade averages in the next 5 min 30 min 60 min and 2 hours, Note that the further you try to predict in the future the less accurate you are, just a symptom of compounding volatility and unpredictability, don't trust 7+ day forecasts...

There are still a lot of strategies you can come up with when given the forecast and confidence in it but its a great starting point and lets you skip over doing some of the AI heavy lifting.

I integrated our forecasts via our API to a sample market maker on BitMex https://github.com/BitBanknz/sample-market-maker and i should be able to find some time to create a Gekko strategy based on our forecasts too.

Heres a video intro to our platform https://www.youtube.com/watch?v=9httSKXpT1w Let me know if your from gekko/open source and ill throw you some free time on the platform :), we also have a referral program where you can earn .003 btc per paying user referred.

[dramida]

Lee Penkman thank you for your contribution, i am watching your youtube clip right now. It's a great idea and it revives gekko as a automated platform, thanks alot! I will try it asap. I will get into details later but at the first glance we need more explanation of: -the input data you use and the general AI algoritm you use -the output parameters and their meaning, -markets you work on, -an example on how to use your data, in a better English or with captions. Auto captions does not work at all, sorry.

A good ideea would be to display the results in order of their probability that will happen. Keep up the good work and i hope other programmers from Gekko will join your effort.

[ansonphong]

First of all, this is awesome, I very much love the idea of integrate more sophisticated data sets in addition to basic TA to see the bigger picture. TA is great for reading current trends and making trades on a candle scale, though in terms of predicting growth over longer periods, I think AI is required.

To keep on the radar here, there is a forecasting tool called Neurobot which shows on their website that they're developing an API for integrating into third party apps. So far I have found their predictions to not always be accurate by any means, though I use them as a reference before making mid term trades and it's proved itself to be worthwhile. http://neurobot.trading/

The simplest implementation I would use it for is somehow being able to actually re-configure trading pairs on the fly and genetically test all my strategies against the history of the cryptos that it predicts the highest growth in, then deploy the trader automatically. That level of AI would automate the day to day task of configuring Gekko for each pair, and deploy based on current market conditions to the pair which it predicts the highest growth on.

[dramida]

Hello Guys, i checked the neural network output API from Bitbank.nz and we stroke gold for our old lizard : Can you integrate such data into Gekko? https://bitbank.nz/api/forecasts/usdt_btc

If you cannot access the link, it might be because you are not signed in...it's free for a day :) Yeee!

[JasLin]

@FLYBYME was it overfitting?

[dramida]

If you try to develop AI for crypto stock prediction, you are hurting yourself if you don't see Siraj Raval videos regarding Deep Neural Network and crypto price prediction:

https://youtu.be/G5Mx7yYdEhE

[salimelakoui]

I would like to share some code with you guys.

The main concept is to identify "normal moment" and "random moments" in order to invest.

Who would be intrested ?

Send your emails at salim.elakoui@nway.org

[dumbanddumbererest]

https://github.com/cazala/synaptic is a nodejs project, actively developed and has recurrent neural networks implemented.

while we're at it - can we have balances and the ability to proportion each trade? With that done we could have some version of the kelly formula working in the background ensuring balances always grow.

[salimelakoui]

It would be great indeed.

But what would be the input of such a neural Network ?

Elakoui Directeur Technique 0628730009 salim.elakoui@nway.org

Paris France [image: Facebook] https://facebook.com/salim.elakoui [image: Twitter] https://twitter.com/salimelakoui [image: LinkedIn] https://linkedin.com/in/salimelakoui [image: Google+] https://plus.google.com/+SalimElakoui [image: YouTube] https://youtube.com/channel/UCFsNUh_6pp3bLXWMBNbiltQ [image: Skype] [image: Flickr] https://www.flickr.com/photos/salimelakouimartin

On Tue, Jan 23, 2018 at 6:48 AM, dumbanddumbererest < notifications@github.com> wrote:

https://github.com/cazala/synaptic is a nodejs project, actively developed and has recurrent neural networks implemented.

while we're at it - can we have balances and the ability to proportion each trade? With that done we could have some version of the kelly formula working in the background ensuring balances always grow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/askmike/gekko/issues/789#issuecomment-359683431, or mute the thread https://github.com/notifications/unsubscribe-auth/ATRgT1sngcxpREGXNhPsGGGz8sfC-wdfks5tNXKjgaJpZM4N9gh3 .

[dumbanddumbererest]

previous prices, volume, etc. I imagine being able to add multiple series to something like a self organising map ala http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.124.3646&rep=rep1&type=pdf

[hansixxxx]

I think the neural network needs bbands, macd, cci, dema and candlestick like said above. Seems to be very helpful indicators. I´m on at develop a neural network. Can i join a group of you or somebody wants to join me?

[hansixxxx]

@FLYBYME can you contact me at baerle@feinripptraeger.de to help me? I´m trying to train the Neataptic neural network but no chance yet. I have to study this for my bachelor. I need it for backtestign, no coins for me.

[salimelakoui]

I would be intresting to join yours.

I suspect that these indicators are not sufficent to "predict" with enough precision something.

By the way, what do you predict via your network ? % of increase for the next candle ? % of increase for the next week ?

Elakoui Directeur Technique 0628730009 salim.elakoui@nway.org

Paris France [image: Facebook] https://facebook.com/salim.elakoui [image: Twitter] https://twitter.com/salimelakoui [image: LinkedIn] https://linkedin.com/in/salimelakoui [image: Google+] https://plus.google.com/+SalimElakoui [image: YouTube] https://youtube.com/channel/UCFsNUh_6pp3bLXWMBNbiltQ [image: Skype] [image: Flickr] https://www.flickr.com/photos/salimelakouimartin

On Tue, Jan 23, 2018 at 10:44 AM, hansixxxx notifications@github.com wrote:

I think the neural network needs bbands, macd, cci, dema and candlestick like said above. Seems to be very helpful indicators. I´m on at develop a neural network. Can i join a group of you or somebody wants to join me?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/askmike/gekko/issues/789#issuecomment-359735216, or mute the thread https://github.com/notifications/unsubscribe-auth/ATRgT-IedEXj8L31KFpq1dE32AxqsGm0ks5tNaoJgaJpZM4N9gh3 .