A question

[Gabriel-RABHI]

This is "of" subject. I'm trying - as side project to my job - to build a working trading bot, a scalping one at tick or low time frames. I'm using an IG broker account, trading CFD derivative market (with high laverage). Do you know if it is realistic to build a such program, with a stable, good profitability ? Do you have one or do you know people that had reached this goal ?

It is really hard. I have developped the basis : a back-test program that take care of all parameters, including spread cost - to have exact results.

I think there is potential benefits to go in a completly new approach. I do :

• geometry computations : working with angles, distances, distribution in a fixed cartesian space
• work with linear regressions
• pattern recognition using deep learning

My first results are not bad, but low volatility sequences are plenty of false signals. Sometime I reach a 20%+ profitability in a day, but it is not stable. I'm working on a self permanently recalibration process : hard work here too.

Thank you !

[amittleider]

Hello Gabriel,

I admire your commitment to your personal projects. I don't doubt that there are some strategies that are yet to be exploited, but I think one can spend ones whole life searching and not find anything.

Regardless, here are some thoughts.

1. High frequency, small trades, are particularly difficult to deal with because of fees. double check your fees calculation to make sure it's properly accounted for
2. Even random trading can make you 20% on a lucky day. Looking at the best days is a type of bias. Stick with hard numbers like the Sharpe or Sortino to analyze your strat
3. The code hosted in this repo probably won't help too much with what you're doing. The code here makes using IB more convenient for automated entry of orders, but you will take a perf impact for it. For most use cases, that would be fine, but for tick by tick strategies, it likely won't be a good solution (let me know if I'm wrong here, i never actually tested it like that)

Good luck and have fun!

[Gabriel-RABHI]

"one can spend ones whole life searching and not find anything" - I guess you're right ! Have you got a bot ? What mean "Beating the markets with software" ?

To react to your points, and share what I have already learned :

1. True : having an perfect per trade cost computation for back-testing is necessary, especially in small time frame trading. In fact, it is important for any time frame, because it can lead to a completely biased result. IG takes only fees on spread for CFD trades, witch is easy to compute. But I do not see that as a major problem while most positive trades are N times larger than the spread. Spread factor can be an input parameter.
2. Of course, the best results are not a proof - stable results on many days (for small time frame or tick) or months (larger time frame) on various markets is a proof that the bot is able to do money. It must manage flat market as well as high volatility.
3. Your code is helpful for me while I need to benchmark my own computations with standard indicators. There is absolutely no performance issues, and can ingest dozens thousands new inputs per seconds. As a video game industry veteran, I have a great notion of what performance is. The API delay is far more problematic than the computations.

Here are a visualisation in my own hand-made tool (developed with SkiaSharp) :

To have a perfect computation, I compute all at ticks level, and is able to visualize in candle changes. The blue curve is Bid one, the brown one is Offer one (or inverse). The White curv is a the "unfavorable" outcome of bid/offer as base curv for all computations. At tick level, there is no more candle concept like high, low and close values :

So, I can visualize my own oscillator. For example, the simplest one based on a linear regression :

This oscillator is a little bit more stable than a tri-exponential one.

My back testing program can open and close trades and compute the result on the given period :

It open and close at the right spread curv a the exact time. It can work on large tick datasets. It can work live on the IG API too.

So, what I've learned at this time :

• I'm at the really beginning of my research. I have tested perhaps 15% of my hypothesizes at this time, after 2 months of hard work. Test all approaches may take one year or more. If I don't identify a good approach then, I will abandon the project.
• Having a good tooling is needed : you must be able to visualize, understand and analyze every single peace of data and behavior, change parameters live. Navigation in visualization must be perfect, and rendering accurate. When the bot is paused, you must be able to render the automate variables directly on graph.
• The program must not be too complicated. Each computation must be splited in classes that can be duplicated to multiply testings. The architecture must be pipelined : computation of the basic data -> basic oscilators / indicators -> automat that decide when to open and close trades -> finally, a general purpose trade manager (with resistance, drawdowns, etc).
• Most algorithms are efficient at a time, and not at another time. Here, we can see large gains and lot of small false signals : And this is the biggest challenge : choose the right parameters / algorithms at the right time, having indicators that prevent from opening on a false signals. A market had various volatility "phases" that must be taken into account to adapt in real-time the bot. To solve this problem I have two approach :
• Separate the opening bot and the one which manage the closing. When the trade is opened, it is an another story to close it at the right time. The bot managing the best closing must be able to optimize any trade, even a random ones. Back testing closing bot must be done.
• The bot can scan the recent market history data to find the parameters that is the most efficient. For example, every 1 minutes, it must take the last 15 minutes and back test thousands of various parameters to identify the one that is the most efficient. This approach may permit to adapt bots to current market phase.
• Finally, the goal is to have a ratio of good and bad trade that is positive, taking fees into account. The ratio have to be significant (2/3 min). Too low number of trades is a danger, because it is less reproducible / too much make fees to larges.

So, I'm jsute ate the beginning : this is a really interesting challenge. I learn a lot by understanding and implementing existing indicators, and completely new ones. My background is geometry, 3d, state machines, physical simulation - i'm not a math guy. I haven't yet tested all approaches, and pure geometry approach seems to be not bad.

Thank you !

[Gabriel-RABHI]

To go a little deeper in Tick time frame scalping...

Tick time frame have really specific properties. First, it is not linear : the delivery by the servers are not regular. Sometimes you have 3 updates in the same second, and you can have few seconds where there is no updates. Compared to candle stick, you have to take care of this unstable delivery of prices. A candle stick is a discretization of the price variations for a given period of time, that group updates in a single time frame with Start, High, Low and Close prices. But if you go on two brokers, candle sticks may not be the same. The main advantage of the candle sticks is that steps are all time in the same time interval, while ticks entries are a little chaotic. If you take price update at regular time interval, your curves are wrong – you must take the exact time stamp of each price update. So, you have two solutions :

• The easy one is to resample the curves to obtain price updates that are linearly distributed in time. You can use a linear interpolation or spline smoothing. Generally, you oversample the curve.
• The hard way is to make everything that ingest the prices taking account of the timestamp of all entries, by interpolation : it is an hard work… A simple EMA is a lot more complicated.

When you have good inputs to build a scalping bot, you must take the right path. A good way is to start from the perfect, ideal, maximal trading result. In my program, there is a “medium trader” back-testing bot : it’s a code that compute the perfect trades for a given period of time. It is not a prediction, it compute the maximal gains like if the bot should read the future to make the best, maximal gains taking care of spread and fees. This is not to make it running on a real trading day, but only to show the maximal gains that a series of trade can theoretically make if we can see the future. This maximized trade sequence is a starting point to make algorithms optimized. To make it “realistic” you have to adapt this engine to take account of technical facts :

• you cannot open a trade through the API in 1 millisecond.
• The data that permit to take the decision to open and close trades are already in the past when your bot analyze it.

So, you can shift all the trades for 1 second to 1.5 seconds. An average delay of one second is quite conservative. If you shift all trades one second latter, the result gain is falling. This is what we have with moving average : we are all time too latter to take the gains. This is a far more complicated problem to deal with than the fees, because being late cost generally more than the fees. Second step to adapt the “medium” trading bot, is to makes trades decently predictable : it means than you may let the time to a super-smart bot to detect the trend change in a consistent manner. So, you will shift the trades from 3 to 10 tick updates to detect the trend change before opening a new trade. This two constraints make the code of the “medium” trader bot change deeply. You’ll have to think in a reverse manner : “is it possible that an prediction bot can reproduce this perfect serie of trade ?”. Reversing is the goal of Deep Learning : a neural network takes a bunch on data (ticks), outputs (trades), and learn the underway links between the two. I’m at the beginning of this approach : it is one I have to test. But the mental process of this strategy is probably one of the most realistic for self tuning algo like Deep Learning, Decision Trees, Genetics, etc.

[Gabriel-RABHI]

What I'm learning... a little bit more !

It seems there is globally two way to create a good scalping bot :

1. Focus on a strategy based bot : searching for the all road algorithm.
2. Focus on a contextual adaptive bot.

The second approach seems to be a lot more efficient than the first one. What seems important is not to focus too much on the silver bullet strategy. There is no silver bullet : a given strategy will work at a time, not at another time where another strategy may get better results. Actually, I'm working on a derivative of the SuperTrend. SuperTrend is a really simple yet efficient way to have an entry point and a close point. I have optimized it to be more "efficient". It work well on any market while you have large movements. If trend and variations are significant, it will permit you to get most of the gains you should expected.

So, the most important thing is to adapt the parameters of the bot for each market variation. I have made a back-test that test thousands of parameter combinations (brute force optimization), and the result is impressive : in the same day, hour after hour, I can find parameters sets that make gains. When the market is flat, there is no hope of gains. We must detect it and avoid any trade. When the market move, then we can compute the best parameters to get gains, for a sequence of few minutes to an hour. Whatever is the strategy, there is parameters that fit a given period of time.

My next try out will be a continuous adaptation of parameters. If, for a given period, some parameters are efficient, probability they still efficient for the next few trade is high while the market is sequential : this is my hypothesis. If it works, it may test various strategy with thousand of parameters, in a continuous way to get a new setup every few minutes. It may be possible to have a fine grained parameter selection by gains computation in a dichotomy time laps selection approach. A Depp Learning pattern detection to associate a parameter set to a market situation may be possible too.

I will stop to share my researches, results may start to be better - I hope, super efficient...

Good luke !

[Gabriel-RABHI]

What I'm learning, again !

(tell me if this thread is not appropriate - I think it can have value for anyone which try to find the "right" automated trading strategy)

The starting point though that "random trades" can be efficient is... true.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3708927/

For those who do not have the time to read the summary : non random strategies are not significantly better than a pure Random Strategy (open and close randomly). In other words : a strategy can have a little better result in a particular time windows on a specific market, because if you are lucky, you will fit with market sequence. It will give you a "sentiment" of control that is, scientifically, non proved.

This is mind blowing, but something logical. And this make my assumption that a brute force parameter search for a given time window may permit to have few winning trade just after that is false. That's all.

So. Anybody can see that there is a logic in price moves : yes, there is a logic, which is the result of all orders on a given market. I should divide this in 3 layers of time scale :

1. The tick level : a mix of noise made by individuals and the system that is processing the orders.
2. The minute to hours moves : small to mid institutional that have thousands of traders with a middle size order book.
3. The big institutions : large order books that make moving prices. They have all time problems of liquidity : they have to manipulate the market to make money. One of the most visible fact is imbalance. Accumulations, distribution are their strategy core : they make it appear.

The first 2 are following the third : all concepts like resistance, trend, breaks are at the same time a fairy tale and a reality. All traders are learning all this basics, but in reality only the traders that can manipulate the market with huge orders can make performance greater than the global average prices progression on long term. In average, any trader will, in his life, obtain a gain that is roughly the progression of the total value of the stock market. Some are more lucky, but generally on a limited time window. And central banks or rumors are more powerful to make volatility than any other actors. And if everybody is doing random trading, no one can do benefits.

This is what is saying this study.

It is something that I can feel, but see it written and proved is a game changer. I have to rethink all my strategy ! Again...

PS : 3 years ago I've done a 4 months full-time research on the French lottery, to find a way to optimize played combination, based on analysis of past results. This was really hard and I have computed terabytes of data to obtain theoretically a enhancement of ROI (return on investment) of 0.03% (0.29... to 0.32) - if my computations are correct. To obtain this enhancement, I have found a bunch of 12 probabilistic rules that must be respected (for example, do not choose only odd or even numbers because this combinations are too few). In other words, if number have an equally probability to be drawn, combinations are not. My final though is "my enhancement computation may be false". One thing I discover is that to obtain a realistic draw series, I have to use cryptographic cipher, a generator of random number serie. Any other random generator was terribly bad and probabilistic characteristics where not the same than the real draws. So, now I come back to this fact : make a trading bot is fighting the randomness of the world.

[Gabriel-RABHI]

Random strategy

I have back-tested the random strategy, and the result seems to be perfectly stable : on an intra-day tick time frame, you loose the fees. Using only a gain limit and a stop loss with a random direction, you can obtain tremendous gains on a short time-window - what I've already see with brute force best parameter search. Change only one parameter and you loose. All computations taking care of sophisticated indicator seems to be useless...

So this exceptional result is only a lucky result of a correlation of a price evolution and parameters. 15% in one hour is a dream... but it is like play to the lottery !

So, is it possible to bias the randomness to stabilize a gain rate ? Is there any statistical element that permit to establish a like between market characteristics and the parameters ?

It is a research way.

For me, at this time, the second most promising way is to adapt the strategy to market manipulations done by the biggest actors. "Imbalance" is a key principle. You can see imbalance on any market. Imbalance is a disorder in the order log where bid or offer cannot be balanced. You can see it on candlesticks.

When an imbalance is done, the price all time come back to the imbalance level to balance it, later. This rule is so true that it may be possible to detect it and wait compensations. This strategy was the main reason I start this research (yes, build an bot based on imbalance analysis).