kenorb
commented
6 years ago Basically, it should work the same as the MT4 platform does it by default when backtesting. It's got its own built-in converter, so we can compare the result data.
Closed kenorb closed 6 years ago
kenorb
commented
6 years ago Basically, it should work the same as the MT4 platform does it by default when backtesting. It's got its own built-in converter, so we can compare the result data.
ghost
commented
6 years ago Hello again. I have been going through the data to understand how to solve this issue, but I have some questions.
First, what exactly is a price bar? From what I can see, it appears to be a collection of ticks for a given timeframe. The examples you provided appear to just take all the available ticks and group them within every 1, 5, or 30 minute timeframes that occur in the input ticks.
Second, do you have any examples of what the other models would generate for a given set of ticks? I am having some difficulty understanding the control points model in particular, but the open price one appears to be very simple. It sounds like it uses only the opening price to generate the tick for a given price bar.
Thanks.
Edit: I also notice that you say the "every tick" model is already implemented, but it does not appear that any interpolation is already performed. Is interpolation necessary for any of this?
kenorb
commented
6 years ago EURUSD1_0.fxt into the platform (or generate a small one using the scripts from CSV to FXT), select Control points and run a backtest, this will generate EURUSD1_1.fxt which can be read using the mentioned Python scripts. Backtesting steps can be found here.kenorb
commented
6 years ago "every tick" model is already implemented
That mean every tick in CSV is present in FXT (downside is slowness and a big file). When you convert to control points, you're filtering out the ticks which are less useful. So basically instead of having 100-200 ticks per minute, you've only 4 major ticks per minute (OHLC values). In open price mode, you've only open values (a single tick) per timeframe (e.g. M1).
kenorb
commented
6 years ago Let me know if anything needs further clarification.
ghost
commented
6 years ago I've been reviewing the open prices model, and it appears MT4 does some weird things. For some timeframes, it chooses to output one tick, on others it chooses to output two ticks. When generating two ticks, it appears to generate one for the start and another for the end of the timeframe.
Any ideas how I should handle this? I can't seem to find any correlations to explain why and when it chooses to act in this manner. I had thought the open price model might be as simple as just keeping the first tick of a timeframe and discarding the rest, but there's evidently more to it. In particular, it appears I am expected to modify the tick's volume to be the sum of all the tick volumes for the timeframe.
kenorb
commented
6 years ago Maybe MT4 is using data from 'the nearest less timeframe must be available'.
Please check this Support point section. But I'm not sure if this describes the logic of Control points mode. If it's not, or it's not clear, try to make it more logical, as far as MT4 won't generate any validation data errors during testing.
In my understanding Control mode should be done by keeping the ticks which are either higher highs (keep the next tick if it's higher than the previous high), lower lows (keep the next tick only if it's lower than the previous low). This way we can filter out useless ticks which doesn't provide any new low/highs. The main goal of Control mode is to filter out the useless ticks, and keeping the major changes of the price. So if that make sense, you can implement this way, even the method from MT4 differ. We can always compare the results later and find which method is more reliable.
And Open mode can be implemented by having only 4 ticks per timeframe (OHLC data), unless they're all the same. The volume should still match to avoid any data validation errors.
Let me know if that make sense.
Extend
convert_csv_to_mt.pyscript to have 3 modes of FXT conversion:-m 0: Every tick (the most precise method) - this is how it works by default now,-m 1: Control points (the nearest less timeframe, 1 minute OHLC),-m 2: Open prices onlySpecifying multiple modes by:
-m 0,1,2should be supported as well, so as result 3 files should be generated at one run. This-mparam only applies to FXT format. For HST it should be ignored.Filename
The generated FXT filename must have type name SSSSSSPP_M.fxt where:
Currently it's generated correctly for Every tick model (with 0), we need to add support for 1 and 2. If model is not specified, by default generate the Every tick model (0).
Resources
For more explanation about mode of modelling, see:
Steps
In some folder, clone repo with CSV files:
Combine CSV data from 2 days into single file:
or 4 days:
Clone this repo:
Convert CSV file into FXT (for M1, M5 and M30 timeframe) or using Docker:
Similar to hst4 format if you need to for testing.
Read the generated files by:
Now generate file with control points (
-m 1to be implemented):This should generate 3 files
EURUSD1_1.fxt,EURUSD5_1.fxt,EURUSD30_1.fxthaving only control point prices as described above.Sample 1
M30
M5
M1
Check files from EURUSD-2014/EURUSD/2014/02 for more accurate CSV data to compare with.
The above FXT files has been uploaded below.
The above rows are in the following order: bar timestamp, open, high, low, close, volume, timestamp.
Sample files
These files can be read by
convert_mt_to_csv.pyscript as shown above. Or visually you can open them by MT4 platform and open by File, Open Offline (files needs to be placed in tester/history folder of the platform dir.See also related issue: #86
Est. 16h