Write Python script to read symbols.sel file. [$50 awarded]

[kenorb]

Write a new script in Python 3 which will print the values from symbols.sel to the output.

Then print the known values to the output such as:

• symbol
• digits
• index
• group
• pointSize
• spread
• tickType
• time
• bid
• ask
• sessionHigh
• sessionLow
• bid_2
• ask_2
• etc.

Can be in any output format, but format of fields should match its type (ask/bid should print float and time should print time, etc.).

See: symbols.sel.h

Check some other code examples for format details. Also this one.

Check these symbols.sel files as example.

--- The **[$50 bounty](https://www.bountysource.com/issues/31094557-write-python-script-to-read-symbols-sel-file?utm_campaign=plugin&utm_content=tracker%2F20487492&utm_medium=issues&utm_source=github)** on this issue has been claimed at [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F20487492&utm_medium=issues&utm_source=github).

[rosasurfer]

Hi. Let me provide a little contribution: SYMBOL_SELECTED.h.txt I'v managed to implement testing and pseudo real-time charting in a regular chart (not an offline one) of virtual instruments (ie. symbols that do not exist on the trade server). Means reading/writing of .hst, .fxt, symbols.sel, symbols.raw (ticks.raw too but this isn't needed). So if you need infos just drop me a line.

[kenorb]

@emmzett Thank you, that's very useful. Do you have similar .h file for .raw format (symbols & ticks)? If that's ok, I could include that in MT-formats repo (or you can send PR) where I keep track different formats of MT, so we can write some Python scripts reading such formats. Currently we've some issues where FXT files needs to be read only, because MT doesn't understand it, so we suspect that the symbols.sel needs to written specially for that, so it understand the fxt ticks.

[rosasurfer]

@kenorb Yes, here are some more. All done by me, FXT header with help of Birt. SYMBOL_GROUP.h.txt SYMBOL.h.txt FXT_HEADER_405.h.txt

MAX_SYMBOL_LENGTH is defined as 11, not as 12 as you would find elsewhere.

define MAX_SYMBOL_LENGTH 11

I wrote a PHP script to read symbol files which for example produces something like this: mt4.symbols.log.txt

As for now I don't have a TICK.h file but the format is simple. It's only used for the little tick chart and at the moment I have no use for it. You can use anything I provided here as you like it.

[rosasurfer]

@kenorb If you want to work with your own read-only tick files you can (a) use Birt's suite with a current build to patch the process at runtime (not free) or (b) use an old build and patch it with Birt's patch script (free).

In both cases and assumed the terminal is disconnected you only need the symbols.sel file for margin/profit conversion to the account currency. Otherwise only symbols.raw is needed. Example: If I test SGDFX7 (that's a synthetic Singapure Dollar index that doesn't exist anywhere in the world but in my terminals symbols.raw) I set the account currency during testing to SGD and it works right away. If I would change it to something else I would get the error in the terminal log: "cannot convert margin currency" or similar. So, I can provide symbols.sel but I don't have to.

Additionally I stopped comparing profits to money values and analyze PL only in pips. If I run a ten years test the results in money terms are misleading because the account currency itself moved 20-50% in the test range. I like to see absolute results/dd and this is given to me by pips.

If you want to test with your own data (this is what I understand from your post) I suggest the following procedure: (1) Write a few scripts do download the Dukascopy M1 history for the instruments of interest. Do not convert it to CSV as Birt it is doing, save it in (may be your own) binary format. Validate the data during download (every single bar, there are very few errors). Update this history once a day via cron. (2) Convert this history to MetaTrader .hst format (400, 401, whatever you prefer) and keep it up-to-date via cron. In .hst format every bar needs at least a volume of 1, zero is invalid and will crash your test. (3) Let the terminal create the tester tick files from your data (you can force a zero spread by modifying symbols.sel) and only test with the BarOpen model. This is fast and exact even in optimizer (one year M1 BarOpen 10-15 seconds here). The MetaQuotes tick models are all fake and tests with it will take your life-time and the life-time of your children. Make sure you have zero errors in the tick files!!!!!! With luck the terminal will complain "invalid tick file", without it will just stop the test somewhere in the middle of it. If you write your own tick files pay attention to the different tick bar formats before and after build 509 (the tick header is the same). (4) Only use the every tick model if you really need to and only for validating an already proven strategy. Do not start with it. (5) Write your own reporting tools and make everything as simple as possible (strategies, signal conditions, indicators, ea's).

[kenorb]

Fixed in PR #38

[kenorb]

@emmzett By any change, do you have any header format or some guesses for .srv format (server configuration files)?

[rosasurfer]

This is getting more difficult. Without reverse engineering you can only change the 1st ip address and port. A file can hold multiple entries, only the first one is in plain text. I use such a customized .srv file for my synthetic instruments.

The encrypted .ini-files and the .srv-files need to be decrypted but it's not easy. You need a good C++ developer for this. I never did it but I know the way to go:

You can start with build 225, that one is not copy protected and debugging is easy. You step through the application and put breakpoints on the file functions. Once the addresses are found you can go back from there and patch the decryption code away with no-ops, jumps etc. A smart way would be to patch the terminal to be able to read/write encrypted and decrypted files. Look at WinAPIOverride, that's an excellent piece of software: http://jacquelin.potier.free.fr/winapioverride32/

With all the later builds it's getting more complicated because the terminal code is protected/obfuscated. It is still achievable and the encryption algorythm only changed marginally. However, if you do this or find somebody doing it for you do not publish it in a public place, like for example here at GitHub. MetaQuotes is desperately looking after reversing attempts. The moment somebody breaks their protection (which is not too hard) they will change the code (as they did a few times in the past with the broken server protocol) and your efforts will be wasted. So, keep the success for you. :-)

Imho you don't really need the .srv format for anything. With a modified file you can make for example the terminal act on online charts as it would do on offline charts. This way you can for example update an online chart from disk with ID_CHART_REFRESH wich doesn't work for regular online charts. Other than that I had no use for modified .srv files, at least until now.

[kenorb]

@emmzett Thanks for advise, really useful.

I've main 2 issues when testing with MT4: with lot step and spread values, these are very not stable and you never know how the test is going to be like.

Basically I'm expecting Lot step to be 0.1 for 4 digits, and 0.01 for 5 digits brokers. These values seems to be connected to the server and stored in accounts.ini which is encoded. So as for workaround I've now two files: accounts-d4.ini to use for 4 digits, and accounts-d5.ini to use for 5 digits (to copy corresponding file depending for how many digits I'm testing). In accounts.ini my account, server and password is encrypted, but this is necessary to set the right Lot step and if I change Login=1809640 to something else, this stops working (My Lot step is 0.1, or even 0, instead of 0.01). This workaround is still not stable, as on some instances of MT4 the accounts.ini is wiped out even when I'm trying to copy the right one (ini file), and Lot step most likely is 0 which is invalid, so I assume MT4 needs also the right Server value and the corresponding .srv file. This is all messed up. Maybe I don't need to read .srv, but definitely accounts.ini would be helpful to change the associated Lot step with the account in order to know with what settings I'm backtesting.

Problem with Spread is that if I've backtest files read-only - spread is 0 points (always, doesn't matter if I change in fxt, ini, etc.). Only when files can be read-write, so platform can rewrite them the spread is fine (as set for the test). As the side-effect, the tick data is quadrupled (EA31337/FX-MT-VM/issues/54).

Have you come across similar problems? Haven't you noticed that the spread is always 0 when you've read-only backtest files? It reports spread 10, but there is no real difference between Bid and Ask prices which makes it spread 0.

I've some things on GitHub, so it can be joined effort in the future. If we've some stable testing platform for the backtesting, it'll be easier to fix it when they will change something. But you've also a good point.

[rosasurfer]

Ah, now I understand what you meant last time. Yes, I agree. This setup is not stable. I follow a different approach. For testing I use another installation where I don't have any issues with different data coming from different connected brokers. I test always disconnected. The test installation is running on the same history folder as the online installation, this way price data is always up-to-date. Because I'm disconnected I can choose/set whatever spread I like, via the tester dialog or via a self generated tick file. As long as I do NOT choose 0 (zero spread) the terminal will not apply the current spread found in symbols.raw/symbols.sel. If you choose 0 the terminal will set the current spread sent by the broker. If you look in the symbols.raw file: a spread of non-zero means a fixed spread. A spread of 0 (zero) doesn't mean a zero spread but the current spread sent by the broker.

So, if I use the history folder and data of my broker and want to specify a low spread I select 1. That's more than realistic for a 5-digits broker (I don't use 4-digits brokers).

In general I generate history files by myself and write symbols.raw by myself. This way data is more accurate and spread is not an issue because I set a real 0 (zero) spread as the current spread in symbols.raw (bid = ask). Same for self-generated tick files.

Does this make sense to you?