New script: dl_bt_metaquotes.py

[kenorb]

New script needs to be created: dl_bt_metaquotes.py for downloading backtest data (DAT) for given symbol (-p) and years (-y) from metaquotes.net into given folder (-d, e.g. download/metaquotes/yyyy/mm/) and convert it to CSV (-c), similar to dl_bt_dukascopy.py.

Example URL: http://history.metaquotes.net/symbols/EURUSD/list.txt

The script should be fairly similar to: dl_bt_dukascopy.py.

Once downloaded, when -c option is specified, it should be converted into CSV format (the same format as generated via: dl_bt_dukascopy.py). Unfortunately I can't find any documentation on DAT format, possibly could be similar to HST/FXT (see: convert_mt_to_csv.py).

[kenorb]

This should be similar to FXT format (see CSV into HST conversion in convert_csv_to_mt.py, and opposite: convert_mt_to_csv.py).

So we expect something like (multiple rows):

2014-01-01 22:00:00    1.37553    1.37553    1.37553    1.37553          1  2014-01-01 22:00:58  4
2014-01-01 22:00:00    1.37553    1.37553    1.37552    1.37552          1  2014-01-01 22:00:59  4

The DAT formats are converted automatically by the MT4 platform, once they're downloaded at some point.

---

When you start downloading historical data from History Center, MT4 saves DAT-files in MetaTrader\history\downloads\EURUSD folder. When downloading are finished,MT4 makes HST-files from these DAT-files, and saves in MetaTrader\history\Server-Name folder. See: DAT-file.

We know the links for the .dat files on server. We can download things from internet. We can save files to any directory. We have period converter to make .hst files out of .dat files. As I see, these .dat files holds M1 data and are compressed and/or encrypted somehow. Source: Load History from Server Programmatically

[bernardosulzbach]

@kenorb Could you tell me anything about convert_dat.py? It is very poorly documented and I don't think I completely understand to what it supposedly converts .dat files to. Thanks.

[kenorb]

@mafagafogigante So far it's uncompressing it from LZO into raw format. From there it needs to be converted (32bit Little-Endians such as dates, etc.) into CSV, which is not done.

[abcht]

@kenorb I managed to extract exchange rates as follows:

When? 2015-06-30 14:15:00, rate: 1.24831
When? 2015-06-30 14:16:00, rate: 1.24846
When? 2015-06-30 14:38:00, rate: 1.24755
When? 2015-06-30 15:00:00, rate: 1.24745

Which obviously I can format as CSV later on. I'm wondering though, what data is supposed to be displayed in columns / fields 3 to 8 in your sample data above?

2014-01-01 22:00:00 1.37553 1.37553 1.37553 1.37553 1 2014-01-01 22:00:58 4 2014-01-01 22:00:00 1.37553 1.37553 1.37552 1.37552 1 2014-01-01 22:00:59 4

[abcht]

Ok I found what I need actually.

[kenorb]

Btw. Be aware that what's left (to split) is actually half of the bounty, as the first part was result of convert_dat.py (decrypting the DAT format) which was the hardest part and the work is suspended now (wasn't completed yet), that's why the issue is still open. So feel free to finish the 2nd part, which is converting into CSV format as described in the initial issue.

[daniellockyer]

@kenorb I managed to modify scripts to download what we want + save them. I couldn't manage to convert the data though.

[Andrej730]

I've managed to extract timestamp intervals and some ratio (have no idea what they are mean to) that have very similar value throughout any .dat file. Maybe you can help me with it, check my fork - https://github.com/Andrej730/FX-BT-Scripts/

[kenorb]

@Andrej730 I think the most difficulty in here was knowing how to decompress dat file (which was done), and 2nd part how the data is structured and stored. It's a bit pity that many people was working at the same time, and didn't register that are working on the solution.

@d-a-t-a-d-i-t sent some PR today where his solution seems to be more complete, as it deals with the data it-self, and reading these values. So these values are not ratio, but open/high/low/close/volume values as shown in other CSV formats/scripts, and they seems to be stored in incremental way.

Let me test these scripts and close the issue when it works.

[kenorb]

As per PR: #19