Open gfic-limited opened 3 months ago
Just as a question, how did you determine the file format, and what is the meaning of any control bytes that less than 0x3f, i.e. type-0 or something?
Just as a question, how did you determine the file format, and what is the meaning of any control bytes that less than 0x3f, i.e. type-0 or something?
There are no available docs for this format, so at first it was with help of some friend a while back then by trial and error. He had some software already to read this format and gave us some hints.
Got it. In any case, it seems there is no data that has a type-0 style entry. I screened all of the files on the history server from 2020 onwards (not interested in anything before that). Initially I had a bug which suggested type-0 data (stupid javascript in place array annoyance).
From 2015 onwards. All good;
-rw-r--r-- 1 17M Apr 18 19:17 cache/AUDCAD-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:17 cache/AUDCHF-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:17 cache/AUDJPY-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:17 cache/AUDNZD-metaquotes-202403-2015.dat
-rw-r--r-- 1 4.7M Apr 18 19:23 cache/AUDSGD-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:23 cache/AUDUSD-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:23 cache/CADCHF-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:24 cache/CADJPY-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:24 cache/CHFJPY-metaquotes-202403-2015.dat
-rw-r--r-- 1 4.6M Apr 18 19:24 cache/CHFSGD-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:25 cache/EURAUD-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:25 cache/EURCAD-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:26 cache/EURCHF-metaquotes-202403-2015.dat
-rw-r--r-- 1 25M Apr 18 19:26 cache/EURCZK-metaquotes-202403-2015.dat
-rw-r--r-- 1 12M Apr 18 19:27 cache/EURDKK-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:27 cache/EURGBP-metaquotes-202403-2015.dat
-rw-r--r-- 1 21M Apr 18 19:28 cache/EURHKD-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:28 cache/EURHUF-metaquotes-202403-2015.dat
-rw-r--r-- 1 19M Apr 18 19:29 cache/EURJPY-metaquotes-202403-2015.dat
-rw-r--r-- 1 28M Apr 18 19:29 cache/EURNOK-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:30 cache/EURNZD-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:30 cache/EURPLN-metaquotes-202403-2015.dat
-rw-r--r-- 1 25M Apr 18 19:31 cache/EURSEK-metaquotes-202403-2015.dat
-rw-r--r-- 1 4.9M Apr 18 19:31 cache/EURSGD-metaquotes-202403-2015.dat
-rw-r--r-- 1 29M Apr 18 19:31 cache/EURTRY-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:32 cache/EURUSD-metaquotes-202403-2015.dat
-rw-r--r-- 1 37M Apr 18 19:32 cache/EURZAR-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:33 cache/GBPAUD-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:33 cache/GBPCAD-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:34 cache/GBPCHF-metaquotes-202403-2015.dat
-rw-r--r-- 1 6.9M Apr 18 19:34 cache/GBPDKK-metaquotes-202403-2015.dat
-rw-r--r-- 1 19M Apr 18 19:34 cache/GBPJPY-metaquotes-202403-2015.dat
-rw-r--r-- 1 34M Apr 18 19:35 cache/GBPNOK-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:35 cache/GBPNZD-metaquotes-202403-2015.dat
-rw-r--r-- 1 32M Apr 18 19:36 cache/GBPSEK-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:36 cache/GBPSGD-metaquotes-202403-2015.dat
-rw-r--r-- 1 4.9M Apr 18 19:37 cache/GBPTRY-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:37 cache/GBPUSD-metaquotes-202403-2015.dat
-rw-r--r-- 1 4.6M Apr 18 19:37 cache/NOKJPY-metaquotes-202403-2015.dat
-rw-r--r-- 1 4.9M Apr 18 19:37 cache/NOKSEK-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:38 cache/NZDCAD-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:38 cache/NZDCHF-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:38 cache/NZDJPY-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:39 cache/NZDUSD-metaquotes-202403-2015.dat
-rw-r--r-- 1 4.6M Apr 18 19:39 cache/SEKJPY-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:40 cache/SGDJPY-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:40 cache/USDCAD-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:41 cache/USDCHF-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:41 cache/USDCNH-metaquotes-202403-2015.dat
-rw-r--r-- 1 36M Apr 18 19:42 cache/USDCZK-metaquotes-202403-2015.dat
-rw-r--r-- 1 22M Apr 18 19:42 cache/USDDKK-metaquotes-202403-2015.dat
-rw-r--r-- 1 16M Apr 18 19:42 cache/USDHKD-metaquotes-202403-2015.dat
-rw-r--r-- 1 19M Apr 18 19:43 cache/USDHUF-metaquotes-202403-2015.dat
-rw-r--r-- 1 18M Apr 18 19:43 cache/USDJPY-metaquotes-202403-2015.dat
-rw-r--r-- 1 38M Apr 18 19:44 cache/USDMXN-metaquotes-202403-2015.dat
-rw-r--r-- 1 31M Apr 18 19:44 cache/USDNOK-metaquotes-202403-2015.dat
-rw-r--r-- 1 21M Apr 18 19:45 cache/USDPLN-metaquotes-202403-2015.dat
-rw-r--r-- 1 5.3M Apr 18 19:45 cache/USDRUB-metaquotes-202402-2015.dat
-rw-r--r-- 1 29M Apr 18 19:46 cache/USDSEK-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:46 cache/USDSGD-metaquotes-202403-2015.dat
-rw-r--r-- 1 6.7M Apr 18 19:46 cache/USDTHB-metaquotes-202403-2015.dat
-rw-r--r-- 1 27M Apr 18 19:47 cache/USDTRY-metaquotes-202403-2015.dat
-rw-r--r-- 1 39M Apr 18 19:48 cache/USDZAR-metaquotes-202403-2015.dat
-rw-r--r-- 1 4.2M Apr 18 19:48 cache/XAGEUR-metaquotes-202403-2015.dat
-rw-r--r-- 1 16M Apr 18 19:48 cache/XAGUSD-metaquotes-202403-2015.dat
-rw-r--r-- 1 4.5M Apr 18 19:48 cache/XAUAUD-metaquotes-202403-2015.dat
-rw-r--r-- 1 4.4M Apr 18 19:48 cache/XAUEUR-metaquotes-202403-2015.dat
-rw-r--r-- 1 17M Apr 18 19:49 cache/XAUUSD-metaquotes-202403-2015.dat
-rw-r--r-- 1 27M Apr 18 19:49 cache/XPDUSD-metaquotes-202403-2015.dat
-rw-r--r-- 1 30M Apr 18 19:49 cache/XPTUSD-metaquotes-202403-2015.dat
-rw-r--r-- 1 5.0M Apr 18 19:49 cache/ZARJPY-metaquotes-202403-2015.dat
Hi,
Your metaquotes history download decoding of type-2 block needs to be fixed, otherwise the data is junk.
The Type-3 block has 5 bytes, which is decoded correctly. The Type-2 block has 6 bytes, of which the first byte is a minutes offset, and the remaining 5 bytes are the same as the Type-3 block. To decode, you do the same as you would do for a Type-3 block, but instead of incrementing the minutes by one (or, as you do, msec by 60000), you increment the minutes by the value of the first byte.
I have manually compared this to the data in the history centre to confirm it is correct.
This is the Javascript implementation:
Thank you for your implementation, otherwise I would have been stuck. Standing on the shoulders of others, and all that.
EDIT: after debugging, refactoring and testing, this is the final result if useful for others. Tested on data 2010-2024