Closed JafferWilson closed 4 years ago
JafferWilson
commented
4 years ago @kenorb This is the error:
2020.06.29 18:17:39.304 TestGenerator: file "C:\Users\test\AppData\Roaming\MetaQuotes\Terminal\F38A25554834D479B1CEC58CC84D6E68\tester\history\EURUSD1_0.fxt" cannot open [5]
2020.06.29 18:17:39.300 TestGenerator: file "C:\Users\test\AppData\Roaming\MetaQuotes\Terminal\F38A25554834D479B1CEC58CC84D6E68\tester\history\EURUSD1_0.fxt" is read-only
kenorb
commented
4 years ago This is the error:
2020.06.29 18:17:39.304 TestGenerator: file "C:\Users\test\AppData\Roaming\MetaQuotes\Terminal\F38A25554834D479B1CEC58CC84D6E68\tester\history\EURUSD1_0.fxt" cannot open [5] 2020.06.29 18:17:39.300 TestGenerator: file "C:\Users\test\AppData\Roaming\MetaQuotes\Terminal\F38A25554834D479B1CEC58CC84D6E68\tester\history\EURUSD1_0.fxt" is read-onlyfile is read-only
Can you make it not read-only? So platform has write access to it. Then it should work.
JafferWilson
commented
4 years ago But if I make it not read only then the Strategy Tester rewrite my file with the system generated ticks and not my ticks.
Please can you help?
I have a request. Can you please make your convert_csv_to_mt.py headerchanged to
struct TestHistoryHeader
{
int version; // 405
char copyright[64]; // copyright
char description[128]; // server name
// 196
char symbol[12];
int period;
int model; // for what modeling type was the ticks sequence generated
int bars; // amount of bars in history
int fromdate;
int todate;
int totalTicks;
double modelquality; // modeling quality
// 240
//---- general parameters
char currency[12]; // currency base
int spread;
int digits;
int unknown1;
double point;
int lot_min; // minimum lot size
int lot_max; // maximum lot size
int lot_step;
int stops_level; // stops level value
int gtc_pendings; // instruction to close pending orders at the end of day
// 292
//---- profit calculation parameters
int unknown2;
double contract_size; // contract size
double tick_value; // value of one tick
double tick_size; // size of one tick
int profit_mode; // profit calculation mode { PROFIT_CALC_FOREX, PROFIT_CALC_CFD, PROFIT_CALC_FUTURES }
// 324
//---- swap calculation
int swap_enable; // enable swap
int swap_type; // type of swap { SWAP_BY_POINTS, SWAP_BY_DOLLARS, SWAP_BY_INTEREST }
int unknown3;
double swap_long;
double swap_short; // swap overnight value
int swap_rollover3days; // three-days swap rollover
// 356
//---- margin calculation
int leverage; // leverage
int free_margin_mode; // free margin calculation mode { MARGIN_DONT_USE, MARGIN_USE_ALL, MARGIN_USE_PROFIT, MARGIN_USE_LOSS }
int margin_mode; // margin calculation mode { MARGIN_CALC_FOREX,MARGIN_CALC_CFD,MARGIN_CALC_FUTURES,MARGIN_CALC_CFDINDEX };
int margin_stopout; // margin stopout level
int margin_stopout_mode;// stop out check mode { MARGIN_TYPE_PERCENT, MARGIN_TYPE_CURRENCY }
double margin_initial; // margin requirements
double margin_maintenance; // margin maintenance requirements
double margin_hedged; // margin requirements for hedged positions
double margin_divider; // margin divider
char margin_currency[12];// margin currency
// 420
//---- commission calculation
double comm_base; // basic commission
int comm_type; // basic commission type { COMM_TYPE_MONEY, COMM_TYPE_PIPS, COMM_TYPE_PERCENT }
int comm_lots; // commission per lot or per deal { COMMISSION_PER_LOT, COMMISSION_PER_DEAL }
// 436
//---- for internal use
int from_bar; // fromdate bar number
int to_bar; // todate bar number
int start_period[6]; // number of bar at which the smaller period modeling started
int set_from; // begin date from tester settings
int set_to; // end date from tester settings
// 476
//----
int end_of_test;
int freeze_level; // order's freeze level in points
int generating_errors;
// 488
//----
int reserved[60];
};
struct TestHistory
{
datetime otm;
double open;
double high;
double low;
double close;
long volume;
int ctm;
int flag;
};
This is working. Here you will find it: https://www.mql5.com/ru/code/viewcode/11686/131989/fxtfilemaker_script_ad.mq4
JafferWilson
commented
4 years ago The issue is through the header only I guess. It is not the problem with file being read only.
kenorb
commented
4 years ago I have a request. Can you please make your
convert_csv_to_mt.pyheader changed to:
The header is fine, and you can compare with https://github.com/EA31337/MT-Formats/blob/master/fxt-405-refined.mqh which the script is based on. The actual structure you can find here: https://github.com/FX31337/FX-BT-Scripts/blob/844ef0a1308106349cf5a8fb4370fdda8a803a1f/bstruct_defs.py#L106. The structure is the same overall, otherwise it wouldn't work, but the values can make that difference.
This is working. Here you will find it: https://www.mql5.com/ru/code/viewcode/11686/131989/fxtfilemaker_script_ad.mq4
Different scripts are working differently. Maybe there is a special value in header which allows files to be read-only, but I don't know which one. So this file require you to have not read-only file, as it's not needed. You'll get the same test.
Older version of platform could allow read-only files, now it's erroring. There is some workaround, but I'm not sure which header values needs to change.
Allowing the write access, the platform is able to correct the historical errors, revalidate ticks, convert to the appropriate timeframe and narrow down to the specific period. I'm doing a lot of backtesting and optimization, and the files are with the write and I've no issues.
If you want to help to track which values are corresponding to this limitation, you can generate both small files from these different scripts and compare the header. Then you can use mt_read.py and mt_modify.py scripts to read and modify the headers and see if that helps. Or you can zip both them and upload here (make a file for 1 day only to make it small, only header needs to be compared).
Related ticket: https://github.com/EA31337/EA-Tester/issues/94
kenorb
commented
4 years ago So the converted header values, last time when I've checked few years back, should looks like:
symbol = EURUSD
timeframe = 1
modelType = 0
totalBars = 372823
modelStart = 2015-01-01 22:00:00
modelEnd = 2015-12-31 21:59:00
padding1 = <...>
modelQuality = 99.9
baseCurrency = EUR
spread = 10
digits = 5
padding2 = <...>
pointSize = 1e-05
minLotSize = 1
maxLotSize = 50000
lotStep = 1
stopLevel = 0
GTC = 0
padding3 = <...>
contractSize = 100000.0
tickValue = 0.0
tickSize = 0.0
profitMode = 0
swapEnabled = 1
swapMethod = 0
padding4 = <...>
swapLong = 0.0
swapShort = 0.0
swapRollover = 3
accountLeverage = 100
freeMarginMode = 1
marginCalcMode = 0
marginStopoutLevel = 30
marginStopoutMode = 0
marginRequirements = 0.0
marginMaintenanceReq = 0.0
marginHedgedPosReq = 0.0
marginLeverageDivider = 1.0
marginCurrency = EUR
padding5 = <...>
commission = 0.0
commissionType = 0
commissionPerEntry = 0
indexOfFirstBar = 0But still I don't know which value makes it read-only friendly. There is nothing obvious.
JafferWilson
commented
4 years ago Thank you for your kind replies. But I do not understand why the download FXT file or the converted FXT file not working as they should have. May be I am not able to understand the usage of the script. I will try it again. But if you do not make the file read-only, the platform will edit the fxt file and feed the auto generated ticks. Not the one we have feed. I do not know what to do. But I will try.
kenorb
commented
4 years ago But if you do not make the file read-only, the platform will edit the fxt file and feed the auto generated ticks. Not the one we have feed.
If you run backtest in off-line test, and remove the previous FXT files (while not running the platform), then place the new one, it won't add any new ticks, it'll just re-generate the existing one for the period and timeframe that you're testing.
When you allow to regenerate FXT file, the platform will make sure it's valid and not polluted by other broker data. Otherwise if it is not valid, you'll get the data errors in the logs, or it become zero size (not valid at all). I've seen scripts which were creating non-valid FXT/HST files where OHCL values didn't match tick data, forcing 99.9% fake quality (since it's a simple value in the header), or making spread 0, then forcing platform to not access it (without ability to fix it), then it was too easy for EA to demonstrate the good results, whereas in the real trading did losing all the time.
Also, are you sure that your 6 years old random Russian script (fxtfilemaker_script_ad.mq4) is bug-free? You cannot, because you're forcing platform to not validate its data by setting it to read-only. I've tried similar scripts, and even popular Birt's conversion scripts had bugs and errors (when disabled read-only), that's why I've created my own script, so I can have full control over the data that is created. Most script instructions will tell you to set read-only attribute, because basically they don't know how to make it work otherwise.
kenorb
commented
4 years ago May be I am not able to understand the usage of the script.
Let me give you the usage example:
From practical point of view, here are working backtesting runs:
Here are the example of optimization rests:
All above mentioned projects are using these FXT converted files (in write mode using that script). It works as far as you won't connect to online broker. So for my automation purposes where I run tests via Docker, it's isolated, consistent and repeatable environment where I make sure the platform is not connected to anywhere for the test, and the tests works fine.
JafferWilson
commented
4 years ago Wow. Thank you for the help. I will definitely go through this.
I have tried converting the csv file into fxt. But I have got error while I tried to use it in the metatrader 4 strategy tester. I used this script
convert_csv_to_mt.pyand the data from here https://github.com/FX-Data/FX-Data-EURUSD-DS/tree/EURUSD-2019 But the metatrader Strategy tester not accepting the file. Please help me @kenorb . I tried to use the telegram but the app was not working in the mobile. I do not have any other means to contact you.