Simple syntax of usage

[JafferWilson]

@kenorb I am having trouble understanding how to use the files. Please give example of the usage. If possible then for the read and modify.

[kenorb]

Most of Python scripts got -h/--help param to show the syntax. E.g.

% ./convert_csv_to_mt.py -h
usage: convert_csv_to_mt.py -i INPUTFILE [-f OUTPUTFORMAT] [-s SYMBOL]
                            [-t TIMEFRAME] [-p SPREAD] [-d OUTPUTDIR]
                            [-S SERVER] [-v] [-m MODEL] [-h]

optional arguments:
  -i INPUTFILE, --input-file INPUTFILE
                        input file
  -f OUTPUTFORMAT, --output-format OUTPUTFORMAT
                        format of output file (FXT/HST/Old HST/HCC), as:
                        fxt4/hst4/hst4_509/hcc
  -s SYMBOL, --symbol SYMBOL
                        symbol code (maximum 12 characters)
  -t TIMEFRAME, --timeframe TIMEFRAME
                        one of the timeframe values: M1, M5, M15, M30, H1, H4,
                        D1, W1, MN
  -p SPREAD, --spread SPREAD
                        spread value in points
  -d OUTPUTDIR, --output-dir OUTPUTDIR
                        destination directory to save the output file
  -S SERVER, --server SERVER
                        name of FX server
  -v, --verbose         increase output verbosity
  -m MODEL, --model MODEL
                        one of the model values: 0, 1, 2
  -h, --help            Show this help message and exit

Example:

• convert_csv_to_mt.py -v -i file.csv -s EURUSD -p 10 -t M1 -f hst4 (see: Makefile)

Furthermore, I've moved convert_mt_to_csv.py and convert_csv_to_mt.py to FX-Data-Convert-Action, where they're going to be maintained separately. From FX-Data-Convert-Action they can be used as GitHub Action or using Docker image. There is some description on parameters in the README file.

Regarding mt_read.py/mt_modify.py scripts, have been integrated into the following project (before they were called differently):

• EA-Tester and used within the source files (see: mt_read and mt_modify.

Usage:

Usage: mt_read -i INPUTFILE -t INPUTTYPE

Usage: mt_modify -i INPUTFILE -t INPUTTYPE -k KEYGROUP [-d] [-a DOADD] [-m DOMODIFY]

Basically mt_read is the script to read header values from MT formats, and mt_modify is used to modify these header values. Basically any values in FXT/HST/HCC files and other (sel, ticksraw, symbolsraw, symgroups, srv).

Usages (extracted from the EA-Tester source code:

EA-Tester % grep -rw mt_read .
./scripts/tests/test_bt_data_gen.sh:    mt_read -f "$file" -t "${format}-header" > /dev/null
./scripts/.funcs.cmds.inc.sh:# Usage: mt_read -i INPUTFILE -t INPUTTYPE
./scripts/.funcs.cmds.inc.sh:        read _ _ prev_value < <(mt_read -f "$file" -t ${type}-header | grep -w ^$key)
./scripts/.funcs.cmds.inc.sh:            && read _ _ new_value < <(mt_read -f "$file" -t ${type}-header | grep -w ^$key) \

EA-Tester % grep -rw mt_modify .
./scripts/.funcs.cmds.inc.sh:# Usage: mt_modify -i INPUTFILE -t INPUTTYPE -k KEYGROUP [-d] [-a DOADD] [-m DOMODIFY]
./scripts/.funcs.cmds.inc.sh:mt_modify()
./scripts/.funcs.cmds.inc.sh:          mt_modify -f "$file" -t ${type}-header -m "$key=$value" \
./scripts/.funcs.cmds.inc.sh:    mt_modify -m "digits=$digits" -k ${BT_SYMBOL:-"EURUSD"} -t "symbols-raw" -f "$symbols_raw_file"
./scripts/.funcs.cmds.inc.sh:    mt_modify -m "pointSize=$psize" -k ${BT_SYMBOL:-"EURUSD"} -t "symbols-raw" -f "$symbols_raw_file"
./scripts/.funcs.cmds.inc.sh:  mt_modify -m "webRequestUrl=$url" -m "webRequestUrlEnabled=1" -t "experts-ini" -f "$EXPERTS_INI"

If you narrow down which script you're interested, I can provide more examples.

[JafferWilson]

I was interested in the modify script. I have tried the -h but there was not much information about how to place the key group in the command line. I tried my ways but failed to do so. Hence, I was looking for some syntax/example of the usage. I guess it will be helpful to all.

[kenorb]

mt_read/mt_modify usage examples

Changing symbol

$ mt_read -i EURUSD1.hst -t hst-header | grep symbol
symbol = EURUSD
$ mt_modify -i EURUSD1.hst -t hst-header -m symbol=XXXYYY
$ mt_read -i EURUSD1.hst -t hst-header | grep symbol
symbol = XXXYYY

Modifying test date

$ mt_read -i EURUSD1_0.fxt -t fxt-header | grep ^testBeginDate
testBeginDate = 2018-01-01 22:00:00
$ mt_modify -i EURUSD1_0.fxt -t fxt-header -m "testBeginDate=2018-01-02 23:00:00"
$ mt_read -i EURUSD1_0.fxt -t fxt-header | grep ^testBeginDate
testBeginDate = 2018-01-02 23:00:00

Modifying float type

$ mt_read -i EURUSD1_0.fxt -t fxt-header | grep pointSize    
pointSize = 1e-05
$ mt_modify -i EURUSD1_0.fxt -t fxt-header -m pointSize=0.01
$ mt_read -i EURUSD1_0.fxt -t fxt-header | grep ^pointSize
pointSize = 0.01

Modifying HCC header values

$ /opt/scripts/py/mt_read.py -i EURUSD1.hcc -t hcc-header | grep name
name = History
$ /opt/scripts/py/mt_modify.py -i EURUSD1.hcc -t hcc-header -m name=Foo
$ /opt/scripts/py/mt_read.py -i EURUSD1.hcc -t hcc-header | grep name
name = Foo

Modifying server files (.srv)

$ mt_read -i FILE.srv -t srv | grep serverName
serverName = SomeValue
$ mt_modify -i FXCM-USDDemo01.srv -t srv -m serverName=Foo
serverName = Foo

Related: https://github.com/EA31337/EA-Tester/issues/154

[kenorb]

How to place the key group in the command line.

Key group param (-k) is only used for symbols.raw file format, where symbol entries are grouped.

Modifying digits for specific symbol in symbols.raw

Docker example:

% docker run -it ea31337/ea-tester bash
/opt/scripts$ source .funcs.cmds.inc.sh
$ cd ~/.wine/drive_c/MetaTrader*
$ find . -name "sym*"
./history/symbolsets
./history/default/symbols.raw
./history/symbols.raw

$ mt_read -f ./history/symbols.raw -t symbols-raw | grep ^name
... (these are group names, each pair-group have some settings) ...
name = EURAUD
name = EURCAD
name = EURCHF
name = EURGBP
name = EURJPY
name = EURNZD
name = EURUSD
...

$ mt_read -f ./history/symbols.raw -t symbols-raw | grep -A5 EURUSD
name = EURUSD
description = Euro vs US Dollar
altName =
baseCurrency = EUR
group = 0
digits = 4

$ mt_read -f ./history/symbols.raw -t symbols-raw | grep -A5 GBPUSD
name = GBPUSD
description = Great Britain Pound vs US Dollar
altName =
baseCurrency = GBP
group = 0
digits = 4

$ mt_modify -f ./history/symbols.raw -t symbols-raw -k EURUSD -m digits=5

$ mt_read -f ./history/symbols.raw -t symbols-raw | grep -A5 EURUSD
name = EURUSD
description = Euro vs US Dollar
altName =
baseCurrency = EUR
group = 0
digits = 5 // <-- We've just changed digits to 5 for EURUSD only.

[JafferWilson]

That will do. Thank you. Hey I was just wondering if you can tell me whether it is possible to decompile the Ex4 file to MQ4 file? I saw your wrapper, but it is not working as the version of the MT4 is higher. I hope I do not offend you by asking this.

[kenorb]

It is possible to decompile the Ex4 file to MQ4 file?

I don't know any tool/methods for the new builds. Could be possible, but it'll require a lot of coding effort (a month or so) such as cracking and reverse-engineering to decrypt the binary format, but it's basically out of reach for regular programmers, no matter how good they are. The 99% of online tools and services are scams. Secondly, I don't think it's legal to decompile 3rd party files, unless it's your binary (I think). For example when you've lost your source code or you're testing how secure it is to prevent others to steal the code from you.

The wrapper tool was only to give CLI tool capability for the existing old tool for old builds only, so instead of drop & dragging files which was annoying, to allow automation for hundred of files in batch from the command-line, that's all.

[JafferWilson]

@kenorb Yeah.. Thank you for the help. I will use the syntax. I can even add a wiki type page to your repository that will include the syntax. It will help many.