Logs

[mazurok]

Logs with date-time stamps

• best prices for each exchange (top bid, top ask);
• volume, profit and percentage up to red point;
• list of orders to obtain optimal profit (up to red point).

Specification of format / usage:

1. There is configuration .json file with the following structure :

   {
   "exchange_name": {
       "converter": {
           "our_symbol": "exchange_symbol",
           ...
       },
       "request": GET request with placeholders for symbol and limit (if available),
       "path": {
           "bids": [path inside JSON],
           "asks": [path inside JSON]
       }
   }
   }

Example for CEX and Exmo:

{
    "cex": {
        "converter": {
            "BTC_USD": "BTC/USD",
            "LTC_BTC": "LTC/BTC"
        },
        "request": "https://cex.io/api/order_book/{}/",
        "path": {
            "bids": ["data", "bid"],
            "asks": ["data", "ask"]
        }
    },
    "exmo": {
        "converter": {
            "BTC_USD": "BTC_USD",
            "LTC_BTC": "LTC_BTC"
        },
        "request": "https://api.exmo.com/v1/order_book/?pair={}",
        "path": {
            "bids": ["{}", "bid"],
            "asks": ["{}", "ask"]
        }
    }
}

2. Logger will be called manually from command line with parameters:
   • mandatory: symbol;
   • extra: limit (how many top orders should we get and process, default = 50);
   • extra: config_file (path to configuration file described above, default = orders_config.json).
3. Changes to configuration file are performed manually. After this logger instances are killed and re-run with new configuration file
4. For each currency pair (symbol) there are two files: log_symbol.csv, last_symbol.csv. The former conatins the whole log, the latter - last entry and this is the file to be read for web page updates.
5. Format for log files:
   • first line:
   • timestamp
   • optimal volume
   • optimal profit
   • percentage
   • top bid
   • top ask
   • number of orders (N)
   • each of next N lines contains an order in the following form:
   • type of order (0 - BID, 1 - ASC);
   • exchange on which an order should be put;
   • volume;
   • price (highest ask price or lowest bid price for all "mini-orders" generated for current exchange).

[Odessit007]

We need to establish:

a) time requirements for logging (how often data needs to be collected? is it important to have asynchronous requests? is it enought to work with REST API or we need websockets to obtain streaming data?)

b) format specification for logs (personal file for each exchange? one file for all? formating inside the file?)

c) will logging be run from Heroku, ONU server or local machine?

d) which possible extensions will logging module have? (will log file be regularly read by web application? or it will only be used for exploratory offline analysis? which additional data might be required later?)

e) which programming language should be used for implementing logging system?

[Odessit007]

1) How to cope with limits? (to obtain top best orders instead of all orders where it's possible) 2) Structure of log directory: separate file for each pair? 3) Structure of log files: timestamp, number of orders, optimal volume, optimal profit, percentage, top bid, top ask 4) How will PHP use this files? (read from end up line by line until current section is finished?) 5) When and how will configuration file be updated?

[Odessit007]

DONE: 1) Command line arguments parsing implemented 2) Help and usage examples added 3) Basic configuration file structure defined and tested 4) Code for getting symbol, url and json-path from configuration file written 5) Error handling added: wrong config file, timeout for responses, wrong json in config file or response 6) Implement asynchronous requests via aiohttp 7) Fill configuration file for 6 current exchanges 8) Translate PHP script into Python (computation of orders list / profit / volume / optimal profit and volume) 9) Generate orders which must be performed on each exchange 10) Return JSON as used by front end 11) Create some testing and monitoring utilities:

• logging of time taken to perform requests (almost done: which format should be used?);
• test server imitating exchanges' responses (DONE). 12) Perform lots of testing (system is already stable under timeouts and errors on exchanges's sites, it seems that everything works fine) 13) Add orders visualization 14) Redirect JS to PHP which queries Firebase (or implement queries in JS) 15) Implement visual design 16) Implement dynamic updates (each n seconds table, titles, plots and orders must be updated, not the whole page) 17) Add list of available currency pairs on web page (some discussion required) and implement dynamic update system 18) Add new exchanges to configuration file and wiki