Separated sqlite db files for each config

[xFFFFF]

What do you thing about it? Could input asset name in version value in config or use different folders. With 20 bots runned on this same db I had busy errors.

[bettimms]

I used to monitor 19 assets in binance with multiple strategies and 3-4 assets in gdax with 1-2 strategies in total would come more than 25 and I didn't have problem. Just make sure you've disabled candleWriter?! Try and see if this helps, if not we can see what can we do regarding your approach! PS: I actually run more than 35 in binance on two different PCs until i got Error 418 and got IP blacklisted for a day!

[xFFFFF]

This is rare, but it happens. I use candleWriter, so that after accidental crash or reboot the bots do not have to re-download the value for my SMA400 only after they were ready to trade immediately.

When candleWriter is turned off, the gekko probably does not use DB at all.

Why do not I use postgre or mongo instead? Because it's a server, it needs more RAM than an SQLite file, and RAM is very valuable on VPS :) Besides, I do not like to open ports, local ones too.

BTW

Integrate with one of current backtesting tools, read most profitable strategies periodically (daily, weekly, monthly) and generate config files with most profitable strategies and assets/currencies

If you were considering BacktestTool and needed some feature for this project, let me know.

[xFFFFF]

I actually run more than 35 in binance on two different PCs until i got Error 418 and got IP blacklisted for a day!

I have not succeeded ban yet, but on one host I have 16 bots running with tickrate 4 seconds. Some time ago I wrote such a script with the hope that desynch tickrate will help you avoid the ban. https://forum.gekko.wizb.it/thread-1479-post-58536.html#pid58536

But I dont know how accuracy it is.

[bettimms]

Thank for your sharing the script for tickcerate and being very active on this gekko field, I'll give a try as soon as I've some time.

Regarding integration with backtest tools; yes I'd like to integrate with current tools for backtesting so it could switch and run automatically periodically, but these days I'm quite busy, as soon as I'll get some time I'll go back to it and try to implement it.

I think this sqlite issue now will get priority, but it needs further research to see what would be best solution for the case.

[bettimms]

As a dirty workaround for sqlite busy error, since bots will run in local machine and for single user, have you tried leaving sqlite connection open?! Go to gekko/plugins/sqilte/writer.jsand comment line 101 and 102. As e temp. solution for this case (not the best)!

var finalize = function(done) {
  this.writeCandles();
  // this.db.close(() => { done(); });
  // this.db = null;
   done();
}

PS: I didn't test if it's writing candles but error is not being thrown. If it works please share!

[xFFFFF]

I will try, but it need time for tests. This error I have maybe 2 times on month. BTW sqlite dont support only one connection to this same DB file? If I dont close first connection then second could connect? And : https://www.sqlite.org/faq.html#q5 " But only one process can be making changes to the database at any moment in time, however." And : "When any process wants to write, it must lock the entire database file for the duration of its update. But that normally only takes a few milliseconds. Other processes just wait on the writer to finish then continue about their business. " Unfortunately gekko is crashing after get this error. Should retry after few miliseconds... BTW I dont know JS etc. but sqlite has not nice feature called autocommit, enabled by default. Maybe if we can disable it from node level and commit periodically then we fix problems with busy on forever. For import gekko mode autocommit=0 should improve performance too.

Feature with backtesters are really good. You can create something like reactive trader. From GA I recommend japonicus. It's smartest and I'm tiring @gab0 for a few more settings. If he implement it in japonicus then I will do a reactive-trader with a few lines in bash: D

I have a lot free time in last time and I have fun from learning coding and improve my bots :)

[bettimms]

@xFFFFF Currently I see that the problem of sqlite is on the gekko side since multiple files depend on it and the path they read is local. Also making multiple sqlite files might not be the best approach since then we would have to make a single db for asset, having many assets make it difficult to maintain dbs(for example to update for missed candles one by one). Unfortunately for now the issue is on gekko side, we're hoping with the next release to have some more flexibility. #2163

[bettimms]

@xFFFFF Regarding Gekko-Datasets in the google drive to include these two other types for mongodb and PostgreSql?! I think right now only sqlite is supported?!

[xFFFFF]

I will implement postgre to BacktestTool soon. Mongo? Maybe in future. I dont know any statement and these looking weird :P

When you write Google drive, do you mean Gekko-Datasets? If so, I am not able to provide the converted SQLite and mongoDB databases at the moment, because I've already had 500 MB of free space on Google Drive. To set up a new account, I need a phone number, in my country, phone numbers are registered on the ID so its difficult. I have a non-shared full Bitx history and start collecting candles for Bittrex but havent space.

However, I think a script that converts SQLite to postgreSQL and mongoDB would be a good solution. I used to play with it, but I always have something else to write. Maybe you want to take it? :)

[bettimms]

@xFFFFF updated comment!

[bettimms]

I'll close it for now since it's not directly related to multi-gekko. Maybe for now we just have to stick for mongodb or postgresql solution.