search

timolson / cointrader

Coin Trader is a Java-based backend for algorithmically trading cryptocurrencies. It provides data collection and export, complex event processing and triggering, and backtesting - paper trading - live trading.
Other
450 stars 166 forks source link

Spin wait timeout when running save-data on aws micro instance #62

Closed douggie closed 10 years ago

douggie commented 10 years ago

When running save-data on free tier aws, esper engine shutdown to to one of the threads not retuning within 100ms, resulting in a spin wait time out.

Back test and console modes work fine,so i suspect the save data run mode is making a http request that fails or does not return when running on aws, possible due to https authentication. The watch mode works fine from the console so the get ticker requests seem to work.

timolson commented 10 years ago

The only difference with the console mode is saving to the database. If it's Book data, then a diff is also calculated but that shouldn't take long. Some full Book snapshots are up to 500k in the database, so it might be that such a large record takes more than 100ms to send over the network, save to the DB, and return a confirmation. If it really is that slow (just a hypothesis), then I don't see what could be done for AWS freebies. Spinning up a thread in the background to batch & flush DB inserts wouldn't help, I guess?

douggie commented 10 years ago

Hi tim,

Indeed it is the inserts taking a while. Unless you have any objection i will change the persitutil to use a thread pool and loog an error if not persited with a timeout set in the config file.

timolson commented 10 years ago

I thought there was already a separate insert thread? Maybe I'm thinking of another project... Definitely a good idea.

On Mon, Aug 11, 2014 at 11:56 PM, douggie notifications@github.com wrote:

Hi tim,

Indeed it is the inserts taking a while. Unless you have any objection i will change the persitutil to use a thread pool and loog an error if not persited with a timeout set in the config file.

— Reply to this email directly or view it on GitHub https://github.com/timolson/cointrader/issues/62#issuecomment-51879827.