search

AlgoTraders / stock-analysis-engine

Backtest 1000s of minute-by-minute trading algorithms for training AI with automated pricing data from: IEX, Tradier and FinViz. Datasets and trading performance automatically published to S3 for building AI training datasets for teaching DNNs how to trade. Runs on Kubernetes and docker-compose. >150 million trading history rows generated from +5000 algorithms. Heads up: Yahoo's Finance API was disabled on 2019-01-03 https://developer.yahoo.com/yql/
https://stock-analysis-engine.readthedocs.io/en/latest/README.html
1.04k stars 251 forks source link

Fix S3 feature flag publishing to S3 even if the intraday.s3.enabled: False and fetch -U 0 is used #350

Open jay-johnson opened 5 years ago

jay-johnson commented 5 years ago

Engine logs before PR https://github.com/AlgoTraders/stock-analysis-engine/pull/349 showing the timeout and the jobs taking >30 seconds to finish

2019-03-01 02:01:19,220 - celery.worker.strategy - INFO - Received task: ae.work_tasks.get_new_pricing_data.get_new_pricing_data[0c8d4817-c84a-4dff-8550-724083d216f6]  
2019-03-01 02:01:24,347 - celery.worker.strategy - INFO - Received task: ae.work_tasks.get_new_pricing_data.get_new_pricing_data[eb2e14bb-e8d5-4af2-b715-b0ce708278d7]  
2019-03-01 02:01:36,947 - ae.work_tasks.publish_pricing_update - ERROR - ticker=XLF-tdcalls failed uploading bucket=backtest key=XLF_2019-02-28_tdcalls ex=Could not connect to the endpoint URL: "http://0.0.0.0:9000/backtest/XLF_2019-02-28_tdcalls"
2019-03-01 02:01:45,095 - ae.work_tasks.publish_pricing_update - ERROR - ticker=XLP-tdcalls failed uploading bucket=backtest key=XLP_2019-02-28_tdcalls ex=Could not connect to the endpoint URL: "http://0.0.0.0:9000/backtest/XLP_2019-02-28_tdcalls"
2019-03-01 02:01:52,414 - ae.work_tasks.publish_pricing_update - ERROR - ticker=XLF-tdputs failed uploading bucket=backtest key=XLF_2019-02-28_tdputs ex=Could not connect to the endpoint URL: "http://0.0.0.0:9000/backtest/XLF_2019-02-28_tdputs"
2019-03-01 02:01:52,455 - celery.app.trace - INFO - Task ae.work_tasks.get_new_pricing_data.get_new_pricing_data[0c8d4817-c84a-4dff-8550-724083d216f6] succeeded in 33.23137504400802s: None
2019-03-01 02:02:02,134 - ae.work_tasks.publish_pricing_update - ERROR - ticker=XLP-tdputs failed uploading bucket=backtest key=XLP_2019-02-28_tdputs ex=Could not connect to the endpoint URL: "http://0.0.0.0:9000/backtest/XLP_2019-02-28_tdputs"
2019-03-01 02:02:02,179 - ae.work_tasks.custom_task - INFO - on_success label=[ticker=XLP] - task_id=eb2e14bb-e8d5-4af2-b715-b0ce708278d7
2019-03-01 02:02:02,179 - celery.app.trace - INFO - Task ae.work_tasks.get_new_pricing_data.get_new_pricing_data[eb2e14bb-e8d5-4af2-b715-b0ce708278d7] succeeded in 37.829348045022925s: None