search

scrtlabs / catalyst

An Algorithmic Trading Library for Crypto-Assets in Python
http://enigma.co
Apache License 2.0
2.47k stars 721 forks source link

Papermode bug: pandas `60T` resampling vs ccxt `1h` missmatch #114

Closed vonpupp closed 6 years ago

vonpupp commented 6 years ago

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Now that you know a little about me, let me tell you about the issue I am having:

60T resampling vs 1h ccxt missmatch

When I try to run with hourly resampling using papermode it breaks. Same applies to 2h resample.

[2017-12-22 15:36:05.321559] INFO: exchange_algorithm: got totals from exchanges, cash: 868.027881268 positions: 91.7521953708 [2017-12-22 15:36:05.324173] WARNING: DataPortalExchange: get history attempt 0: Request failed: '60m' [2017-12-22 15:36:10.327806] WARNING: DataPortalExchange: get history attempt 1: Request failed: '60m' [2017-12-22 15:36:15.333984] WARNING: DataPortalExchange: get history attempt 2: Request failed: '60m' [2017-12-22 15:36:20.335723] WARNING: DataPortalExchange: get history attempt 3: Request failed: '60m' [2017-12-22 15:36:25.339672] WARNING: DataPortalExchange: get history attempt 4: Request failed: '60m' [2017-12-22 15:36:30.343947] WARNING: DataPortalExchange: get history attempt 5: Request failed: '60m' Traceback (most recent call last): File "d_abba_papermode.py", line 225, in live_graph=False File "/home/av/repos/example/catalyst/catalyst/utils/run_algo.py", line 563, in run_algorithm stats_output=stats_output File "/home/av/repos/example/catalyst/catalyst/utils/run_algo.py", line 346, in _run overwrite_sim_params=False, File "/home/av/repos/example/catalyst/catalyst/algorithm.py", line 724, in run for perf in self.get_generator(): File "/home/av/repos/example/catalyst/catalyst/gens/tradesimulation.py", line 224, in transform for capital_change_packet in every_bar(dt): File "/home/av/repos/example/catalyst/catalyst/gens/tradesimulation.py", line 137, in every_bar handle_data(algo, current_data, dt_to_use) File "/home/av/repos/example/catalyst/catalyst/utils/events.py", line 216, in handle_data dt, File "/home/av/repos/example/catalyst/catalyst/utils/events.py", line 235, in handle_data self.callback(context, data) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_algorithm.py", line 648, in handle_data self._handle_data(self, data) File "/home/av/repos/example/strategies/family/abba.py", line 154, in default_handle_data frequency=context.candles_frequency) File "catalyst/_protocol.pyx", line 120, in catalyst._protocol.check_parameters.call.assert_keywords_and_call File "catalyst/_protocol.pyx", line 646, in catalyst._protocol.BarData.history File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 114, in get_history_window ffill) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window attempt_index + 1) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window attempt_index + 1) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window attempt_index + 1) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window attempt_index + 1) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window attempt_index + 1) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 93, in _get_history_window error=e catalyst.exchange.exchange_errors.ExchangeBarDataError: Unable to retrieve bar data: history, giving up after 5 attempts: Request failed: '60m'



Here is how you can reproduce this issue on your machine:

## Reproduction Steps

1. Try to run a strategy with `60T` resample on paper trading mode.
...

## What steps have you taken to resolve this already?

I have been looking directly into the code to find the mapping of the re-sampling string (i.e: `60T`) to the ccxt frequency parameter (i.e: `1H`, `1W`, `1M`), which has a different notation. Since my father got hospitalized recently I don't know if I will have the time to submit a PR. I will try if possible.

Sincerely,
V.
fredfortier commented 6 years ago

We’ve done some adjustments in the develop branch specifically to support H, W, not just T and D. This will be fixed in the next release coming out soon. Catalyst uses the Pandas time offset convention. On Fri, Dec 22, 2017 at 7:49 AM Vonpupp notifications@github.com wrote:

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment: Environment

  • Operating System: (Linux 4.13.11-1-ARCH #1 SMP PREEMPT Thu Nov 2 10:25:56 CET 2017 x86_64 GNU/Linux)
  • Python Version: 2.7.14
  • Python Bitness: 64
  • How did you install Catalyst: pip
  • Python packages: $ pip freeze:

agate==1.6.0 agate-dbf==0.2.0 agate-excel==0.2.1 agate-sql==0.5.2 alabaster==0.7.10 alembic==0.9.6 Babel==2.5.1 backports-abc==0.5 backports.functools-lru-cache==1.4 backports.shutil-get-terminal-size==1.0.0 bcolz==0.12.1 bleach==2.1.1 boto3==1.4.8 botocore==1.8.13 Bottleneck==1.2.1 ccxt==1.10.382 certifi==2017.11.5 chardet==3.0.4 click==6.7 configparser==3.5.0 contextlib2==0.5.5 csvkit==1.0.2 cycler==0.10.0 cyordereddict==1.0.0 Cython==0.27.3 dbfread==2.0.7 decorator==4.1.2 docutils==0.14 empyrical==0.2.1 enigma-catalyst==0.4.0+1.g94583c26 entrypoints==0.2.3 enum34==1.1.6 et-xmlfile==1.0.1 functools32==3.2.3.post2 future==0.16.0 futures==3.2.0 haikunator==2.1.0 html5lib==1.0b10 idna==2.6 imagesize==0.7.1 intervaltree==2.1.0 ipdb==0.10.3 ipykernel==4.6.1 ipython==5.5.0 ipython-genutils==0.2.0 ipywidgets==7.0.5 isodate==0.6.0 jdcal==1.3 Jinja2==2.10 jmespath==0.9.3 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.1.0 jupyter-console==5.2.0 jupyter-core==4.4.0 leather==0.3.3 Logbook==1.1.0 lru-dict==1.1.6 Mako==1.0.7 MarkupSafe==1.0 matplotlib==2.1.0 mistune==0.8.1 multipledispatch==0.4.9 names==0.3.0 nbconvert==5.3.1 nbformat==4.4.0 networkx==2.0 notebook==5.2.1 numexpr==2.6.4 numpy==1.13.3 openpyxl==2.4.9 pandas==0.19.2 pandas-datareader==0.5.0 pandocfilters==1.4.2 parsedatetime==2.4 pathlib2==2.3.0 patsy==0.4.1 petname==2.2 pexpect==4.3.0 pickleshare==0.7.4 prompt-toolkit==1.0.15 ptyprocess==0.5.2 pyfolio==0.8.0 Pygments==2.2.0 pyparsing==2.2.0 python-dateutil==2.6.1 python-editor==1.0.3 python-slugify==1.2.4 pytimeparse==1.1.7 pytz==2017.3 pyzmq==16.0.3 qtconsole==4.3.1 requests==2.18.4 requests-file==1.4.2 requests-ftp==0.3.1 s3transfer==0.1.12 scandir==1.6 scikit-learn==0.19.1 scipy==1.0.0 seaborn==0.8.1 simplegeneric==0.8.1 singledispatch==3.4.0.3 six==1.11.0 snowballstemmer==1.2.1 sortedcontainers==1.5.7 Sphinx==1.6.5 sphinxcontrib-websupport==1.0.1 SQLAlchemy==1.1.15 statsmodels==0.8.0 subprocess32==3.2.7 TA-Lib==0.4.10 tables==3.4.2 terminado==0.7 testpath==0.3.1 toolz==0.8.2 tornado==4.5.2 traitlets==4.3.2 transitions==0.6.2 typing==3.6.2 Unidecode==0.4.21 urllib3==1.22 wcwidth==0.1.7 webencodings==0.5.1 widgetsnbextension==3.0.8 xlrd==1.1.0

Now that you know a little about me, let me tell you about the issue I am having: 60T resampling vs 1h ccxt missmatch

When I try to run with hourly resampling using papermode it breaks. Same applies to 2h resample.

  • What did you expect to happen?: I expect to be able to use various timeframes: 30min, 60min/1h, 120min/2h, etc as expressed [here]

    '1m': '1minute',
'1h': '1hour',
'1d': '1day',
'1M': '1month',
'1y': '1year',

  • What happened instead?: it crashed with the following output:

[2017-12-22 15:35:52.882634] INFO: run_algo: running algo in paper-trading mode [2017-12-22 15:35:58.458045] INFO: exchange_bundle: pricing data for [u'btc_usdt'] not found in range 2015-03-01 00:00:00+00:00 to 2017-12-21 00:00:00+00:00, updating the bundles. [====================================] Ingesting daily price data for btc_usdt on poloniex: 100% [2017-12-22 15:36:00.181265] INFO: Loader: Cache at /home/av/.catalyst/data/treasury_curves.csv does not have data from 1990-01-02 00:00:00+00:00 to 2017-12-21 00:00:00+00:00. [2017-12-22 15:36:00.181430] INFO: Loader: Downloading treasury data for 'btc_usdt'. [2017-12-22 15:36:04.569014] WARNING: Loader: Still don't have expected data after redownload! [2017-12-22 15:36:04.581458] INFO: exchange_algorithm: initialized trading algorithm in live mode [2017-12-22 15:36:04.752854] INFO: abba_test: STARTING SIMULATION. Candles sample rate: 60 Candles frequency: 60T Candles buffer size: 48

[2017-12-22 15:36:05.321559] INFO: exchange_algorithm: got totals from exchanges, cash: 868.027881268 positions: 91.7521953708 [2017-12-22 15:36:05.324173] WARNING: DataPortalExchange: get history attempt 0: Request failed: '60m' [2017-12-22 15:36:10.327806] WARNING: DataPortalExchange: get history attempt 1: Request failed: '60m' [2017-12-22 15:36:15.333984] WARNING: DataPortalExchange: get history attempt 2: Request failed: '60m' [2017-12-22 15:36:20.335723] WARNING: DataPortalExchange: get history attempt 3: Request failed: '60m' [2017-12-22 15:36:25.339672] WARNING: DataPortalExchange: get history attempt 4: Request failed: '60m' [2017-12-22 15:36:30.343947] WARNING: DataPortalExchange: get history attempt 5: Request failed: '60m' Traceback (most recent call last): File "d_abba_papermode.py", line 225, in live_graph=False File "/home/av/repos/example/catalyst/catalyst/utils/run_algo.py", line 563, in run_algorithm stats_output=stats_output File "/home/av/repos/example/catalyst/catalyst/utils/run_algo.py", line 346, in _run overwrite_sim_params=False, File "/home/av/repos/example/catalyst/catalyst/algorithm.py", line 724, in run for perf in self.get_generator(): File "/home/av/repos/example/catalyst/catalyst/gens/tradesimulation.py", line 224, in transform for capital_change_packet in every_bar(dt): File "/home/av/repos/example/catalyst/catalyst/gens/tradesimulation.py", line 137, in every_bar handle_data(algo, current_data, dt_to_use) File "/home/av/repos/example/catalyst/catalyst/utils/events.py", line 216, in handle_data dt, File "/home/av/repos/example/catalyst/catalyst/utils/events.py", line 235, in handle_data self.callback(context, data) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_algorithm.py", line 648, in handle_data self._handle_data(self, data) File "/home/av/repos/example/strategies/family/abba.py", line 154, in default_handle_data frequency=context.candles_frequency) File "catalyst/_protocol.pyx", line 120, in catalyst._protocol.check_parameters.call.assert_keywords_and_call File "catalyst/_protocol.pyx", line 646, in catalyst._protocol.BarData.history File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 114, in get_history_window ffill) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window attempt_index + 1) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window attempt_index + 1) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window attempt_index + 1) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window attempt_index + 1) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 88, in _get_history_window attempt_index + 1) File "/home/av/repos/example/catalyst/catalyst/exchange/exchange_data_portal.py", line 93, in _get_history_window error=e catalyst.exchange.exchange_errors.ExchangeBarDataError: Unable to retrieve bar data: history, giving up after 5 attempts: Request failed: '60m'

Here is how you can reproduce this issue on your machine: Reproduction Steps

  1. Try to run a strategy with 60T resample on paper trading mode. ...

What steps have you taken to resolve this already?

I have been looking directly into the code to find the mapping of the re-sampling string (i.e: 60T) to the ccxt frequency parameter (i.e: 1H, 1W, 1M), which has a different notation. Since my father got hospitalized recently I don't know if I will have the time to submit a PR. I will try if possible.

Sincerely, V.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/114, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QmKGL7THKZMW27F4qWZyNW3ET222ks5tC8-SgaJpZM4RLK8E .

vonpupp commented 6 years ago

Awesome @fredfortier, thank you very much for your quick response as usual.

From what I infer on your answer, you included other resampling methods internally to support something like 2H. Is that correct? Instead of following the pandas convention (120T). Could you please exemplify how to do then a 2 hour resample?

Would it be possible to link the commits to this issue? I am curious to know more about catalyst internals. I tried to fix this myself since it seems reasonably easy but I couldn't find the related code section in the time I spent trying (which unfortunately wasn't much due to personal issues at the moment).

fredfortier commented 6 years ago

I’ll double check today and link it. There is one more thing to do first. 120T will work of Bittrex supports 120 Minutes. On Sat, Dec 23, 2017 at 9:25 AM Vonpupp notifications@github.com wrote:

Awesome @fredfortier https://github.com/fredfortier, thank you very much for your quick response as usual.

So I should use something like 120T for 2H equivalent, is that correct?

Would it be possible to link the commit message to this issue? I am curious to know more about catalyst internals. I tried to fix this myself since it seems reasonably easy but I couldn't find the related code section in the time I spent trying (which unfortunately wasn't much due to personal issues at the moment).

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/114#issuecomment-353738107, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-Qk8LK6AW6fO__mA-GJA8OVLAt8P5ks5tDTdrgaJpZM4RLK8E .

vonpupp commented 6 years ago

Thank you very much @fredfortier.

vonpupp commented 6 years ago

Hi guys,

I was away due to family medical issues and I just got the chance to test this out. It doesn't seem to be working for me when I use 120T resampling. Herewith the error:

Traceback (most recent call last):
  File "d_abba_papermode.py", line 225, in <module>
    live_graph=False
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/utils/run_algo.py", line 527, in run_algorithm
    stats_output=stats_output
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/utils/run_algo.py", line 308, in _run
    overwrite_sim_params=False,
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/exchange_algorithm.py", line 297, in run
    data, overwrite_sim_params
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/algorithm.py", line 724, in run
    for perf in self.get_generator():
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/gens/tradesimulation.py", line 224, in transform
    for capital_change_packet in every_bar(dt):
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/gens/tradesimulation.py", line 137, in every_bar
    handle_data(algo, current_data, dt_to_use)
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/utils/events.py", line 216, in handle_data
    dt,
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/utils/events.py", line 235, in handle_data
    self.callback(context, data)
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/exchange_algorithm.py", line 738, in handle_data
    self._handle_data(self, data)
  File "/home/av/repos/example/strategies/family/abba.py", line 154, in default_handle_data
    frequency=context.candles_frequency)
  File "catalyst/_protocol.pyx", line 120, in catalyst._protocol.check_parameters.__call__.assert_keywords_and_call
  File "catalyst/_protocol.pyx", line 646, in catalyst._protocol.BarData.history
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/exchange_data_portal.py", line 95, in get_history_window
    ffill))
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/redo/__init__.py", line 162, in retry
    return action(*args, **kwargs)
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/exchange_data_portal.py", line 69, in _get_history_window
    ffill)
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/exchange_data_portal.py", line 218, in get_exchange_history_window
    False)
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/exchange.py", line 510, in get_history_window
    end_dt=end_dt if not is_current else None,
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 391, in get_candles
    params={}
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/ccxt/bitfinex.py", line 516, in fetch_ohlcv
    'timeframe': self.timeframes[timeframe],
KeyError: '120m'
vonpupp commented 6 years ago

Hello @fredfortier,

Sorry to ask but is there any news regarding this issue?

Thanks.

fredfortier commented 6 years ago

Yes, we've been addressing this on a separate branch along with other further standardization / integration with CCXT. We're planning to release by the end of this week.

On Wed, Jan 31, 2018 at 9:11 AM Vonpupp notifications@github.com wrote:

Hello @fredfortier https://github.com/fredfortier,

Sorry to ask but is there any news regarding this issue?

Thanks.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/114#issuecomment-361943220, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QidmNe5IQFzOQvDcz30WLqQw5T-Uks5tQHR6gaJpZM4RLK8E .

vonpupp commented 6 years ago

Awesome news @fredfortier. Thank you very much for the update.

fredfortier commented 6 years ago

Sorry for the delay, we've been bundling up a few items which require refactoring into a separate branch.

On Wed, Jan 31, 2018 at 5:04 PM Vonpupp notifications@github.com wrote:

Awesome news @fredfortier https://github.com/fredfortier. Thank you very much for the update.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/114#issuecomment-362086377, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QrJHhwz4TBAtINdzcmD74fB6CM48ks5tQONegaJpZM4RLK8E .

vonpupp commented 6 years ago

No problem. I don't want to be annoying either. Thankfully I could postpone a bit the starting of the second phase so I haven't been impacted too much by this issue, so everything is good. Keep calm and hodl =P. Thank you very much for your awesome support.

vonpupp commented 6 years ago

Hello @fredfortier, I really don't want to be annoying, but do you think is there something I could help you out to try to have this feature by this week?

Thank you very much.

fredfortier commented 6 years ago

We're pushing a release now but this is not included. The plan is to release the branch associated with this on Friday or Monday. I really apologize for the delay. Do you not have a workaround for this though? For example, can you request 1H candles and resample? I can help you with a workaround in the meantime if it's possible.

vonpupp commented 6 years ago

If you could give me some help it would be awesome. I have noticed some really weird behavior on paper trading while fetching prices. Would it be possible for us to talk more about this? I think I have found a couple of bugs which I would like to discuss more about. In the meantime I will create a minimal example to see if the bugs are also available on that example.

Please ping me out on discord when possible. As always thank you very much Fred!

fredfortier commented 6 years ago

Happy to help. Feel free to post your minimal example where with the candles which you are trying to fetch. I'll see if a workaround is possible. The issue again is that we are going directly to each exchange to fetch candles in live / paper trading mode. Unfortunately, exchanges don't support every timeframe so resampling is sometimes necessary.

vonpupp commented 6 years ago

Dear @fredfortier,

Herewith a minimal example as you proposed. It is almost identical to my prior example. If you change the MODE attribute to papermode you will notice that it runs fine re-sampling every 2 hours. I need to accomplish something similar but on paper mode.

I thought I could use the same code base for both, but I am having the problems related to this issue. Ideally I should only change the CANDLES_SAMPLE_RATE and CANDLES_FREQUENCY attributes only to match the supported time frames by the exchange.

Could you please help me out? Thank you very much in advance.

fredfortier commented 6 years ago

Thanks, will look into it today. On Thu, Feb 8, 2018 at 4:58 PM Vonpupp notifications@github.com wrote:

Dear @fredfortier https://github.com/fredfortier,

Herewith a minimal example https://gist.github.com/ac7c909ed80a3ec020846d5475bfe183 as you proposed. It is almost identical to my prior example. If you change the MODE attribute to papermode you will notice that it runs fine re-sampling every 2 hours. I need to accomplish something similar but on paper mode.

I thought I could use the same code base for both, but I am having the problems related to this issue. Ideally I should only change the CANDLES_SAMPLE_RATE and CANDLES_FREQUENCY attributes only to match the supported time frames by the exchange.

Could you please help me out? Thank you very much in advance.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/114#issuecomment-364301062, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-Qra2Z_IveBrwhXUmYRUdMBN-64WWks5tS5gcgaJpZM4RLK8E .

vonpupp commented 6 years ago

Awesome, thank you very much Fred! I am really thankful for your help indeed.

vonpupp commented 6 years ago

Hello again Fred,

This seems to still be failing, I am not sure if it is something specific to my environment setup, but I don't think so. I am using enigma-catalyst==0.5.3+85.g4a794aa

What I am doing:

ipdb> context.candles_open
2018-03-08 13:45:00+00:00    9835.9
2018-03-08 13:46:00+00:00    9843.0
...
Freq: T, Name: TradingPair(98871 [btc_usd]), dtype: float64

Up until now everything seems fine, however when I try to get candles on other time frames greater than 30T I get errors.

Example 1: working fine, using 30T

ipdb> result = data.history(context.coin_pair, 'open', bar_count=context.candles_buffer_size, frequency='30T')
ipdb> result
2018-03-07 15:00:00+00:00    10661.000000
2018-03-07 15:30:00+00:00    10682.000000
...
Freq: 30T, Name: TradingPair(98871 [btc_usd]), dtype: float64

Example 2: not working, using 1H

ipdb> result = data.history(context.coin_pair, 'open', bar_count=context.candles_buffer_size, frequency='1H')
ipdb> result
Series([], Freq: H, Name: TradingPair(98871 [btc_usd]), dtype: float64)

Example 3: not working either, using 60T

ipdb> result = data.history(context.coin_pair, 'open', bar_count=context.candles_buffer_size, frequency='60T')
*** catalyst.exchange.exchange_errors.UnsupportedHistoryFrequencyError: bitfinex does not support candle frequency 60T, please choose from: ['6H', '15T', '1M', '2W', '1D', '5T', '12H', '1W', '30T', '1T', '3H', '1H'].

It has been more than 3 months since I opened this issue. This issue unfolded into issues #227 and #253, and despite progress has been done and these are now solved, this issue persists. This is becoming a real problem now for me since I couldn't deliver on time and it is now impacting me heavily. Could you please help me out? How should I proceed with this?

Should I re-sample as part of the user code? Something like get 30T candles and re-sample directly on the pandas df? On this example I am using 2H for testing purposes.

ipdb> result = data.history(context.coin_pair, 'open', bar_count=context.candles_buffer_size, frequency='30T')
ipdb> result.resample('120T').mean()
2018-03-07 14:00:00+00:00    10682.000000
2018-03-07 16:00:00+00:00    10171.300000
...
Freq: 120T, Name: TradingPair(98871 [btc_usd]), dtype: float64

Would this yield into accurate data? Wouldn't this give problems with other internal Catalyst calculations?

Thank you very much.

WPsites commented 6 years ago

In true hack style, with great help from the other 'fixes' I've got hourly working on bitfinex and im sure it will work the same on binance etc.

I've edited 2 files, I will show you snippets of where iv edited which you could replicate to get things working if you so needed... i wont show you snippets because i can't paste any in here as code because the code function breaks once you put code in here ffs. I'll try Summerise

in catalyst/exchange/ccxt/ccxt_exchange.py I added:

elif data_frequency == 'hourly' and not freq.endswith('D'):

In the get_candle_frequencies() method

in catalyst/exchange/utils/datetime_utils.py i changed

TODO: standardize time aliases to avoid any mapping

    #unit = 'd' if unit == 'D' else 'm' #simoncommentedout
    unit = 'd' if unit == 'D' else 'h' if unit == 'H' else 'm'

Then also down in the get_frequency() method i added under another todo bit, after the other if unit.lowers...

elif unit.lower() == 'h':
    #candle_size = candle_size * 60
    alias = '{}H'.format(candle_size)
    unit = 'H'
    data_frequency = 'hourly'

and im now able to get data for 3hr etc... hope thats of some use. Not sure how that transfers to backtesting, im just doing this while paper trading right now.... mark down has added all sorts of styling to this post... i give up

WPsites commented 6 years ago

I've just gone back to doing some back testing and the code changes don't break backtesting but you do have to change the frequency in your algo to the 180T way of specifying time rather than 3h because otherwise it's expecting hourly data to look at and that's not available option with ingest

vonpupp commented 6 years ago

Thanks WPsites,

I would prefer not to modify Catalyst core code due to unexpected possible behaviour. For the time being I am manually resampling on top of Catalyst resample, however I noticed that the candle values does not match with tradingview, so it doesn't seem to be the same. Have you checked out if your patch are matching the candles on tradingview correctly?

I would suggest you to submit a PR (if you haven't yet) with your work so it can be reviewed by the core devs and included into Catalyst if everything is okay.

WPsites commented 6 years ago

I'd prefer not to make any changes to core but the alternative for me is just to not use catalyst as i'd come to a brickwall that meant i couldn't live/paper trade algos and i feel im too far down the road to turn back and choose another platform... and I'd like to make some money from some of my data ideas ;)

I'm no python coder and don't really know the catalyst code base so shied away from PR.. my changes weren't even in git

vonpupp commented 6 years ago

Thank you very much @WPsites.

I opened a PR277 for review, hopefully this feature will go into the develop branch soon so others can benefit too. I had to change a bit more than what you explained but your help pointed me into the right direction, so thank you.

frisitano commented 6 years ago

In the name of hacking a solution, which I have done, you can easily modify the timeframes dictionary of your ccxt exchange. For example adding an entry of '120m' : '2h' in this dictionary, https://github.com/ccxt/ccxt/blob/master/python/ccxt/bitfinex.py#L47, and then from within Catalyst using 120T.

vonpupp commented 6 years ago

Thank you @frisitano.

I remember to have seen that in the past. Hopefully the PR will work for other exchanges also and we can finally close this issue.

lenak25 commented 6 years ago

Fixed at version 0.5.5.