Add Exchanges with bigger universe (Cryptopia, YoBit, Novaexchange)

[avn3r]

Dear Catalyst Maintainers,

Currently, we have 3 exchanges which are among the most well know exchanges in terms of Volume and activity. This is great for more stable investing, but for those interested in daily and high interval trading a bigger universe will help capitalize in smaller coins that may lead to higher returns in shorter periods.

For this reason, I propose the addition of the following exchanges: 1) Cryptopia: 990 Trade Pairs 2) YoBit: 397 Trade Pairs 3) NovaExchange: 554 Trade Pairs

With the addition of three exchanges, we pretty much cover the entire Cryptocurrency market.

Sincerely, avn3r

[cyzanfar]

Would love to contribute in adding those exchanges. I am getting familiar with the whole library at the moment. Ready to tackle some of these features soon :)

[fredfortier]

We'll announce more details this week. But we are adding these three along with other exchanges. They'll be available at least for paper trading and live trading.

For backtesting, it depends on whether their APIs all have a historical data service with minute data. If so, we'll download it and make it available for backtesting. If not, we may have to chase down that data. Help with that would be welcome.

[cyzanfar]

That's great. Well, I'd love to do that: become a 'data curator' as well as contribute to the Catalyst application layer. Let me know how I can get started ;)

Thanks @fredfortier

[fredfortier]

@cyzanfar that would be super helpful! We've integrated the ccxt library which has an interface to multiple exchanges. Today, I'll compile a list of which exchanges have a usable historical data API. The reminder will require some type of work to collect the data in a CSV format.

[cyzanfar]

You got it! I'll be on the lookout for that list. Let me know I'm ready to code <3

[fredfortier]

Sorry for the delay, from what I can tell so far, here are the exchanges for which we do have historical data. We may be able to hit some more but this is what I have so far.

We are also actively trying to get the Bittrex data but don't have it yet. Their historical data api is limited to recent transactions.

Any help trying to locate other historical data sources would be appreciated.

[cyzanfar]

Great, thank you @fredfortier. Do you need help integrating some of the exchanges listed above to Catalyst? I could take a stab at one of them.

[ykgoon]

@fredfortier @cyzanfar I'm trying to get started contributing, this sounds like something that needs serious help but can't tell if it's appropriate for a starter.

Can anyone tell me where's a good place to start looking?

[fredfortier]

The area where we need the most help is locating historical data for these exchanges.

We're close to having most of the exchanges work in live and backtesting mode (currently unit testing with all of them). However, we have very little historical data for them.

I know that historical data packages are sometimes found on Reddit or other sources like this but we are not tracking them down in that way at the moment. If you can locate some of this data and package it in the attached format bittrex_bat_eth.csv.zip, we can make it available in our repository. We'd also need to make sure that this data is free of use.

CSV file specs: Here are the proposed required headers:

• symbol: symbol of the currency pair using the Catalyst convention (e.g. eth_btc)
• last_traded: the close date of the candle, use any date format which Pandas can parse (e.g. 2017-11-25 17:00)
• open: float
• high: float
• low: float
• close: float
• volume: float

[ykgoon]

My findings for NovaExchange so far: it probably requires a scaper tool.

POST

curl 'https://novaexchange.com/api/public/getohlc/' -H 'Host: novaexchange.com' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: https://novaexchange.com/market/BTC_DOGE/' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'X-CSRF-TOKEN: IjNhZDEyYjJhZDQ2ZDI4N2UzYWU2YzIyYWE0ZmViZmU2M2ZjNmY3NWYi.DRTcyg.Z-t7S0Y1KPEodm5S5-HgtrmrwiA' -H 'X-Requested-With: XMLHttpRequest' -H 'Cookie: session=379f4291-5081-481e-91ff-5a69826c6711; SERVERID=wsapp0702' -H 'Connection: keep-alive' --data 'marketid=159&marketname=BTC_DOGE&candlesize=100&history=14515200'

Sample JSON output. This sample covers a large time frame but minute-resolution can be done too.

JSON output can then be mapped to the desired CSV specs.

[davidaam]

I found minute historical data up to August 2017, I don't know if that's still useful. It seems very hard to find more recent data

[fredfortier]

Yes, I found that one too but nothing more recent yet. On Tue, Jan 9, 2018 at 8:05 PM David Atias notifications@github.com wrote:

I found minute historical data up to August 2017, I don't know if that's still useful. It seems very hard to find more recent data

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/33#issuecomment-356493908, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QmReU4d_--mF4kwRdebUTq69igvAks5tJDb-gaJpZM4Pcvpm .

[avn3r]

@fredfortier From my understanding, Cryptopia works using the CCXT or is it working nowadays with catalyst API. From my understanding, CCXT only works for paper trading and live trading, but that might be good enough for me for now. Would it be possible for you to guide me on a dummy/pseudo code example of how to paper trade with Cryptopia.

Basically, all I want is to iterate through all the coins in Cryptopia and purchases then similar to my simple_universe example but making a paper order.

Code: Get all coin Print their price make a small order.

Let me know if you more details or help making this work.

[fredfortier]

I have not tried running an algo with Cryptopia but it does have all the right features so it should work. It might just work like Binance and some other. You'd basically specify exchange='cryptopia' in your algo interface. I'll run some unit tests and get back to you as soon as I can. We've noticed that despite having CCXT as a common interface, which is incredibly useful, each exchange tend to have its quirks. We're focusing on the details at the moment to make sure that understand and deal with those reliably.

On Sat, Jan 13, 2018 at 8:27 PM Abner Ayala-Acevedo < notifications@github.com> wrote:

@fredfortier https://github.com/fredfortier From my understanding, Cryptopia works using the CCXT or is it working nowadays with catalyst API. From my understanding, CCXT only works for paper trading and live trading, but that might be good enough for me for now. Would it be possible for you to guide me on a dummy/pseudo code example of how to paper trade with Cryptopia.

Basically, all I want is to iterate through all the coins in Cryptopia and purchases then similar to my simple_universe example but making a paper order.

Code: Get all coin Print their price make a small order.

Let me know if you more details or help making this work.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/33#issuecomment-357481265, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QnQd1ZQOpm0sQlhfusmUd5p_LzZRks5tKVfkgaJpZM4Pcvpm .

[lacabra]

@abnera I started using your simple_universe in backtest mode, and when I switched to paper mode, I had to modify it to the following:

context.base_currency = 'btc'
context.coins = context.exchanges[context.exchange].assets
context.coins = [c for c in context.coins if c.quote_currency==context.base_currency]

The second line retrieves all the coins available on the exchange at a given time, and the third line filters that list of coins to being in the BTC base currency. I had no need for your context.universe variable. I am just using context.coins as an array of Catalyst symbol objects that you can pass to data.history() or placing orders, and that's all I need.

[avn3r]

@lacabra thanks for the tip, these might simplify things. Some people might still need the universe method because it also filters based on a look_back period. In my case, it backtested and looks back at the coin that existed 7 days prior the current back_test date.

However, for paper trading, no backtesting is needed but if you looking at history data you want to ensure that the coin existed 7 days ago for example.