Price Feed Script for BitShares

Installation

Installation using Docker

Build docker image:

cd ~
git clone https://github.com/Zapata/bitshares-pricefeed.git
cd bitshares-pricefeed
docker build -t bitshares-pricefeed .

Alternative manual installation on Ubuntu 16.04 LTS or similar.

Install globally:

cd ~
git clone https://github.com/Zapata/bitshares-pricefeed.git
cd bitshares-pricefeed
python setup.py install

Or in a dedicated virtualenv environment:

cd ~
git clone https://github.com/Zapata/bitshares-pricefeed.git
cd bitshares-pricefeed
pip install virtualenv 
virtualenv -p python3 wrappers_env/ 
source wrappers_env/bin/activate
python setup.py install

Create the configuration file:

You can generate a sample configuration file using:

bitshares-pricefeed create

Or copy one from the examples directory:

File	Description
default	Default configuration generated by `create` command. Use to publish all the main Bitshares assets.
bsip42	Sample configuration to publish a 'target price' compliant with BSIP42. See below for more information.
hero	HERO asset publication configuration.
hertz	HERTZ asset publication configuration.
XCD	XCD asset publication configuration.
stocks	Sample configuration to publish Bitshares NASDAQ stocks trackers.
composite	Showcase the Composite source feature (see below)

Add a feed producer name to the config.yml file just created:

vim config.yml
# The producer name(s)
producer: your_witness_name

Manual publication

IF you use docker image you can run:

docker run -v /path/to/config.yml:/config/config.yml bitshares-pricefeed update --active-key=XXXXXXX

If you installed bitshares-pricefeed locally use:

bitshares-pricefeed update --active-key=XXXXXXX

See help for the full description of the command.

Above solution will pass active key credentials on the command line, if you want to reuse a pybitshares wallet see instruction in Use pybitshares encrypted wallet.

Schedule publication

Using cron:

$ crontab -e
ACTIVE_KEY="XXXXXXXXXXXXXXXX"

0,15,30,45 * * * * docker run -v /path/to/config.yaml:/config/config.yaml bitshares-pricefeed update --skip-critical --active-key=$ACTIVE_KEY

Help

$ bitshares-pricefeed --help
Usage: bitshares-pricefeed [OPTIONS] COMMAND [ARGS]...

Options:
  --configfile TEXT
  --node <wss://host:port>  Node to connect to
  --help                    Show this message and exit.

Commands:
  addkey  Add a private key to the wallet
  create  Create config file
  update  Update price feed for assets

$ bitshares-pricefeed update --help
Usage: bitshares-pricefeed update [OPTIONS] [ASSETS]...

  Update price feed for assets

Options:
  --dry-run                       Only compute prices and print result, no
                                  publication.
  --active-key WIF                Active key to be used to sign transactions.
  --confirm-warning / --no-confirm-warning
                                  Need for manual confirmation of warnings
  --skip-critical / --no-skip-critical
                                  Skip critical feeds
  --help                          Show this message and exit.

Sources

The following data sources are currently available:

Name	Status	Assets type	API Key	Description
AEX	OK	Crypto	No	last and volume (in quote currency) from CEX ticker api with 15 sec delay
AlphaVantage	OK	FIAT, Stocks, BTC	Yes	last from unknown source for currencies and from iex for stocks. volume only for stocks (in nb of shares).
Biki	OK	Crypto	No	last and volume (in quote currency) from CEX ticker api
Big.One	OK	Crypto	Yes	bid/ask average and volume (in quote currency) from bulk CEX ticker API.
Binance	OK	Crypto	No	last and volume (in quote currency) from CEX ticker api
BitcoinAverage	OK	Crypto	No	Use APIv2, get last and volume from an average of multiple exchanges.
Bitcoin Venezuela	OK	Crypto	No	ticker from api with 15 minutes delay, no volume
BitsharesFeed	OK	Crypto (MPA)	No	current feed price in Bitshares DEX, no volume.
BitsharesOrderbook	OK	Crypto	No	measure orderbook depth on Bitshares DEX
Bitstamp	OK	Crypto	No	last and volume (in quote currency) from CEX ticker api
Bittrex	OK	Crypto	No	last and volume (in quote currency) from summary api (bulk)
Coinbase	OK	Crypto	No	Use Coinbase Pro (ex GDAX) ticker api to get last and 24h volume.
Coincap	Fails, to be fixed	ALTCAP & ALTCAP.X	No	use provided market cap, no volume
CoinEgg	OK	Crypto	No	last and volume (in quote currency) from CEX ticker api
CoinGecko	OK	Crypto	No	volume weighted price, sum of market volume.
Coinmarketcap	Warn	Crypto	No	volume weighted average of all prices reported at each market, volume in USD, 5 minutes delay (see https://coinmarketcap.com/faq/). V1 API will be closed December 4th, 2018.
CoinmarketcapPro	OK	Crypto	Yes	volume weighted average of all prices reported at each market, volume in quote, 1 minutes delay. Use v2 api.
Currencylayer	Fails, to be fixed	FIAT, BTC	Yes	ticker from api, only USD as base and hourly updated with free subscription, no volume info. From various source (https://currencylayer.com/faq)
CoinTiger	OK	Crypto	No	last and volume (in quote currency) from summary api (bulk)
Fixer	OK	FIAT	Yes	Very similar to CurrencyLayer, ticker from api, daily from European Central Bank, only EUR with free subscription, no volume info.
GateIO	OK	Crypto	No	last price and volume (in quote currency) from CEX API in realtime
Graphene	OK	Crypto, FIAT, Stocks	No	last and volume (in quote currency) from Bitshares DEX in realtime
HitBTC	OK	Crypto	No	last price and volume (in quote currency) from CEX API in realtime
Huobi	OK	Crypto	No	close price and volume (in quote currency) from CEX API in realtime
Huobi OTC	OK	Crypto	No	mid price from CEX OTC market in realtime
IEX	Fails, to be fixed	Stocks	No	last ("IEX real time price", "15 minute delayed price", "Close" or "Previous close") and volume.
IndoDax	OK	Crypto	No	last and volume (in quote currency) from CEX ticker API.
Kraken	OK	Crypto	No	last and volume (in quote currency) from CEX ticker API.
LBank	OK	Crypto	No	last and volume (in quote currency) from CEX API in realtime
MagicWallet	OK	BITCNY/CNY	Yes	BITCNY/CNY ratio from deposti/withdraw on MagicWallet.
OkCoin	Fails, to be fixed	Crypto	No	last and volume (in quote currency) from CEX API in realtime
OkEx OTC	OK	Crypto	No	mid price from CEX OTC market in realtime
OpenExchangeRates	OK	FIAT, BTC	Yes	ticker from api, only USD as base and hourly updated with free subscription, no volume info. From unknown sources except Bitcoin wich is from CoinDesk (https://openexchangerates.org/faq#sources)
Poloniex	OK	Crypto	No	last and volume (in quote currency) from CEX API in realtime
Quantl	OK	Commodities	Yes	daily price from London Bullion Market Association (LBMA), no volume
RobinHood	Fails, to be fixed	Stocks	No	last, no volume, from unknown source in real time
WorldCoinIndex	OK	Crypto	Yes	volume weighted price, sum of market volume.
ZB	OK	Crypto	No	last and volume (in quote currency) from CEX API in realtime

Special sources:

Manual

A manual source is available to inject some manually set / constand data to the source feed. This could be usefull for:

testing to avoid connection to a third party datasource.
inject a static rate like BitUSD/USD

Example:

exchanges:
  manual:
    klass: Manual
    feed:
      USD:
        BTS:
          price: 42
          volume: 1

Composite

A composite source could be used to group multiple sources together in order to aggregate them using a specific fomula.

Aggregation types formulas could be:

min: select the minimum value for each pairs
max: select the maximum value for each pairs
mean: compute the mean price of all the pairs and sum the volume.
weighted_mean: compute the volume weighted mean price of all the pairs and sum the volume.
median: compute the median price of all the pairs and sum the volume.
first_valid: select the first pairs where the source match an ordered list of sources.

The main use cases is if you want to retrieve a pair like BTC/USD from multiples sources (worldcoinindex, bitcoinaverage, coinmarketcap), but you want to use only one of the value in all your computations. The same apply for FIAT exchange rates from (fixer, currencylayer, openexchangerate, ...).

See example configuration.

Algorithm Based Assets

Some assets use a computed formula as price like HERO or HERTZ. Those are implemented with a source class that return the computed value, usually with USD as quote.

Target price mode (BSIP42)

In order to better maintain the peg of Smartcoins, it might be necessary to apply a negative feedback to the computed price. The published price is no more the market price but a target price. See BSIP42 for more information.

This could be enabled setting the target_price_algorithm option on an asset publication:

assets:
    CNY:
        target_price_algorithm: 'adjusted_feed_price'
        target_price_adjustment_scale: 1.2

The algorithm used and available are under constant development by Bitshares witnesses, so please look at examples/bsip42.yaml for the latest documentation on the possible algorithms and their parameters.

Price threshold (BSIP76)

Price sources could be manipulated, so a price publisher may want to ensure that the price do not go below a threshold. See BSIP76.

This could be enabled setting the price_threshold option on an asset publication configuration:

assets:
    USD:
        price_threshold: 0.0350
    CNY:
        price_threshold: 0.2200

See examples/bsip76.yaml as a full example.

Loopholes protection (BAIP2)

In order to avoid loopholes, BAIP2 propose to use as feed price the highest between the current price and the two-day moving average price.

This could be enabled setting the price_threshold option on an asset publication configuration, or in the default asset configuration section:

default:
    loopholes_protection_days: 2

assets:
    USD:
        loopholes_protection_days: 3

In that case, all the assets will have the loopholes protection activated, and will use a two days moving average. There is an exception for USD, that will use a three day moving average.

In order to compute the moving average the previously computed prices (before adjustment) will be used. Those prices should be first saved in a storage then they will be loaded to compute the average. The storage mechanism should be configured, as there is no default option. See below for the detailed explanation.

See also examples/baip2.yaml as a working example.

History storage configuration

To store and load the computed prices, multiple storage macanism option are implemented:

file
database

File storage

To use a 'file' storage you should add in the configuration:

history:
    klass: FileHistory
    dirname: prices_db

This will save all the computed prices in the relative prices_db folder using a CSV file per asset.

Database storage

history:
    klass: SqlHistory
    url: "postgres+pypostgresql://user:pass@localhost:5432/postgres"
    schema_name: 'price_feed'
    table_name: 'raw_prices'

Any kind of database supported by SQLAlchemy can be used. However, the appropriate Python drivers should be installed. As an example to use Postgresql you should first run:

pip install SQLAlchemy py-postgresql

As this is a popular option you can also install it using the history_db_postgresql feature:

pip install .[history_db_postgresql]

See SQLAlchemy documentation for more details on how to connect to the database.

By default the script will use a table raw_prices in the price_feed schema. Schema and table names are configurable, they will be automatically created if they does not exists.

Use of encrypted wallet

Initialize wallet and enter credentials:

$ bitshares-pricefeed addkey

You will need to enter your cli wallet encryption passphrase. If you don't have a pybitshares wallet, yet, one will be created:

Wallet Encryption Passphrase:
Repeat for confirmation:

You will need to enter your Private Key (Active key) here. Hit enter the second time it asks you.

Private Key (wif) [Enter to quit]:

Then you can run the feed update without any argument, it will prompt for the Wallet Encryption Passphrase.

$ bitshares-pricefeed update
Current Wallet Passphrase:

If you don't want to type the Wallet Encryption Passphrase each time, use UNLOCK environment variable.

Example in cron:

$ crontab -e

SHELL=/bin/bash
PATH=/home/ubuntu/bin:/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
UNLOCK="PASSWD"

0,15,30,45 * * * * bitshares-pricefeed --configfile /home/ubuntu/config.yml --skip-critical --no-confirm-warning update >> /var/log/bitshares-pricefeed.log 2>&1

Or with Docker image:

docker run -v /path/to/config:/config -v /path/to/wallet:/root/.local/share/bitshares -e UNLOCK=PASS bitshares-pricefeed

Development

To run tests you need get API keys for the providers, and register them as environment variables:

export QUANDL_APIKEY=
export OPENEXCHANGERATE_APIKEY=
export FIXER_APIKEY=
export CURRENCYLAYER_APIKEY=
export ALPHAVANTAGE_APIKEY=
export WORLDCOININDEX_APIKEY= 
export MAGICWALLET_APIKEY=
export COINMARKETCAP_APIKEY=

To run all tests use: PYTHONPATH=. pytest.

To run a specific test: PYTHONPATH=. pytest -k bitcoinvenezuela.

IMPORTANT NOTE

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

Zapata / bitshares-pricefeed

readme