Closed yohplala closed 2 years ago
Hi,
I have been delving a bit more in the topic, by reviewing code of CSVReader
.
I fail to see what a block
is and what provide them?
As Binance files are csv.zip files, I am not sure they are accepted as they are by CSVReader
, but i can confirm pd.read_csv
accepts them without need of additional args.
I can see pd.read_csv
is used by CSVReader
.
So in the end, I wonder what is supposed to do a block_parser
function.
I could guess it could be a dict from which mapping of column names from external data to nautilus is done, as well as mapping of dtype probably?
Please, where can I find information about what block_parser
is expected to do?
Thanks in advance for your help and sorry for these naive questions.
Hi @yohplala - sorry for the delayed response, and the lacking documentation here.
The block
is a chunk of data (in this case it will be a dataframe), which may be a whole file, or in the case where the files are large, a subset of the data.
The parsing function needs to return actual nautilus objects - so you need to convert your raw data in csv into nautilus Bar
objects. Another thing to note is that the parsing functions should return generators (this is mostly for other types of files ie text where a row may contain 0 or more actual data points - so a generator keeps this flexible to handle those situations)
If you update your parsing function to something like this, it should work:
def parse_binance_klines(block: pd.DataFrame, precision=5, bar_spec="1-MINUTE"):
df = pd.read_csv(file, **BINANCE_KLINES_CONF)
for _, row in df.iterrows():
yield Bar(
bar_type=BarType.from_str(f"{TestInstrumentProvider.adabtc_binance}-{bar_spec}-MID-EXTERNAL"),
open=Price(row["open"], precision=precision),
high=Price(row["high"], precision=precision),
low=Price(row["low"], precision=precision),
close=Price(row["close"], precision=precision),
volume=Quantity.from_str(str(row['quote_volume'])),
ts_event=millis_to_nanos(row['ts_open']),
ts_init=millis_to_nanos(row['ts_open']),
)
I've just hard coded the bar type in above, you might need to use a partial function if you're planning on loading in multiple bar types or underlyings and the bar spec is part of the filename, ie:
parser_5min = functools.partial(parse_binance_klines, bar_spec="5-MINUTE")
process_files(
glob_path=f"{DATA_DIR}/*5m*.zip",
reader=CSVReader(block_parser=5min_parser),
catalog=catalog,
)
Let me know if any of that isn't clear, or if you have any more questions?
Hi @limx0
Sorry I just see your answer as I have proposed my very 1st pull request :) #501
So, I had troubles with the fact that Binance official CSV files are headerless (I use 'official' every 4 words, as they do be provided by Binance on their official website, and as Binance is one of the exchange supported by nautilus trader, it seems logical Nautilus trader should be able to read them).
I added a test case to check the fix, please do not hesitate to share your feedback.
The test case can be used as example. It replicates what is done in other test cases. So finally, here is my parser.
import os, shutil
import pandas as pd
from nautilus_trader.persistence.catalog import DataCatalog
from nautilus_trader.persistence.external.core import process_files
from nautilus_trader.persistence.external.readers import CSVReader
from nautilus_trader.backtest.data.wranglers import BarDataWrangler
from nautilus_trader.model.enums import BarAggregation, PriceType
#from tests.test_kit.stubs import TestStubs
from nautilus_trader.backtest.data.providers import TestInstrumentProvider
from nautilus_trader.model.identifiers import Symbol, Venue
from nautilus_trader.model.identifiers import InstrumentId
from nautilus_trader.model.data.bar import BarSpecification, BarType
# https://docs.nautilustrader.io/user-guide/backtesting
DATA_DIR = "~/Documents/code/data/binance/data/spot/monthly/klines/ADABTC/1m"
CATALOG_PATH = os.path.expanduser("~/Documents/code/data/nautilus/catalog")
# Clear if it already exists, then create fresh
if os.path.exists(CATALOG_PATH):
shutil.rmtree(CATALOG_PATH)
os.makedirs(CATALOG_PATH)
catalog = DataCatalog(CATALOG_PATH)
adabtc_id = InstrumentId(Symbol("ADA/BTC"), Venue("BINANCE"))
bar_spec_1min_last = BarSpecification(1, BarAggregation.MINUTE, PriceType.LAST)
bar_type = BarType(adabtc_id, bar_spec_1min_last)
#bar_type = TestStubs.bartype_adabtc_binance_1min_last()
instrument = TestInstrumentProvider.adabtc_binance()
wrangler = BarDataWrangler(bar_type, instrument)
def binance_csvzip_parser2(data):
data['timestamp'] = data['timestamp'].astype('datetime64[ms]')
# data = data[['timestamp', 'open', 'high', 'low', 'close', 'volume']]
bars = wrangler.process(data.set_index("timestamp"))
return bars
binance_spot_header = ['timestamp','open','high','low','close', 'volume',
'ts_close', 'quote_volume', 'n_trades',
'taker_buy_base_volume', 'taker_buy_quote_volume',
'ignore']
binance_spot_csv_reader = CSVReader(block_parser=binance_csvzip_parser2,
header=binance_spot_header)
process_files(
glob_path=f"{DATA_DIR}/*.zip",
reader=binance_spot_csv_reader,
catalog=catalog,
)
But it will not run correctly without #501
Yep that makes sense to me - happy with #501. As you say, we have an "official" source of historic data from binance, so I think it makes sense to add a copy of your parser here into nautilus itself, inside the binance adapter. I'll discuss with @cjdsellers.
This seems like a good idea to me placing the parser into the Binance adapter.
However we could do even better and have it return the BinanceBar
data structure, rather than the standard Bar
(if there is sufficient data).
You find that type defined in adapters/binance/data_types.py
.
This seems like a good idea to me placing the parser into the Binance adapter.
However we could do even better and have it return the
BinanceBar
data structure, rather than the standardBar
(if there is sufficient data).You find that type defined in
adapters/binance/data_types.py
.
Hello @cjdsellers and @limx0, Nice to see you consider this source of data from Binance. Just some comments that you probably already have in mind, but I prefer to share in case you don't have them (may help doing right, right at the 1st time).
On your first few points: I think adding some backtest data download functionality into the Binance adapter would be valuable. I agree taking advantage of check sums is a great idea.
On your 3rd question: Yes, there are some differences between the spot and futures APIs, and at this stage I'm thinking of keeping those as separate classes in the adapter because of this. Work on this will begin soon.
Closing this issue as the PR added the bulk of the functionality, will add some TODO items to #429
Context
Hi, finally starting with nautilus trader, I am first trying to load data into a catalog. I could notice that backtest_config_example.ipynb is probably the best entry point. However, I am adapting to the data I would like to use, which is 1minute bar data from Binance. I take data from Binance official repo, which is made available in csv.zip format here. In below example, I am more specifically starting with spot data, ADABTC pair, with monthly data for October 2021 (file here).
Problem
Here is my current code. Basically, what should be the
block_parser
callable to be used inCSVReader
is not so clear to me. I am providing a pre-configuredpd.read_csv
method for this, I get aTypeError
exception.Steps to Reproduce the Problem
Code.
Error produced:
Specifications
nautilus_trader.__version__ Out[22]: 'latest'
?