search

sadighian / crypto-rl

Deep Reinforcement Learning toolkit: record and replay cryptocurrency limit order book data & train a DDQN agent
860 stars 236 forks source link

Error while exporting to csv #16

Closed epyx25 closed 4 years ago

epyx25 commented 4 years ago

I tried to export some data from the db to csv.

Maybe related https://github.com/man-group/arctic/issues/615

I do not think that it should be a problem, but there was a gap while I recorded that day. I had to stop it for a couple of hours because I was travelling. .

python data_recorder/tests/test_simulator.py [2020-05-04 16:37:49,281] init_db_connection for None... [2020-05-04 16:37:49,281] Connecting to mongo: localhost (localhost) [2020-05-04 16:37:49,290] Getting ['BTC-USD'] data from Arctic Tick Store... ///.pyenv/versions/crypto-rl/lib/python3.7/site-packages/arctic/tickstore/tickstore.py:418: FutureWarning: Conversion of the second argument of issubdtype from int to np.signedinteger is deprecated. In future, it will be treated as np.int64 == np.dtype(int).type. if np.issubdtype(dtype, int): [2020-05-04 16:47:11,561] Got data in 562.269496 secs, creating DataFrame... [2020-05-04 16:47:19,641] 2600000 rows in 570.34618 secs: 4558 ticks/sec [2020-05-04 16:47:22,474] Completed querying 2590711 ['BTC-USD'] records in 573 seconds [2020-05-04 16:47:22,498] Completed get_tick_history() in 573 seconds [2020-05-04 16:47:22,498] querying BTC-USD [2020-05-04 16:47:22,523] Database: [coinbase is recording BTC-USD]

[2020-05-04 16:47:22,523] init_db_connection for BTC-USD... [2020-05-04 16:47:22,575] Connecting to mongo: localhost (localhost) [2020-05-04 16:47:23,048] Starting get_orderbook_snapshot_history() loop with 2590711 ticks for ['BTC-USD'] [2020-05-04 16:47:23,063] ...completed 0 loops in 0 seconds [2020-05-04 16:47:23,063] BTC-USD's order book cleared. [2020-05-04 16:47:26,206] Book finished loading at None [2020-05-04 16:47:26,208] BTC-USD first tick: 2020-04-25 16:11:00.023615+00:00 ///.pyenv/versions/crypto-rl/lib/python3.7/site-packages/arctic/tickstore/tickstore.py:757: FutureWarning: A future version of pandas will default to skipna=True. To silence this warning, pass skipna=True|False explicitly. v = TickStore._ensure_supported_dtypes(v) Traceback (most recent call last): File "data_recorder/tests/test_simulator.py", line 79, in test_get_orderbook_snapshot_history() File "data_recorder/tests/test_simulator.py", line 38, in test_get_orderbook_snapshot_history orderbook_snapshot_history = sim.get_orderbook_snapshot_history(query=query) File "///crypto-rl/data_recorder/database/simulator.py", line 253, in get_orderbook_snapshot_history order_book.new_tick(msg=tick) File "///crypto-rl/data_recorder/coinbase_connector/coinbase_orderbook.py", line 141, in new_tick self.db.new_tick(msg) File "///crypto-rl/data_recorder/database/database.py", line 62, in new_tick self.collection.write(self.sym, self.data) File "///.pyenv/versions/crypto-rl/lib/python3.7/site-packages/arctic/tickstore/tickstore.py", line 597, in write buckets = self._to_buckets(data, symbol, initial_image) File "///.pyenv/versions/crypto-rl/lib/python3.7/site-packages/arctic/tickstore/tickstore.py", line 623, in _to_buckets bucket, initial_image = TickStore._to_bucket(x[i:i + self._chunk_size], symbol, initial_image) File "///.pyenv/versions/crypto-rl/lib/python3.7/site-packages/arctic/tickstore/tickstore.py", line 757, in _to_bucket v = TickStore._ensure_supported_dtypes(v) File "///.pyenv/versions/crypto-rl/lib/python3.7/site-packages/arctic/tickstore/tickstore.py", line 662, in _ensure_supported_dtypes raise UnhandledDtypeException("Casting object column to string failed") arctic.exceptions.UnhandledDtypeException: Casting object column to string failed

epyx25 commented 4 years ago

configuration RECORD_DATA was set to True