bitfinex error: getTrades() returned duplicate results

getfreeze commented 5 years ago

System information

Have I written custom code (as opposed to using zenbot vanilla):NO
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):MAC OS 10.13.6
Zenbot version (commit ref, or version):4.1.0
Zenbot branch:none
NodeJS version:v8.11.3
Python version (when using a python script):none
Exact command to reproduce (include everything):zenbot Stev$ node zenbot backfill bitfinex.BTC-USD --days 5
Did I make any changes to conf-sample.js?:yes, but nothing impactable.

Describe the problem

A bug occur lately when i trying to back fill bitfinex trade data, i keep fail while trying to backfill bitfinex data, at first I thought it was due to bitfinex maintainence and cause the date to be wrong, but then i waited a couple day and try, same error occur again. (binance backfill works well) I have tried to fix and debug this for more than 1 week but no luck. Also tried to drop trades table multiple time , no luck still.

I have reproduce this problem with a fresh version of zenbot.

Source code / Error logs

Tans-MacBook-Pro:zenbot Stev$ node zenbot backfill bitfinex.BTC-USD --days 5 (node:6654) DeprecationWarning: collection.save is deprecated. Use insertOne, insertMany, updateOne, or updateMany instead.

bitfinex.BTC-USD saved 1000 trades 5 days left .. error: getTrades() returned duplicate results { product_id: 'BTC-USD', to: 1546061997511 } { product_id: 'BTC-USD', to: 1546061997511 } Tans-MacBook-Pro:zenbot Stev$ node zenbot backfill bitfinex.BTC-USD --days 5 (node:9419) DeprecationWarning: collection.save is deprecated. Use insertOne, insertMany, updateOne, or updateMany instead.

bitfinex.BTC-USD saved 1000 trades 5 days left .................. error: getTrades() returned duplicate results { product_id: 'BTC-USD', to: 1546212780299 } { product_id: 'BTC-USD', to: 1546212780299 }

getfreeze commented 5 years ago

any 1 can point me in a right direction to fix this? should i work on backfill.js or bitfinex's exchange.js? Thankyou in advance!

mrseeker commented 5 years ago

Most likely bitfinex's exchange.js, since the backfill will attempt to get data from there.

msaraya commented 5 years ago

Same issue for CEXIO ~/zenbot# zenbot backfill cexio.BTC-USD --days 90 (node:11023) DeprecationWarning: collection.save is deprecated. Use insertOne, insertMany, updateOne, or updateMany instead.

skipping 10457 hrs of previously collected data

cexio.BTC-USD saved 1000 trades 1343 days left . error: getTrades() returned duplicate results { product_id: 'BTC-USD', from: 376024 } { product_id: 'BTC-USD', from: 376024 }

DeviaVir / zenbot