search

Leo4815162342 / dukascopy-node

✨ Download historical price tick data for Crypto, Stocks, ETFs, CFDs, Forex via CLI and Node.js ✨
https://dukascopy-node.app
MIT License
350 stars 66 forks source link

Reformat the data #5

Closed lezwright closed 4 years ago

lezwright commented 4 years ago

Dear Leo,

Many thanks for such a wonderful package. Is it possible for you to provide a sample code that it provides the data in a different format? The current format that I receive is: [[1584489600000,1.05717,1.05726,1.05717,1.05721,63.54],[1584489660000,1.05718,1.05719,1.0571,1.05714,48.23],..,[1584490020000,1.05682,1.05693,1.0568,1.05692,50.37]]

I need to have at as:

{ s: "ok", t: [1584489600, 1584489660, 1584490020], c: [1.05721, 1.05714,1.05692], o: [1.05717, 1.05718, 1.05682], h: [1.05726,1.05719, 1.05693], l: [1.05717, 1.05693, 1.0568], v: [63.54, 48.23,50.37, 45000] }

Thank you very much.

Golnaz

Leo4815162342 commented 4 years ago

@lezwright hi, yes, this is a work in progress. I'm working on allowing custom output like json or csv, on top of that I'll be releasing a CLI version of this package soon. I'll keep you updated! ;)

lezwright commented 4 years ago

Thanks Leo, I got it sorted :-)

lezwright commented 4 years ago

Hi Leo, on hourly timeframe I sometimes get the following error message: image Could you please let me know why it happens?

lezwright commented 4 years ago

Also why do I get Error in data stream

?

Leo4815162342 commented 4 years ago

@lezwright can you share the exact search config with which you are getting this error?

lezwright commented 4 years ago

Hi Leo,

Here is the code I use:

var fromdate = '2010-01-01 00:00' let date_ob = new Date() let date = ("0" + date_ob.getDate()).slice(-2); let month = ("0" + (date_ob.getMonth() + 1)).slice(-2);
let year = date_ob.getFullYear(); let hours = ("0" +date_ob.getHours()).slice(-2);
let minutes = ("0" +date_ob.getMinutes()).slice(-2); var todate = year + "-" + month + "-" + date + " " + hours + ":" + minutes

var currency = 'AUDCAD';

getHistoricRates({ instrument: currency.toLowerCase(), dates: { from: fromdate, to: todate }, timeframe: 'h1' }).then(data => { console.log(Data ${fromdate} to ${todate} stored in ${currency.toLowerCase()}.json); }).catch(error => { console.log('error', error); });

Here is the error I get image

Leo4815162342 commented 4 years ago

@lezwright can you update the version of the dukakcopy-node to 1.1.4 - it should be fixed now. Plus, json is now available as the format output

lezwright commented 4 years ago

Thanks Leo!

lezwright commented 4 years ago

Hi Leo,

I am still getting the same error. Based on my observations this error happens around 8pm until midnight and over the weekends.

"error Error in data stream"

Best wishes, Golnaz

Leo4815162342 commented 4 years ago

@lezwright I think I know what can cause an issue here.

We get all the historical data from dukascopy. Their servers impose request-rate limitations. In cases when there are a lot of requests happening in a certain period of time, they might cut the connection - which means the requested data is not resolved in the required format, and we are not able to parse it, hence Error in data stream.

In order not to overwhelm dukascopy servers with a lot of requests, we have a very basic rule - we split all requests in batches and call them sequentially with a pause in between (see batchSize and pauseBetweenBatchesMs in https://github.com/Leo4815162342/dukascopy-tools/blob/master/packages/dukascopy-core/src/buffer-fetcher/index.ts#L12). Those values are hardcoded for now. One of the solutions is to find a more optimal value for those or to search for shorter window.

In case of your search config: AUDCAD h1 from 2010.01.01 till today, we need to fetch ~150 files and properly aggregate them. This will generate ~150 requests to the following URLs:

https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/03/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/04/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/05/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/06/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/09/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/07/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/08/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/10/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2010/11/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/03/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/07/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/06/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/05/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/04/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/08/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/04/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/09/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/11/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/05/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2011/10/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/03/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/06/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/07/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/08/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/09/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/11/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/03/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2012/10/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/05/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/04/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/06/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/10/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/07/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/08/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/11/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2013/09/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/03/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/04/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/09/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/07/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/05/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/06/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/11/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/10/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2014/08/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/03/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/04/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/05/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/06/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/08/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/09/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/07/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/11/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2015/10/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/05/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/06/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/07/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/03/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/04/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/08/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/09/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/11/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/05/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2016/10/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/04/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/03/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/06/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/09/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/07/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/08/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/10/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2017/11/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/03/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/04/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/06/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/08/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/07/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/05/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/09/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/11/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2018/10/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/03/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/05/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/06/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/04/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/08/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/09/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/07/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/10/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2019/11/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/00/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/01/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/02/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/04/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/05/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/01/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/02/BID_candles_hour_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/06/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/03/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/07/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/09/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/08/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/12/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/10/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/11/BID_candles_min_1.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/04h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/03h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/00h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/01h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/02h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/05h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/09h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/06h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/10h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/07h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/08h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/11h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/12h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/13h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/14h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/19h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/16h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/18h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/15h_ticks.bi5
https://datafeed.dukascopy.com/datafeed/AUDCAD/2020/03/13/17h_ticks.bi5
Leo4815162342 commented 4 years ago

Opening a separate issue for it: https://github.com/Leo4815162342/dukascopy-tools/issues/7