Leo4815162342
commented
1 year ago @awakanto
I've done some debugging (with -d or --debug flag - https://www.dukascopy-node.app/debugging).
And it seems that responses for same binary artifacts sometimes have data and other times are empty. This causes holes in the output files - hence the difference in the file size.
In this particular case on first try https://datafeed.dukascopy.com/datafeed/USDCAD/2020/00/10/BID_candles_min_1.bi5 had data, and on the second try was empty:
Example of an item with data (size 10.48 KB):
Examples of an item without data (size 0 Bytes):
Unfortunately we do not have control over the underlying datafeed.dukascopy.com servers.
However, what we can try to do in this case is to add a retry mechanism that would allows us to retry downloading of a file if the response is empty (0 Bytes).
awakanto
Downloading two consecutive files with the command line while using the same parameters results in files of two different sizes. Analyzing the files shows that some records simply failed to download - sometimes missing several days worth of data. No warning or alert is given to indicate that some data is missing.
Here is the command line that I am running. Run it, rename the first file, wait a few seconds, then run again with the exact same params. Compare the file size of the resultant downloads:
npx --max-old-space-size=16000 dukascopy-node -i usdcad -from 2020-01-01T00:00:00.000Z -to 2020-01-31T23:59:59.000Z -t m1 -f csv -v true -dir ./download/test