Implement progress bar for Pulling Data via REST API for CCXT plugins

[rapus95]

I'm stuck in

D:\Tax\Crypto\dali>dali_us -s -o output -p test_ daliconfig.ini
INFO: Country: us
INFO: Initialized input plugin 'dali.plugin.input.rest.binance_com'
WARNING: No Fiat Access Key. Fiat pricing will NOT be available. To enable fiat pricing, an access key from exchangerate.host is required.
WARNING: No Fiat Access Key. Fiat pricing will NOT be available. To enable fiat pricing, an access key from exchangerate.host is required.
INFO: No pair converter plugins found in configuration file: using default pair converters.
INFO: Reading crypto data using plugin 'dali.plugin.input.rest.binance_com'

for 30 mins already. And there are no signs on whether it works/is doing something or not.

Looked for some DEBUG output. Managed to enable it. But now I get spammed with lots of megabytes in a single file. At least I know that it's processing something. Still, a progress meter would be nice!

[macanudo527]

It would be a little tricky, but you could probably add progressbars to the ThreadPools like the one below: https://github.com/eprbell/dali-rp2/blob/a45d44fe9d47f1996a5fbcc439ce05ba093bb880/src/dali/abstract_ccxt_input_plugin.py#L431

That would give you a series of progress bars that could help you understand how far along the REST process is going.

[rapus95]

Hmm, if that's tricky, how about adding a "first time stamp read/most recent timestamp read" pair for each thread? (assuming that entries are imported from oldest to newest)

[macanudo527]

That would create too much terminal spam. If you like you can open another window and tail the log file:

tail -f <timestamp>.log

The -f flag means it will follow the file and keep giving you updates.

I can implement a progressbar easily enough, just don't have the time at the moment. I think it would tricky for someone new to Python, but totally doable if you want to tackle it. We can help you with a code review.

[rapus95]

Regarding terminal spam, I've seen many tools that can update their terminal output instead of appending new lines (also seen for progress bars). I have now clue how that works, but that seems like an option that totally wouldn't spam.

[eprbell]

I don't think spam is an accurate word in this context: when run with LOG_LEVEL=DEBUG, RP2 and DaLI can generate large logs, which is expected and useful in debug scenarios. The tail -f suggestion is good and you can remove the log directory after running, if you want to recover disk space.

[rapus95]

you're totally right, debug output is the totally wrong approach to see IF it's doing something. It just happened to be the only way. That's why I asked for adding some other signs of progress. Be it a progress bar (which then needs to show some update on each few entries) or an updating text that shows info about the most recently imported transaction. So there's visible change as long as the program does something.

[macanudo527]

Regarding terminal spam, I've seen many tools that can update their terminal output instead of appending new lines (also seen for progress bars). I have now clue how that works, but that seems like an option that totally wouldn't spam.

This is of course possible, but I think implementing a progress bar would take about the same amount of effort and would be a more favorable option in my opinion. The transactions would most likely fly by too fast to be of any use.

To be clear, I'm totally fine with putting a progress bar in during the transaction processing part, implementing it in abstract_ccxt_input_plugin.py so that it is available to all the CCXT input plugins. If you'd like to implement it, we can look at a PR.