Provide progress bar when resolving transactions

[qwhelan]

Use progressbar2 to show dali's progress during transaction resolution, which can be particularly slow when fetching prices from the web.

[qwhelan]

@eprbell A few thousand shouldn't be instantaneous unless you've got all prices in cache, network queries disabled, or all prices populated in your inputs. For a rate limited pricing API, that should be at most a few hundred seconds of run time, so I'm guessing one of the above applies. For myself, it's definitely hours.

[macanudo527]

@qwhelan this is huge. This was on my "I'll get to it someday" list, so it is good to see somebody get in here and get it done. I can say from experience that with the Kraken API it takes a good couple of hours and this will really help.

Can plugins still output debugger info messages? I know the CSV downloader for Kraken will output a message when it downloads and parses a CSV file. Does that interrupt the progress bar?

[qwhelan]

@macanudo527 Yeah, they can but that effectively triggers a snapshot of the progress bar to be output, the debug message to be logged, and then the progress bar resumes. So not the prettiest output, but there's settings in progressbar2 that might let that behavior be refined a bit. The ETA also swings around quite a bit as it uses a rolling 10 event window to forecast that I'd probably want to expand to get more stable ETAs.

[eprbell]

@eprbell A few thousand shouldn't be instantaneous unless you've got all prices in cache, network queries disabled, or all prices populated in your inputs. For a rate limited pricing API, that should be at most a few hundred seconds of run time, so I'm guessing one of the above applies. For myself, it's definitely hours.

Duh, it was the cache :-)

[eprbell]

@macanudo527, can you try it out with your setup and see how the output looks? In my case it's fine, but I just use the historic crypto plugin which doesn't print anything during transaction resolution.

[qwhelan]

@eprbell Just fixed the typo you flagged and added a second progress bar to the first for loop that you suggested.

[macanudo527]

This works well enough with my setup. My only issue is that the Info messages get tacked on to the end of the progress bar. They don't get displayed on their own line. I could add a new line to every info message, but that isn't ideal of course.

Or, it seems like progressbar2 has a better solution for this since it wraps the stderr.

It would be more useful if the adaptiveETA sampled the last 100 transactions since the ETA jumps around a lot. The regular ETA may actually work better for us.

I'm not too concerned about having a very pretty output, though.

[qwhelan]

@macanudo527 I can't get any messages to show up with my setup after adding the wrap_stderr() logic, could you check that you're seeing the expected behavior now?

I've updated the widgets to remove the raw counts and switch the AdaptiveETA to use 100 data points. Using the raw ETA caused some issues if the first for loop is very fast relative to the second (which is true for my data) as that causes the ETA to be massively biased low.

[macanudo527]

Sorry for the late reply. This looks great to me!