wcwitt
commented
1 year ago I've been claiming this works for months, but it turns out I was mistaken, sorry.
I interpreted this, from the docs, to mean @showprogress uses length as a cost function:
The @showprogress macro wraps a for loop, comprehension, @distributed for loop, or map/pmap/reduce as long as the object being iterated over implements the length method
Which is why I set Base.length(d::DataPacket) = count_observations(d.data). But it doesn't work that way after all.
My latest experiments use your approach anyway - we can close this once they're merged.
cortner
Does the showprogress line
correctly represent progress? I.e. does it account for the fact that some structures are much bigger than others and more expensive to assemble?
Note in #54 I do this manually.
My evidence for this is that my distributed assembly starts with > 1.5h, then drops to ca 1h for a while and then completes in around 25 minutes total.