Closed dwinston closed 3 years ago
There isn't a way to do this, in that the server doesn't fetch from the database more messages than it needs to return your query (doing otherwise would make the query much slower).
One thing that you could do to hack this would be to fetch the very oldest message in the organization, and the very newest, and then do a progress bar based on what fraction of the ID space you've covered; it wouldn't be super accurate, but would result in a useful progress bar in most cases.
OOC what's the use case you had in mind where you want such a progress bar?
Thanks @timabbott -- that's a great idea with the ID space. My use case is expressed in this gist -- I wanted to get all public-stream messages so that I could use the data to have fun trying to apply PageRank to Zulip entities (e.g. users @-mentioning others).
My initial code was incorrect because I had misunderstood the meaning and usage of the "anchor" key in the API response; it turned out to be re-downloading messages again and again, which I misinterpreted as there just being a lot (millions!) of messages. I was running it for hours, and I just wanted to know when I might expect it to be done!
Once I figured out my bug, the whole thing took under 15 minutes for me for the Zulip server I was fetching from. I still don't have a progress bar with ETA in the gist I linked to, but your ID-space idea would do the trick.
Cool. We would like the Zulip python bindings to have a client.get_all_messages(...)
method that basically does that loop (taking a narrow as a parameter, which could be the streams:public
one that I think your example does) -- it feels like folks reimplement that a lot. If you want to polish your gist and submit it as a PR for that purpose, we can try to merge it as the official implementation.
Also worth noting that https://github.com/zulip/zulip-archive has a copy of that loop embedded in it.
Oh wow, I did not think to poke around for something like the zulip-archive
repo. I pretty much stuck with poring over https://zulip.com/api/get-messages. It does seem that I re-implemented a slice of populate.py! I just might attempt such a PR...thanks for the invitation!
I am aware of the
found_newest
value returned byGET /api/v1/messages
, which I use in paginating requests to decide whether there are more messages to fetch. However, there are a lot of messages on the server, and I would like to be able to show myself a progress bar so as to estimate the time to complete my fetch of messages. Is there a (perhaps undocumented?) was to get such a count for a narrow filter?