Retrying same page over and over in some cases

geekgonecrazy commented 4 years ago

I was trying this out on our github repo: https://github.com/rocketchat/rocket.chat and it got stuck on a single item in history. Added some verbose logs and it was getting stuck downloading the same page over and over again. I had to modify:

https://github.com/google/triage-party/blob/master/pkg/hubbub/timeline.go#L64-L66

to:

if resp.NextPage == 0 || sp.ListOptions.Page == resp.NextPage {
    break
}
sp.ListOptions.Page = resp.NextPage

The one it actually got stuck on: https://github.com/rocketchat/rocket.chat/issues/14278 not sure why 🤔

wreiske commented 4 years ago

👀 I like breaking things. :)

geekgonecrazy commented 4 years ago

Seems to also get stuck downloading the same list of PR's over and over again:

I0916 21:27:59.099728       1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:27:59.099814       1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:27:59.941729       1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:27:59.942050       1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:28:00.845414       1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:28:00.845505       1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:28:02.205749       1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:28:02.205990       1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:28:03.211754       1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:28:03.211843       1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:28:04.190577       1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:28:04.190703       1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)
I0916 21:28:05.277635       1 pull_requests.go:79] Downloading open pull requests for rocketchat/rocket.chat (page 2)...
I0916 21:28:05.277678       1 similar.go:81] Updating similarity table from PR cache "rocketchat-rocket.chat-open-prs" (30 items)

Adding the same sort of logic to the pull_requests.go seemed to do the trick also. I don't know what kind of info might be missed because of this. Just treating the symptom basically 🙈

mik-laj commented 3 years ago

I have a similar but I don't know if the same problem when trying to fetch data for apache/airflow repository.

I get the impression that the program goes into an infinite loop and can never get all the data.

geekgonecrazy commented 3 years ago

Yup seems to try getting the same page over and over blowing through the api limits repeatedly. It wasn’t until I added the check both places that it started working.

I’ll open a PR with the changes I made

google / triage-party

Retrying same page over and over in some cases #213