GL dropped last page from the headers: need to iterate over next

ssc-oscar / gather

For harvesting latest repos

10 stars 9 forks source link

GL dropped last page from the headers: need to iterate over next #4

Closed audrism closed 5 years ago

audrism commented 5 years ago

@zol0 should not be difficult to fix. The loop needs to be while. If killed, need to output the last page served.

zol0 commented 5 years ago

updated glRepos.py to iterate using 'rel=next' header from link. a failed page load does not exit the script immediately, script now only exits after 5 successive failed pages (likely indicating last page has been reached)

audrism commented 5 years ago

worked!