dzfranklin / plantopo

https://plantopo.com
GNU Affero General Public License v3.0
0 stars 0 forks source link

flickr indexer fails on spurious zero page #219

Open dzfranklin opened 2 months ago

dzfranklin commented 2 months ago

based on guesses I think the issue is with oregon and maybe washington

before:

 washington            |    228,186
 oregon                |         50
dzfranklin commented 2 months ago

the issue seems to be that when the search window is too big flickr can return no results.

dzfranklin commented 2 months ago

to make this easier to debug in the future and to enable better monitoring add uploaded_at field to geophotos

dzfranklin commented 2 months ago

these temporary hacks don't work. it's slow with many tiny pages and also sometimes it's too big

dzfranklin commented 2 months ago

what if we slide to slightly before the last seen so we know we'll see something. maybe only in special circumstances. then a zero is a too big

dzfranklin commented 2 months ago

shouldn't that already be the case? what if we just special case page=1 && len(page)=0 to reduce

dzfranklin commented 2 months ago

maybe guess zero page iff previous query returned more than 250 results and this one is zero? but this is awkward. sometimes zero pages are less than 2,000

or maybe we have to first descent and then ascend?

dzfranklin commented 2 months ago

if we start with a wide search window we know any zero is spurious...