Missing Caches on user query

[PascalTurbo]

Hi There,

I'm trying to get all my caches and I'm using this query:

./geotoad.rb --delimiter='|' --format=yourfindgpx --includeArchived --includeDisabled --output='/Users/christianbecker/Desktop/' --queryType=user PascalTurbo

This results in a list of 507 caches but I would expect 547 caches.

When running in debug mode, it shows me the following output for filtering. For me it looks like geotoad did not extract all caches from the result lists.

[###] (28/28) Search results: page 28 (r)
D: waypoints extracted: 507, combined: 507
( o ) Your "user" query "PascalTurbo" returned  507 caches.
D: Filter running cycle 1,  507 caches left.
D: Filter running cycle 2,  507 caches left.
D: Filter running cycle 3,  507 caches left.
( o ) Pre-fetch  filter  complete,  507 caches left.

[PascalTurbo]

I'm analyzing the problem and will share my findings with you.

Geotoad downloads 28 Pages. But only 26 of the pages seems to contain Geocaches. There are two pages with no geocaches listet and that are exactly the 40 caches missing.

[steve8x8]

Thanks for opening this issue - I've been seeing this for several weeks now but couldn't get a grip on the cause.

The two pages without geocaches listed - what's special about them? Apparently they provide a link to the next page, so they at least conform to the structure one would expect?

I hope it would be rather easy to check for such an incomplete page - are there any special characteristics one might check (except the number of cache entries which should always be 20 except for the last page (AFAIR we know the number of pages but not the total number of caches)?

(There are occasions when a query for an individual geocache doesn't return the description page but some weird statistics page instead - checking that for a GC code would easily show that there was a problem. Perhaps something very similar can be done with "nearest" queries too?)

[steve8x8]

Um, btw, this also happens for location queries.

[PascalTurbo]

Great, thank you for the information. I'll investigate deeper in it and when I'm sure, that there is no private data in it, I'll also share the "defect" pages.

[steve8x8]

No need, I was able to reproduce the problem "at first shot", and indeed got one page about 42k in size (normal ones are about 94k, the last one of my query comes in at 64k). Now that I know where to look I can find out what to look for - thanks for getting me on the right track!

And indeed: <table class="SearchResultsTable Table"> is missing from the page in question. It instead looks like a pristine search page...

Hint: Using the -Y option makes searching faster. The number of geocaches returned before filtering must be >= 20* (number of pages reported). The rest of the code seems to be safe.

Update: Running the query (rural area, 10km, resulting in 10 pages of results) multiple times, with different settings of GEO_DIR (so the cache data would not be reused the next time) always returns the same number of 168 caches. So the problem persists across different internal hashes, and repeating a failed search would not help :'( For a while, I had suspected some updates going on on their server (e.g. newly published caches in that area) but that wouldn't happen to an owner looking for his/her own caches without him/her knowing, right?

Now that I know what to look for, I could even use the "nearest.aspx?lat=...&lng=...&dist=6.2137" URL to flip through the pages in the browser, same behaviour at page 7. Trying to go back to former pages also breaks, and I cannot go to arbitrary pages.

In short: there is a problem, it's probably on GC's side, but they probably won't care :/ - unless they actually care and have limited their bandwidth?!? (their refusal to serve my browser might indicate an intention)

Another update: waiting about 15 minutes and retrying fixed it for now, but I think waiting (any amount of time larger than a few seconds) isn't an option for GeoToad, in particular since the issue would show up with the next run (as I have found now, again page 7, this is particularly strange)?

[steve8x8]

$ git diff
diff --git a/lib/search.rb b/lib/search.rb
index ce365fb..3ffb3a6 100644
--- a/lib/search.rb
+++ b/lib/search.rb
@@ -629,6 +629,10 @@ class SearchCache
     end

     while (page_number < pages_total)
+
+      # forced sleep to possibly fix #368
+      sleep(10)
+      
       # get subsequent search pages
       debug "*** On page #{page_number} of #{pages_total}"
       last_page_number = page_number

seems to get rid of the issue, but 10 seconds are a rather long time. A check for <table class="SearchResultsTable Table"> should be added, although it's not completely clear to me what to do if it triggers (in any case, the check must be done in shadowget.rb to invalidate the bad cached file)

[steve8x8]

BTW I was wondering whether it's the "old style query" that causes the problem, so I went for one of your own caches and looked for caches near -> https://www.geocaching.com/play/results/?st=49.576233%2C7.230733&ot=coords&asc=true&sort=distance - scrolling forward quickly will work until page 9 or 10, then produce the same result over and over (with the page number being incremented). So instead of telling me "don't rush" they show me wrong content...

[PascalTurbo]

Great investigation. Is there anything I can support you with?

I've added your sleep-fix and now it's fetching 520 caches. 13 more, 27 Missing 🥇

[steve8x8]

So this time, the last chunk and another one went south... Indeed, when running the same query (my test mentioned above) five times, I'm seeing 149, 169, 189 caches (a new one must have been published ;)).

This means that the delay doesn't fix it, or just postpones the problem.

In some forum article I found that the same issue shows up when updating the map (directly on gc.com) too often and too quickly, and someone mentioned that there's a "per-minute" limit for queries (and that the individual pages of a query seem to be seen as separate queries?). Which is what we're getting here, and adding a delay when it hits may be the only fix.

As I said above, this will have to go into the shadowget.rb routines (which are under heavy reconstruction atm anyway, and them not being finished is the main blocker for a new release I've been preparing for a while).

So what's next, until I can get the code ready for retries? As a workaround you may keep the cache lists from the past queries and merge them... E.g., if you're keeping them in another database (GSAK, c:geo, ...) just add to, not replace the whole set. (The "info" entry will tell you if a particular entry is outdated.) It's a horrible kludge and a mess when it comes to handling found caches (unless you keep a record of those in the same db), but for 95% of all caches it may do the trick - until there's an overall solution.

(I've been checking c:geo whether they have seen something similar but couldn't find a matching issue. Will read their git log when I've got a bit more time on my hands.)

Off for the woods now - with a possibly incomplete list of caches...

[steve8x8]

Good news this time.

After doing some experiments with error codes and wrappers I decided to try another approach.

Since - even with some of the pages failing - we have a clear indication of caches not being reported (if their count is smaller than the number of total pages would suggest), and we have control of what to keep in file cache and what to invalidate (or not store at all): repeating an incomplete search inside the program logic seems to be the best solution.

This will re-use previously received data (from local cache) and only add missing pages, in a loop until everything is there - with some progressive sleep in between.

After some more review on my laptop, I'll merge the changes with my "official" setup that does the nightly runs, and give the code some real-life testing before rolling the release - but I think we're getting closer. Watch the commits ;)

[steve8x8]

...
( o ) "user" search for "PascalTurbo"
[   ] ( 1/28) Search results: page 1 (r)
[.  ] ( 2/28) Search results: page 2 (r)
 ***  File pattern SearchResultsTable not found in data (41584b)
[-  ] ( 3/28) Search results: page 3 (?)
 ***  Interrupting incomplete query
 ***  Search results incomplete: 40 caches from 28 pages
( - ) Restarting query after 5 seconds ...
[   ] ( 1/28) Search results: page 1 (l)
[.  ] ( 2/28) Search results: page 2 (l)
[-  ] ( 3/28) Search results: page 3 (r)
[+  ] ( 4/28) Search results: page 4 (r)
[x  ] ( 5/28) Search results: page 5 (r)
[*  ] ( 6/28) Search results: page 6 (r)
[=  ] ( 7/28) Search results: page 7 (r)
[=  ] ( 8/28) Search results: page 8 (r)
[%  ] ( 9/28) Search results: page 9 (r)
[#  ] (10/28) Search results: page 10 (r)
[#. ] (11/28) Search results: page 11 (r)
[#- ] (12/28) Search results: page 12 (r)
[#+ ] (13/28) Search results: page 13 (r)
[#x ] (14/28) Search results: page 14 (r)
[#x ] (15/28) Search results: page 15 (r)
[#* ] (16/28) Search results: page 16 (r)
[#= ] (17/28) Search results: page 17 (r)
[#% ] (18/28) Search results: page 18 (r)
 ***  File pattern SearchResultsTable not found in data (41745b)
[## ] (19/28) Search results: page 19 (?)
 ***  Interrupting incomplete query
 ***  Search results incomplete: 360 caches from 28 pages
( - ) Restarting query after 5 seconds ...
[   ] ( 1/28) Search results: page 1 (l)
[.  ] ( 2/28) Search results: page 2 (l)
[-  ] ( 3/28) Search results: page 3 (l)
[+  ] ( 4/28) Search results: page 4 (l)
[x  ] ( 5/28) Search results: page 5 (l)
[*  ] ( 6/28) Search results: page 6 (l)
[=  ] ( 7/28) Search results: page 7 (l)
[=  ] ( 8/28) Search results: page 8 (l)
[%  ] ( 9/28) Search results: page 9 (l)
[#  ] (10/28) Search results: page 10 (l)
[#. ] (11/28) Search results: page 11 (l)
[#- ] (12/28) Search results: page 12 (l)
[#+ ] (13/28) Search results: page 13 (l)
[#x ] (14/28) Search results: page 14 (l)
[#x ] (15/28) Search results: page 15 (l)
[#* ] (16/28) Search results: page 16 (l)
[#= ] (17/28) Search results: page 17 (l)
[#% ] (18/28) Search results: page 18 (l)
[## ] (19/28) Search results: page 19 (r)
[##.] (20/28) Search results: page 20 (r)
[##-] (21/28) Search results: page 21 (r)
[##-] (22/28) Search results: page 22 (r)
 ***  File pattern SearchResultsTable not found in data (41020b)
[##+] (23/28) Search results: page 23 (?)
 ***  Interrupting incomplete query
 ***  Search results incomplete: 440 caches from 28 pages
( - ) Restarting query after 5 seconds ...
[   ] ( 1/28) Search results: page 1 (l)
[.  ] ( 2/28) Search results: page 2 (l)
[-  ] ( 3/28) Search results: page 3 (l)
[+  ] ( 4/28) Search results: page 4 (l)
[x  ] ( 5/28) Search results: page 5 (l)
[*  ] ( 6/28) Search results: page 6 (l)
[=  ] ( 7/28) Search results: page 7 (l)
[=  ] ( 8/28) Search results: page 8 (l)
[%  ] ( 9/28) Search results: page 9 (l)
[#  ] (10/28) Search results: page 10 (l)
[#. ] (11/28) Search results: page 11 (l)
[#- ] (12/28) Search results: page 12 (l)
[#+ ] (13/28) Search results: page 13 (l)
[#x ] (14/28) Search results: page 14 (l)
[#x ] (15/28) Search results: page 15 (l)
[#* ] (16/28) Search results: page 16 (l)
[#= ] (17/28) Search results: page 17 (l)
[#% ] (18/28) Search results: page 18 (l)
[## ] (19/28) Search results: page 19 (l)
[##.] (20/28) Search results: page 20 (l)
[##-] (21/28) Search results: page 21 (l)
[##-] (22/28) Search results: page 22 (l)
[##+] (23/28) Search results: page 23 (r)
[##x] (24/28) Search results: page 24 (r)
[##*] (25/28) Search results: page 25 (r)
[##=] (26/28) Search results: page 26 (r)
[##%] (27/28) Search results: page 27 (r)
[###] (28/28) Search results: page 28 (r)
( o ) Your "user" query "PascalTurbo" returned  547 caches.
( o ) Archived   filtering removed  298 caches.
...

;-) Now I hope that there are no regressions after commit https://github.com/steve8x8/geotoad/commit/054d83f92ebc1914de994d54558abf89ea59410c - will test next week

[PascalTurbo]

Wow, great work. I've tested your fix and it seems to work on the first try. Any chance that I can buy you a coffee?

[steve8x8]

Perhaps when I'm done really fixing this (i.e., not requiring a restart of the whole query over and over until it's complete). Let me get the new version out first (plans were to do this over the past weekend but you're never planning alone...) and find the last hidden bug (you may have noticed that the fix for this issue broke extended logbooks > 10 entries...)

[steve8x8]

Just another finding that to me proves something must be wrong at the server side:

Go to https://www.geocaching.com/seek/nearest.aspx and choose "All Event Cache Types" and select your favourite area (for me, country and state will do). Press "Search for Geocaches".

The first page opens, for a URL with the pattern https://www.geocaching.com/seek/nearest.aspx?state_id=xxx&ex=0&cFilter=69eb8534-b718-4b35-ae3c-a856a55b0874-parent&children=y .

Note how many pages there'll be (this of course only works if there are more than 20 events).

Now click "Next" and compare the number of pages - if you're "lucky" the server has switched to the full list of all caches for the selected region, it still may show some events because caches are sorted by date in reverse order - if you're unlucky you'll get an empty search page with "sort By Postal Code" selected.

Replacing cFilter with tx and dropping the -parent suffix seems to fix the issue but their code uses cFilter...

[steve8x8]

I've missed two weekends due to other commitments, will try to get the release out on Friday or Saturday UTC - unless there's a last-minute bug serious enough (sometimes I hate to be my own test user: I cannot cover all the possible test cases without risking my account).

There are some changes to the TUI in the pipeline but those will have to wait for the next minor update.

[steve8x8]

Finally I found the time to roll a new release - this took me long enough to catch a long-time inconsistency which had been tolerated for what feels like ten years ( #369 ).

@PascalTurbo since I didn't hear back from you, the problem seems to be addressed? (It's not fixed since the problem is generated on the server side, as I documented above, but the current workaround appears to be stable and fast enough to cope with the issue.)

Release 3.33.3 has been published, incorporating the changes to handle broken search pages - closing.

[steve8x8]

@PascalTurbo with 3.34.0 to be released soon: any new findings (except that the underlying issue still persists)? Could you help with #370?