Almost overall breakage after Web.HotFix_20130821.1 release

[GoogleCodeExporter]

Obviously the last update broke GeoToad's parsing patterns.
While the Release Notes only mention a change of the cache_details page (which 
is now redirected), something else must have changed "under the hood").

What steps will reproduce the problem?
[Please include the relevant parts of your command line, if applicable.
Don't send your password though!]
1. Run a search with -T 4 (to limit terrain difficulty to "non-climbing")
2. Get 
`block in terrainMax': undefined method `>' for nil:NilClass (NoMethodError)
3. ...

What is the expected output? What do you see instead?
[If possible, include the last ~10--20 lines of verbose output.]
This used to work before...

What version of the product are you using? On what operating system?
[Did you check you're using the latest version?]
3.16.8 / 3.17.10 - there's nothing newer than on my disk ;)

Please provide any additional information below.
N/A

Original issue reported on code.google.com by Steve8x8 on 22 Aug 2013 at 7:38

[GoogleCodeExporter]

A second inspection of search results (nearest.aspx) reveals that there are no 
longer links to GUIDs in the list but URLs of the form 
/geocache/${wid}_${mangled_title}.
GeoToad traditionally used the GUID values to go directly to the "print with 
logs" pages (cdpf.aspx) - which are now out of reach.

On the other hand, WID and GUID searches still seem to work - with my somewhat 
patched-up local version at least (which still surprises me, so the "guts" of 
the new page don't have changed too much).

Until I find a way to expand the two-level search as implemented in Geotoad to 
a three-level one, all other searches won't work any longer.

Unfortunately, I'm in the final stages of an important project at the moment (a 
state that will last for a couple of weeks), therefore I cannot make any 
promises.

Sorry, folks :(

Original comment by Steve8x8 on 22 Aug 2013 at 9:25

• Changed title: Almost overall breakage after Web.HotFix_20130821.1 release

[GoogleCodeExporter]

Here's a first attempt at inserting a third stage (extract WIDs from search 
results, then run a multi-WID query).
Warning: very (!) quick and dirty. It's rather a workaround than a fix, but who 
knows what will be next from GC...

Original comment by Steve8x8 on 22 Aug 2013 at 12:45

Attachments:

• 277b-workaround.diff

[GoogleCodeExporter]

Although the workaround is functional (and therefore found its way into SVN), 
it in its current form produces a lot of GC fetches - in particular, *every* 
cache description page will be fetched, even if the cache will later be 
excluded.
A typical search in my homezone, usually with about 500 fetches, now results in 
more than twice that. 

Original comment by Steve8x8 on 23 Aug 2013 at 5:35

[GoogleCodeExporter]

If you don't want to harm your account, you may want to think twice before 
using the current version.
As I'm sure that the recent changes will be followed by other ones soon, I 
hesitate to announce the issue as "being worked on". Any input is still 
welcome. -S

Original comment by Steve8x8 on 23 Aug 2013 at 6:04

[GoogleCodeExporter]

Just a thought - could we instead get the GUID from 
http://jeep.geocaching.com/seek/nearest.aspx instead of the traditional URL? 
The "jeep" pages still use the GUID. Could then pull the cache details from the 
geocaching.com "print with logs" pages as usual. The results on the "jeep" page 
aren't coded the came as on geocaching.com, so the pattern matching would have 
to be re-done...but if this is viable it might be a better alternative to 
fetching every cache description page and then filtering out the unwanted ones. 
Caveat is that the coordinate search on the "jeep" page seems to be broken; 
don't know if that's temporary or not.

Original comment by theemail...@gmail.com on 25 Aug 2013 at 11:50

[GoogleCodeExporter]

Although that's an intriguing idea, I don't like it too much. (Thank you 
anyway!)

jeep.gc.com seems to be a dead end, a Lost Place of web design, and may vanish 
as soon as the next minion discovers that it still exists after 6 years.

If you feel challenged enough, not only by LPs "out there", you may try to:
- find an older version of GeoToad that matches the page design (from the 3.9.x 
era which still may be found on archive.org)
- add a wp2guid template to it
- modify the code so "nearest.aspx" queries go to jeep.gc.com
- run a search and create wp2guid (w2g) output
- grab the guids from there and inject them into a recent GeoToad
- be happy (?)

I'm afraid this current change to gc.com won't be the last one for now, and I'm 
rather hesitant to spend too much thought on short-lived "solutions" (which 
will cease to be useful within a few weeks, probably).
What's currently in SVN is a workaround, at the cost of a very large number of 
server accesses (a considerable risk to your account, I admit, which is 
definitely a backside), but at least it's not an attempt to completely rewrite 
the existing code (an advantage, at least at the moment).

The worst case I can imagine is a redesign of the "printable page" concept 
(which would go below /geocache/ or somewhere else but not /seek/ any longer) - 
it's been quite stable for a very long time. (Some GS minion will notice, and 
change it.)

On the other hand, that would trigger a complete overhaul of GeoToad's 
approach, which might bring some fresh ideas and possibilities.

I've been in this "business" for almost three years now, perhaps it's time 
again for a change? Too bad that helixblue isn't affiliated with GeoToad 
anymore - I'd appreciate a couple of suggestions.

Original comment by Steve8x8 on 27 Aug 2013 at 7:57

[GoogleCodeExporter]

Issue 278 has been merged into this issue.

Original comment by Steve8x8 on 27 Aug 2013 at 7:51

[GoogleCodeExporter]

There are moderately good news.
I have discovered a way to translate WIDs to GUIDs, _without_ loading (and 
parsing) the cache details page (and it seems to work for PMO caches as well...)
If I find a way to do this in a resource-friendly way (maybe a YAML file, as 
mappings don't change), and manage to test it thoroughly, 3.18.0 may be less 
far than I thought last week.
While hope is not lost yet, I have to ask for your patience.

- S

Original comment by Steve8x8 on 5 Sep 2013 at 7:38

[GoogleCodeExporter]

While working on the issue, things started to move into a different direction.
Since I - like everyone else - have accumulated a lot of "printable cache 
description" (cdpf) files, which have the GUID in their names, and contain the 
WID: why not extract the mapping from them, and build a "dictionary" that can 
be used to look up the mappings, without any need to ask the servers? (Of 
course, for new caches, this is still necessary, but much less often.)

While reverting the workarounds I had applied earlier, I found that the search 
result tables wouldn't be properly parsed anymore, but that was a minor fix 
that immediately went into both trunk and branch.

I'm maintaining the branch now to test the "dictionary" extension, and will 
release another (even if incomplete) devel version 3.17.11 based on that. I 
need feedback, and some more testing time, before I can go for 3.18.0. As 
always, help is appreciated.

Original comment by Steve8x8 on 11 Sep 2013 at 9:43

• Changed state: Started

[GoogleCodeExporter]

3.17.11, as a pre-release for 3.18.0, seems to have closed most of the holes 
which the famous HotFix ripped into GeoToad's logic and parsing.
If you don't like development releases, wait for 3.18.0 - but you've got 
nothing to lose since 3.16.9 and 3.17.10 became obsolete...

Please make sure you have read the "Dictionary" wiki article - a prepared 
dictionary (mapping) file will save a lot of extra requests to GS servers.

Comments, feedback, bug reports -> here (although it's a devel release), thanks

Original comment by Steve8x8 on 12 Sep 2013 at 11:38

• Changed state: FixAvail

[GoogleCodeExporter]

That release retrieves also new geocaches.
Many thanks!

Original comment by geocachi...@gmail.com on 14 Sep 2013 at 9:38

[GoogleCodeExporter]

Yes, it's supposed to fetch new mappings that aren't in the dictionary yet. (I 
might have come up with some instructions how to do this by hand - it's 
possible, and it's basically the "magic" behind one of the two methods 
implemented.)
In theory, it's also possible to get new (and old) member-only caches - the 
mapping mechanism doesn't make a difference.

Thanks for your feedback (after 145 downloads, I presume _some_ quirks would 
have already shown up... none did yet). I got another thumbs-up via Twitter...

Original comment by Steve8x8 on 14 Sep 2013 at 6:01

[GoogleCodeExporter]

Hello, the search of caches around a position working with 3.17.11 perfectly, 
(.. -x gpx -q coord "$position" -y 30km ..) so i'm the lucky to have an actual 
gpx file again. 
But if i run the version for my found caches
(-u myname -p mypw -o FOUND.gpx -x myfindgpx -q user "myname" -e "myname" -z)
i think only the caches are included that are loaded with the position version 
before.
Example:
I logged 3 caches out of my home area with a smartphone. This one are not 
included in the FOUND.gxp file. Maybe this is a little bug. But the normal 
search around a position working for my.   

Original comment by kipp.mic...@googlemail.com on 15 Sep 2013 at 7:18

[GoogleCodeExporter]

I added a comment in this Threat:
 Issue 198: geotoad fails to load Premium caches; maybe this is inline with the mentioned #13 comment

Original comment by HeinzBro...@gmail.com on 15 Sep 2013 at 2:56

[GoogleCodeExporter]

Kipp (#13): Can you re-run the myfindgpx query again, with "-L 1" to get only 
the 20 newest found caches, and perhaps "-v" to get debug output? I'm curious 
what happens to the WIDs that are missing from the output file - was there a 
report of "mapping", do the WIDs show up in the mapping.yaml file, any other 
options in effect? (you don't use the "-Y" option, I hope?)
Try to drop the "-e" option - it's a bit of belt and braces...

Heinz (#14): I've seen, and re-opened, issue 198... please follow up there

Original comment by Steve8x8 on 16 Sep 2013 at 7:22

• Changed state: InfoNeeded

[GoogleCodeExporter]

Kipp,
there's another very simple explanation for the behaviour you were seeing (and 
if it fits, the effect should have gone by now): GeoToad caches files, the 
cache descriptions for almost a week, and search results for a few hours at 
least.
You may have run a query for your user name shortly before, and when running 
the myfindgpx, the old cached nearest.aspx_* files hadn't expired yet... - can 
you confirm?

Original comment by Steve8x8 on 16 Sep 2013 at 11:47

[GoogleCodeExporter]

Hello Steve,

your simply explanation was the right!. With a new run also the caches are 
found that i've missed befor.

Original comment by kipp.mic...@googlemail.com on 17 Sep 2013 at 4:52

[GoogleCodeExporter]

3.18.0 is out, closing the majority of the "HotFix" related bugs that fall into 
this category.
Leaving the issue open for future discoveries.

Original comment by Steve8x8 on 24 Sep 2013 at 10:07

• Changed state: Started

[GoogleCodeExporter]

Closing this issue with the release of 3.18.1 - please open a new ticket if you 
find something that might be related to the August HotFix.

Original comment by Steve8x8 on 21 Oct 2013 at 10:54

• Changed state: Fixed