Wishlist: read found caches from file, don't query gc

[GoogleCodeExporter]

As already suggested in issue 235, it might save a lot of gc page requests if a 
user's found caches could be read from a file.
This would further reduce the load imposed on gc's servers.
For example, if I keep a list of user A's found caches on my local disk, 
consisting of n-hundred entries, this would save 5*n queries to gc.
Of course, file caching would take care of some of these requests, but still it 
might be nice to have it, in addition to the "f=1" shortcut to exclude "own 
finds" from a query.

What would be a reasonable way to tell GeoToad that the --userExclude option 
doesn't contain a user _name_ but a wid list? (Would a single-character prefix 
be sufficient- what about @?)
And is the gain worth the hassle?

Original issue reported on code.google.com by Steve8x8 on 25 Mar 2012 at 7:11

[GoogleCodeExporter]

Using the devel version (3.17, in branches/steve8x8), you may specify "-E 
user=/path/to/file" instead of "-E user" to exclude WIDs listed in that file 
(first column).
This will avoid additional queries to gc.
Error handling (i.e. fallback to old behaviour if file unreadable etc.) to be 
added.

Original comment by Steve8x8 on 25 May 2012 at 12:36

• Changed state: Started

[GoogleCodeExporter]

Nice feature, really speeds up my cache-update-script! Hope to see it in the 
stable version soon.

Original comment by jan.reuc...@gmail.com on 23 Sep 2012 at 7:59

[GoogleCodeExporter]

I'd happily do this - but there's one obstacle still unresolved:
Since ":" is being used as a separation character on the command line, it's 
currently impossible to use files which are located on another Windows drive.
(Imagine "-E user1=d:\Data\user1.list:user2=d:\Data\user2.list", and how it 
would be parsed.)
I'm trying to investigate whether a link would do the trick, but links in 
Windows are different from links in Unixoid OSes.
Until this has been investigated, there will be no hint in the manual page, and 
therefore no inclusion as a stable feature :(
If you have a suggestion: don't hesitate to step forward!

BTW, thanks for using the development version... ;)

Original comment by Steve8x8 on 24 Sep 2012 at 10:43

[GoogleCodeExporter]

IMHO the easiest solution would be to allow specifying the-E option more than 
once, e.g. -E user=file1 - E user=file2. 

Original comment by jan.reuc...@gmail.com on 24 Sep 2012 at 12:31

[GoogleCodeExporter]

If there wasn't a lot of legacy in the code (and, I'm absolutely positive, a 
lot of rationales to do things the way they are done), that would indeed be 
feasible.
But it's a bit more complicated.

Let me explain.

If you check interface/input.rb, just below the large GetoptLong.new() there's 
currently an assignment of the input options to components of a hash. 
While it would be easy to replace the "set" with an "append" operation, that 
would break a lot of other semantics: for example, "-T 4 ... -T 2" would result 
in  a hash entry corresponding to "-T 4|2", instead of just overwriting the 
previous value with the newer one (which actually is something I'm actively 
exploiting, having a default set of query options and an additional environment 
variable which may be empty or contain additional limitations).
We'd have to distinguish between the "simple" options and the ones tagged as 
"(MULTI)" in the man page. Currently there's no code doing that.
Given that throughout usage of "|" instead of ":" as a separator would fix this 
particular issue (but hey, what if someone wants to use a pipe instead of a 
file?), one might wish to give up the ":" separator completely, BUT: ":" causes 
no problems on Unix systems while "|" would require escaping on both platforms.

In short: I'd like to keep both ":" and "|" as separators, in all places, just 
for consistency.

Since files can be combined before running GeoToad and duplicate WIDs are 
tolerated, it makes little sense to have multiple "-E u=f" options - this makes 
my argument in #3 obsolete of course.
If we find a way to overcome the "drive letter" (worst case scenario: let user 
copy file to current drive) we're fine.
A hint not to use drive letters and ":" in the file path would be a good idea, 
wouldn't it?

Original comment by Steve8x8 on 25 Sep 2012 at 7:45

[GoogleCodeExporter]

Thinking even longer about the issue, I found another (supposedly safe) way 
around the problem.

By redefining the delimiter (set) in a global variable, e.g. from ":|" to "+^", 
I can kill several birds with one code-stone:
(a) get the dozen "split(/[:\|]/)" operations use the same delimiters all along
(b) let the TUI (internally) use "|" only
(c) open the way for ":" in file names
Another option letter, "-m" (a.k.a "--delimiter") will be used for that.

Upload to branch to follow later when I got things sorted into separate patch 
sets (within the next 24 hours I hope).

Original comment by Steve8x8 on 25 Sep 2012 at 9:22

[GoogleCodeExporter]

In this case, the user must take care of choosing the right delimiter? Hmm, 
makes things more complex. Does it work from the built-in GUI as well?

IMHO allowing only a single file for -E and therefore not splitting at colons 
would be the best option. As you said, concatenating several files before 
feeding them into GeoToad is easy.

Original comment by jan.reuc...@gmail.com on 30 Sep 2012 at 8:58

[GoogleCodeExporter]

My plan is to use "|" as the only delimiter in the TUI (it's hidden from the 
user anyway, so I could even use "^"). There are patches already to accomplish 
this, but not everything has been committed to my branch yet (and I'm in doubt 
whether I'll make it before going on vacation).

-E is a MULTI option, and will continue to be.

If (by using the default set of delimiters) GeoToad splits a filename at the 
wrong place (as in -E user=c:\path\file, resulting in two selections "user=c" 
and "\path\file") there will be a warning because of a non-existent file c, and 
(yet untested!) another empty list for a non-existent user "\path\file". (Can 
you confirm this behaviour with the current branch version?)

Original comment by Steve8x8 on 1 Oct 2012 at 8:38

[GoogleCodeExporter]

jan,

have you by chance given the devel branch a try? I'm a bit undecided whether to 
merge this into 3.16.6 "stable", and I'm absolutely no TUI user...
The feature has been in since 3.17.0, and if you're not using Windows, that 
would do. 3.17.3 included the "-m" option, to be able to use drive letters (and 
colons) in the file spec. There's no way to tell a drive letter from an 
argument separator...

Original comment by Steve8x8 on 18 Apr 2013 at 1:31

[GoogleCodeExporter]

Steve,

I'm already on the devel branch for several months now, and it works flawlessly 
for me (Linux, only using a single file for -E). I'm also not using the TUI.

Original comment by jan.reuc...@gmail.com on 18 Apr 2013 at 6:59

[GoogleCodeExporter]

I'm planning to include this feature into the upcoming (in a few weeks) last 
stable 3.16 release - is there anyone who can confirm that Windows doesn't have 
issues with it? (Command-line users as well as TUI users, who enter a 
full-fledged path for the "user exclude" option, with drive letter, colon, and 
everything.)
Give me feedback via Gmail or Twitter, please!

Original comment by Steve8x8 on 3 Jun 2013 at 2:01

• Changed state: InfoNeeded

[GoogleCodeExporter]

The TUI now (internally) uses '|' only as delimiter. Like before, commas are 
translated properly - this means you shouldn't notice any difference. 
Nevertheless, feeback is welcome.

Original comment by Steve8x8 on 7 Jun 2013 at 3:02

• Changed state: FixAvail

[GoogleCodeExporter]

This has been in devel for a long time now, and recently merged into stable for 
3.16.8
Closing

Original comment by Steve8x8 on 27 Jun 2013 at 9:24

• Changed state: Fixed