Create script that keeps track of Google search placement for common searches

[GoogleCodeExporter]

Prerequisites: basic scripting abilities.

Task: Write a script that does a simple Google search for a couple of common 
search terms, and log 
Plone's position in these searches, so we can see trends.

Log the position for:
"content management system", "cms", "content management", (suggestions welcome)
and send them to a dedicated mail address. Also log the pagerank of plone.org.

Original issue reported on code.google.com by l...@gmail.com on 20 Nov 2007 at 12:12

[GoogleCodeExporter]

Original comment by l...@gmail.com on 27 Nov 2007 at 7:12

• Changed state: Open
• Added labels: 3-day

[GoogleCodeExporter]

"I claim this task."

Original comment by lobais on 28 Nov 2007 at 4:48

[GoogleCodeExporter]

Hey, I have some partly working code now. (Attached)

It seams that google no longer lets one use the SOAP API, and that they start 
sending
those image codes, if you search too much in short time.
However, if the script is only ran once or twice per day, nobody should notice.

For the pagerank, it seams to be available only from the toolbar and
dictionary.google.com, but as plone.org is in the dictionary, I can probably 
get it
from there.

Cheers, Thomas

Original comment by pych...@gmail.com on 28 Nov 2007 at 7:30

Attachments:

• search.py

[GoogleCodeExporter]

Oops, wrote that with a wrong account. Sry.

However I'm thinking why you want this to email the results.
I believe it would be more nice to run it as a cronscript, appending the 
results to a
textfile, which could later be used for graphs?

- Thomas

Original comment by lobais on 29 Nov 2007 at 10:33

[GoogleCodeExporter]

I claim this Task.

I have already made a script like this and it worked. I've even found a way how 
to do
Google search without being blocked as a robot.

Original comment by gadu...@gmail.com on 29 Nov 2007 at 10:58

[GoogleCodeExporter]

Unfortunately, gadulix, Thomas beat you to it. We have yet to review his
contribution, but we'd need to reject it for you to be able to do this, 
according to
the contest rules. I'm sure Thomas has done a good job, but feel free to check 
back
if the task has been reclaimed later.

As for Thomas, we'll review your entry over the next day or two. Thank you for a
quick response!

Original comment by optil...@gmail.com on 29 Nov 2007 at 11:03

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Original comment by optil...@gmail.com on 29 Nov 2007 at 11:15

• Changed state: Claimed

[GoogleCodeExporter]

This new version adds support for commandline arguments (see --help), emailing 
and
should make the code a bit more easy on google.

I'm having a little trouble with sendmail though.
Everywhere I've seen it used, it's used like in the script, but I doesn't seam 
to
receive any of the emails sent. Not even in my spam folder.
I can't work sendmail work outside the script either though, so perhaps it's 
just my
configuration.

Original comment by lobais on 1 Dec 2007 at 10:27

Attachments:

• search.py

[GoogleCodeExporter]

I've had a play with this now.

The searching is clean, the code is well documented and things like taking into
account "subresults" is very good.  

There was a small error in the --help code, you used .trim() instead if 
.strip(). 
Sendmail integration didn't work for me, I get:

sendmail: fatal: Recipient addresses must be specified on the command line or 
via the
-t option
Sendmail exit status: 75 Program version wrong

Which is clearly wrong as I can see you have specified -t, but if you change the
instantiation of the sendmail process to [SENDMAIL_PATH + " -t"] I get error 0,
unknown error, which may be to do with my implementation.  Who knows!

Either way, this is an excellent piece of work and I'm going to close this 
task. 
Well done!

Original comment by mw4...@googlemail.com on 3 Dec 2007 at 4:56

• Changed state: Closed

[GoogleCodeExporter]

Original comment by ospoad...@gmail.com on 8 Dec 2007 at 6:06

• Added labels: claimedby-lobias