updates for 10/1 session

[skasim]

Hi Nathan,

I apologize, but, I merged my work with the master instead of creating a separate branch and a pull request, and only realized later that I can't create a pull request unless there are differences between the master and the branch, so I'll be sure to correct that next time.

I was on the scifi bookshelf on Project Gutenberg and saw the warning:

   Note on the Project Gutenberg HTML page
   DON'T USE THIS PAGE FOR SCRAPING.

   Seriously. You'll only get your IP blocked.

   Download http://www.gutenberg.org/feeds/catalog.rdf.bz2 instead,
   which contains *all* Project Gutenberg metadata in one RDF/XML file.

But I found a solution. It involves pip installing gutenberg. Here are the instructions: https://github.com/c-w/Gutenberg. It doesn't work perfectly on my computer because of the issue with the bsddb (documented here: http://stackoverflow.com/questions/814041/how-to-fix-the-broken-bsddb-install-in-the-default-python-package-on-mac-os-x-10). It looked like I would have to change something in a python file and I didn't want to do that. However, It turns out I can still access the book text, which is what I wanted.

I made good progress this week:

• updated the crawler to use the gutenberg module
• added the json code we worked
• rewrote the main file to run everything (admin_gender.py)
• a test csv file of 4 books (the scifi bookshelf has about ~395,000 books) to test the code, the total list is in the forcsv folder.

  Questions:

• Json: My main question is with the json datasource. I will keep working through next week to see if can figure it out, but because the dict I am passing to json is a set, it gives me the error: TypeError: set([1.22, 'Weak on Square Roots']) is not JSON serializable. I think if I can work past this, I'll be in a pretty good place. [fixed]
• Virutal Env: I worked in virtualenv using pew and it was great. However, I have a question about the requirements file (I have included two right now: global and virtual env), which one should I include. The virtual environment does not list a lot of the packages I used like beautifulsoup.

  To Do:

  1. Check to see if csv file exists already. if yes, skip to step 4. if no, go to step 2 and 3 [done]
  2. Go to Project Gutenberg scifi bookshelf [done]
  3. Scrape form the bookshelf the names, title, and urls of all scifi books and add them to a csv file [done]
  4. Access and read the csv file [done]
  5. For reach url in the file access the book text using the gutenberg module [done]
  6. Read each text and identify the male-female gendered words ratio [done]
  7. Add the {author:{title, ratio} to json data source

[ndanielsen]

first thought: You might be using the global path in mac instead of the local path in your virtualenv.

[ndanielsen]

Taking a deeper look at your code, and I'm really proud of the progress that you've made and some of the great patterns that you've developed. Now that you've got a good grasp on functions, basic data structures and file i/o, I think that it's time to work on classes (or objects) in python. I'll create a separate ticket.