Improve Tools Documentation

[ben-nathanson]

There are a number of handy tools we should document more well:

Scrapy shell, selectors, etc.:

• Scrapy Shell

Regular Expressions:

• Regex syntax
• Regex checking tools

Figuring out how a website changes over time:

• WaybackMachine

Reading Spider Output

• Scrapy -o flag
• jsonprettify

---

IDEs

Atom

• Pretty JSON
• Platformio IDE Terminal

  Pycharm

  Sublime Text

  Vim

  ---

  How to tackle a page efficiently:

  1. JSON
  2. Legistar
  3. Regex

Feel free to comment below with your favorite tools and I will add them!

[maxachis]

I'm happy to contribute to this, because I looooove documentation. Although I think other people can also contribute to this.

[maxachis]

@bonfirefan @ben-nathanson I think for most or all of these, it makes sense to add another documentation page dedicated to "Tips and Tricks", or "Building Better Spiders", or whatever. The "Development" page has a lot of material as is, and this doesn't really fit into "Troubleshooting". I'll start by putting together a separate .md page with that in mind.

[maxachis]

Assuming my most recent commits are accepted, the next question would be what else requires bolstering?

[maxachis]

We may want to add tools for how to do spiders on PDF files, because as far as I'm aware we don't yet have much in the way of documentation for that.