josh-chamberlain commented 1 year ago

Scrapers repo

These serve as examples of different ways to access data. They're also individually useful.

Problems with the current repo

too intimidating
too complex
- we're trying to systematize everything; scrapers should be standalone
people aren't sure how to contribute
- people don't know what the code is for
- there's no good example of how the code can be used

How people use the repo

find a way to help
run scrapers locally
- find them geographically
- sorted by language
people can run utilities for writing scrapers
- common files
serve as a library of other scrapers

To do

Readme changes

[x] #212
[x] add a visual representation of how this relates to our other work (josh)
- scrapers outside this repo
- investigative process
- data sources
- work being done with data

Issue adjustment

[x] Make issues in the Planning repo for wrappers / parsers for common data portals (josh)
- Kinda like this https://github.com/dwillis/fech-ftp

Structure changes

[x] explain the repo's new structure in the README
[x] reorganize according to the tree below
- one top-level directory for all utilities and common scripts
- organized geographically—the primary way for finding a scraper will not be through these directories, but by searching for Data Sources with a scraper_url present (so we can use record_type, location, any other Data Source property) or via a scraper index file: #196
- one major reason for this is that the "common" scripts are all over the place, so for a new user it's incredibly difficult to figure out how they relate.

setup_gui/Base_Scripts/Scrapers/crimegraphics/crimegraphics_bulletin.py

common/base_scrapers/crimegraphics/crimegraphics_bulletin.py

Base_Scripts/Scrapers/crimegraphics/crimegraphics_bulletin.py

CODE_OF_CONDUCT.md
CONTRIBUTING.md
LICENSE.md
README.md
requirements.txt
examples_templates/
   -- scraper_template/
     -- README.md
     -- scraper.py
   -- scraper_example_1/
      -- README.md
      -- scraper.py
   -- etc
scrapers/
    -- data_portals/
        -- cityprotect/
        -- crimegraphics/
             -- README.md
             -- crimegraphics.py
    -- federal/
    -- AR/
    -- CA/
    -- FL/
        -- scraper/
        -- county/
            -- scraper/
                -- scraper.py
                -- README.md
            -- municipality/
                -- scraper/
                    -- scraper.py
                    -- README.md
    -- etc
utils/
  -- meta/
    -- all_fields_extractor/
    -- etc
  -- setup_gui/
  -- etc

EvilDrPurple commented 11 months ago

Just want a clarification real quick, you put:

scrapers/
    -- data_portals/
    -- cityprotect/
    -- crimegraphics/
         -- README.md
         -- crimegraphics.py
  -- federal/

You indented this more than the other lines so I'm assuming you want what is currently the top-level Base_Scripts directory to go inside the scrapers folder, or did you want them inside the utils folder instead?

one major reason for this is that the "common" scripts are all over the place, so for a new user it's incredibly difficult to figure out how they relate.

Based on this line I'm also assuming you want some of the duplicate scripts in various places removed and streamlined to one area. Just want to make sure this is what you meant

josh-chamberlain commented 11 months ago

@EvilDrPurple yes, the duplicate scripts should each have one home. I grouped some of them under data_portals/ and some under utils/ because there are both kinds, and they do different things—data portal scrapers can be run on any portal built on that platform, the utils do other random stuff.

I know it might be a pain taking something that's currently spread across 3 random directories and refactoring it to work in just 1. As you work, if you find things that are broken when you get there, let's talk about whether it's worth fixing for the sake of reorganizing the repo vs. just pulling it out / putting it on ice somehow.

Police-Data-Accessibility-Project / scrapers

(Don't Fear the) Repo overhaul #197