search

Police-Data-Accessibility-Project / scrapers

Code relating to scraping public police data.
https://pdap.io
GNU General Public License v3.0
157 stars 33 forks source link

Fix scraper crash #209

Closed EvilDrPurple closed 11 months ago

EvilDrPurple commented 11 months ago

As a consequence of commit 7c0975e an unresolved import pointing to one of the deleted files is causing a crash when running many of the older scrapers that use the common scripts:

  File "/home/kylie/Programming/PDAP-Scrapers/USA/MN/hennepin_county/municipal/minneapolis/archive_opendata.py", line 8, in <module>
    from common import opendata_scraper2
  File "/home/kylie/Programming/PDAP-Scrapers/common/__init__.py", line 8, in <module>
    from .etl.main import schema_load
ModuleNotFoundError: No module named 'common.etl.main'

Also I wanted to ask if we're planning on removing the etl.py files and their references that are sprinkled throughout the scrapers. If so, I can probably clean these up as well.

josh-chamberlain commented 11 months ago

Rockstar! Yes, I attempted to pull out most of the ETL stuff in this PR, but I missed references in the scrapers themselves. Thanks for cleaning up my mess. https://github.com/Police-Data-Accessibility-Project/PDAP-Scrapers/pull/198/files

It's all part of https://github.com/Police-Data-Accessibility-Project/PDAP-Scrapers/issues/197. The vision for what PDAP is has changed significantly since most of the scrapers repo was written.