Scraping locally v2.0 - Githubissues

wilbertom commented 10 years ago

I had some issues scraping locally.


(ccj)~/Programming/SC3/cookcountyjail (v2.0-dev) wil$ ./scripts/scraper.sh 
Cook County Jail 2.0 API scraper started at Wed May  7 11:40:08 CDT 2014
./scripts/scraper.sh: line 22: /home/wil/.virtualenvs/cookcountyjail_2.0-dev/bin/activate: No such file or directory
./scripts/scraper.sh: line 24: cd: /home/wil/website/2.0/websites/active: No such file or directory
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/wil/Programming/SC3/cookcountyjail/scripts/scraper.py", line 82, in <module>
    dpc = DailyPopulation(get_dpc_dir())
  File "ccj/models/daily_population.py", line 39, in __init__
    self._initialize_file()
  File "ccj/models/daily_population.py", line 117, in _initialize_file
    self.clear()
  File "ccj/models/daily_population.py", line 74, in clear
    "at '{0}'.".format(self._path))
Exception: There's something wrong with the path configured for our file's creation on your system, at '/home/ubuntu/website/2.0/data/dpc.csv'.
Cook County Jail 2.0 API scraper finished at Wed May  7 11:40:09 CDT 2014

Then python -m scripts.scraper threw a 500 on the server. So I had to:

diff --git a/scripts/ccj_api_v1.py b/scripts/ccj_api_v1.py
index 7c523ba..d41c2d4 100644
--- a/scripts/ccj_api_v1.py
+++ b/scripts/ccj_api_v1.py
@@ -58,7 +58,7 @@ class CcjApiV1:
     @staticmethod
     def _convert_to_beginning_of_day(starting_date):
         starting_date_time = datetime.strptime(starting_date, DATE_FORMAT)
-        return starting_date_time.replace(hour=0, minute=0, second=0, microsecond=0).isoformat()
+        return starting_date_time.date()

     @staticmethod
     def _convert_to_end_of_day(ending_date):

@nwinklareth does this have to do with the fixing of #355? If so then I can apply the patch above. If not then I must have something wrong in my environment.

nwinklareth commented 10 years ago

Yes that is related to #355, your fix is correct.

On Wed, May 7, 2014 at 11:48 AM, Wilberto Morales notifications@github.comwrote:

I had some issues scraping locally.

(ccj)~/Programming/SC3/cookcountyjail (v2.0-dev) wil$ ./scripts/scraper.sh Cook County Jail 2.0 API scraper started at Wed May 7 11:40:08 CDT 2014 ./scripts/scraper.sh: line 22: /home/wil/.virtualenvs/cookcountyjail_2.0-dev/bin/activate: No such file or directory ./scripts/scraper.sh: line 24: cd: /home/wil/website/2.0/websites/active: No such file or directory Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/wil/Programming/SC3/cookcountyjail/scripts/scraper.py", line 82, in dpc = DailyPopulation(get_dpc_dir()) File "ccj/models/daily_population.py", line 39, in init self._initialize_file() File "ccj/models/daily_population.py", line 117, in _initialize_file self.clear() File "ccj/models/daily_population.py", line 74, in clear "at '{0}'.".format(self._path)) Exception: There's something wrong with the path configured for our file's creation on your system, at '/home/ubuntu/website/2.0/data/dpc.csv'. Cook County Jail 2.0 API scraper finished at Wed May 7 11:40:09 CDT 2014

Then python -m scripts.scraper threw a 500 on the server. So I had to:

diff --git a/scripts/ccj_api_v1.py b/scripts/ccj_api_v1.py index 7c523ba..d41c2d4 100644 --- a/scripts/ccj_api_v1.py +++ b/scripts/ccj_api_v1.py @@ -58,7 +58,7 @@ class CcjApiV1: @staticmethod def _convert_to_beginning_of_day(starting_date): starting_date_time = datetime.strptime(starting_date, DATE_FORMAT)
return starting_date_time.replace(hour=0, minute=0, second=0, microsecond=0).isoformat()
 return starting_date_time.date()
@staticmethod def _convert_to_end_of_day(ending_date):
@nwinklareth https://github.com/nwinklareth does this have to do with the fixing of #355 https://github.com/sc3/cookcountyjail/issues/355? If so then I can apply the patch above. If not then I must have something wrong in my environment.

— Reply to this email directly or view it on GitHubhttps://github.com/sc3/cookcountyjail/issues/362 .

Regards

Norbert

Norbert Winklareth

bepetersn commented 10 years ago

Did you apply this patch, Wil?

wilbertom commented 10 years ago

Ill do it right now.

sc3 / cookcountyjail

Scraping locally v2.0 #362

Norbert