search

Call-for-Code-for-Racial-Justice / Legit-Info

"Legit Info" app helps residents understand the impact of legislation and policies based on their country, state, county, city, district location
Apache License 2.0
67 stars 74 forks source link

Code does not extract CA, TX, GA, and MN #87

Closed upkarlidder closed 3 years ago

upkarlidder commented 3 years ago

The code extracts files for US and AZ. Then moves quickly from CA, GA, and MN without extracting anything. It then starts extracting OH.

Logs from Getdata job:

Jun 15 17:00:33 Code Engine legit-info-job-getdata-jm6b4-0-0 **Using PostgreSQL**
Jun 15 17:00:41 Code Engine legit-info-job-getdata-jm6b4-0-0 Starting get_datasets at Jun-15 05:00PM MST
Jun 15 17:00:41 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:34 log_time:time_now [INFO] Starting get_datasets at Jun-15 05:00PM MST
Jun 15 17:00:41 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:35 get_datasets:fetch_dsl_api [INFO] Fetching Dataset: DatasetList-2021-06-16.json
Jun 15 17:00:41 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:37 get_datasets:handle [INFO] Processing: United States (US)
Jun 15 17:00:41 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:38 get_datasets:handle [INFO] Processing: Arizona, USA (AZ)
Jun 15 17:00:41 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:39 get_datasets:handle [INFO] Processing: California, USA (CA)
Jun 15 17:00:41 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:40 get_datasets:handle [INFO] Processing: Georgia, USA (GA)
Jun 15 17:00:41 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:40 get_datasets:handle [INFO] Processing: Minnesota, USA (MN)
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:42 get_datasets:handle [INFO] Processing: Ohio, USA (OH)
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:42 get_datasets:handle [INFO] Processing: Texas, USA (TX)
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:43 get_datasets:show_results [INFO] Session 1823 Year: 2021-2022 Date: 2021-06-13 Size: 15403964 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: US-Dataset-1823.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:43 get_datasets:show_results [INFO] Session 1658 Year: 2019-2020 Date: 2021-01-24 Size: 43863720 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: US-Dataset-1658.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:43 get_datasets:show_results [INFO] Session 1794 Year: 2021-2021 Date: 2021-06-13 Size: 6191050 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: AZ-Dataset-1794.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:43 get_datasets:show_results [INFO] Session 1717 Year: 2020-2020 Date: 2020-12-25 Size: 4792708 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: AZ-Dataset-1717.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:43 get_datasets:show_results [INFO] Session 1623 Year: 2019-2019 Date: 2020-12-25 Size: 4655138 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: AZ-Dataset-1623.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:43 get_datasets:show_results [INFO] Session 1791 Year: 2021-2022 Date: 2021-06-13 Size: 7359183 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: CA-Dataset-1791.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:43 get_datasets:show_results [INFO] Session 1624 Year: 2019-2020 Date: 2020-12-25 Size: 17505836 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: CA-Dataset-1624.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1786 Year: 2021-2022 Date: 2021-06-13 Size: 4288501 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: GA-Dataset-1786.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1738 Year: 2020-2020 Date: 2020-12-25 Size: 118521 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: GA-Dataset-1738.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1618 Year: 2019-2020 Date: 2020-12-25 Size: 8040234 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: GA-Dataset-1618.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1770 Year: 2021-2022 Date: 2021-06-13 Size: 7154106 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: MN-Dataset-1770.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1787 Year: 2020-2020 Date: 2020-12-25 Size: 210056 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: MN-Dataset-1787.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1779 Year: 2020-2020 Date: 2020-12-25 Size: 142542 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: MN-Dataset-1779.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1773 Year: 2020-2020 Date: 2020-12-25 Size: 236953 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: MN-Dataset-1773.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: MN-Dataset-1767.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1767 Year: 2020-2020 Date: 2020-12-25 Size: 172166 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1764 Year: 2020-2020 Date: 2020-12-25 Size: 180536 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: MN-Dataset-1764.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1757 Year: 2020-2020 Date: 2020-12-25 Size: 339520 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: MN-Dataset-1757.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1749 Year: 2020-2020 Date: 2020-12-25 Size: 612301 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: MN-Dataset-1749.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:44 get_datasets:show_results [INFO] Session 1588 Year: 2019-2020 Date: 2020-12-25 Size: 13795492 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: MN-Dataset-1588.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:45 get_datasets:show_results [INFO] Session 1666 Year: 2019-2019 Date: 2020-12-25 Size: 175403 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: MN-Dataset-1666.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:45 get_datasets:show_results [INFO] Session 1815 Year: 2021-2022 Date: 2021-06-13 Size: 1919109 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: OH-Dataset-1815.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:45 get_datasets:show_results [INFO] Session 1646 Year: 2019-2020 Date: 2021-01-24 Size: 4942412 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: OH-Dataset-1646.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:45 get_datasets:show_results [INFO] Session 1776 Year: 2021-2022 Date: 2021-06-13 Size: 19556774 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: TX-Dataset-1776.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:45 get_datasets:show_results [INFO] Session 1611 Year: 2019-2020 Date: 2020-12-25 Size: 20963127 bytes
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Found session dataset: TX-Dataset-1611.json
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 Ending get_datasets at Jun-15 05:00PM MST
Jun 15 17:00:52 Code Engine legit-info-job-getdata-jm6b4-0-0 06-16 00:00:45 log_time:time_now [INFO] Ending get_datasets at Jun-15 05:00PM MST

Logs from Extractdata job:

Jun 16 14:39:33 Code Engine legit-info-job-extractfiles-jobrun-kvjfw-0-0 06-16 21:39:28 extract_files:handle [INFO] 194:Processing: Arizona, USA (AZ)
Jun 16 14:41:13 Code Engine legit-info-job-extractfiles-jobrun-kvjfw-0-0 .......>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.>.
...
...
Jun 16 14:43:43 Code Engine legit-info-job-extractfiles-jobrun-kvjfw-0-0 06-16 21:43:38 extract_files:handle [INFO] 194:Processing: California, USA (CA)
Jun 16 14:43:43 Code Engine legit-info-job-extractfiles-jobrun-kvjfw-0-0 06-16 21:43:39 extract_files:handle [INFO] 194:Processing: Georgia, USA (GA)
Jun 16 14:43:43 Code Engine legit-info-job-extractfiles-jobrun-kvjfw-0-0 06-16 21:43:40 extract_files:handle [INFO] 194:Processing: Minnesota, USA (MN)
Jun 16 14:43:43 Code Engine legit-info-job-extractfiles-jobrun-kvjfw-0-0 06-16 21:43:42 extract_files:handle [INFO] 194:Processing: Ohio, USA (OH)
Jun 16 14:43:53 Code Engine legit-info-job-extractfiles-jobrun-kvjfw-0-0 06-16 21:43:46 extract_files:process_source [WARNING] 318:No texts found for OH-HB52-1815
Jun 16 14:44:05 Code Engine legit-info-job-extractfiles-jobrun-kvjfw-0-0 06-16 21:43:56 extract_files:process_source [WARNING] 318:No texts found for OH-SR379-1422
Jun 16 14:44:05 Code Engine legit-info-job-extractfiles-jobrun-kvjfw-0-0 06-16 21:43:57 extract_files:process_source [WARNING] 318:No texts found for OH-SCR59-1422

FYI @tonypearson @tnadams @demilolu

upkarlidder commented 3 years ago

@tnadams @tonypearson I added a try and except in the Hash model. The code was failing previously if it did not find a hash value. Instead, I just set record to None so a new hash is created. Please validate that this is the right thing to do.

    def find_item_name(name, mode=settings.FOB_METHOD):
        """ get item if exists, None if not """

        try:
            record = Hash.objects.get(item_name=name,
                                     fob_method=mode)
        except ObjectDoesNotExist:
            record = None
        return record

That seems to have fixed the problem. I see logs for the new states now in extractjob. Here are sample of logs:

TX

Jun 17 20:21:37 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:29 extract_files:handle [INFO] 194:Processing: Texas, USA (TX)
Jun 17 20:21:37 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:33 extract_files:write_file [INFO] 478:Writing: TX-HB0001-1776-Y2021.txt
Jun 17 20:21:37 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:34 extract_files:write_file [INFO] 478:Writing: TX-HB0002-1776-Y2021.txt
Jun 17 20:21:37 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:35 extract_files:write_file [INFO] 478:Writing: TX-HB0003-1776-Y2021.txt
Jun 17 20:21:37 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:35 extract_files:write_file [INFO] 478:Writing: TX-HB0004-1776-Y2021.txt
Jun 17 20:21:37 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:36 extract_files:write_file [INFO] 478:Writing: TX-HB0005-1776-Y2021.txt
Jun 17 20:21:37 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:37 extract_files:write_file [INFO] 478:Writing: TX-HB0006-1776-Y2021.txt
Jun 17 20:21:37 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:37 extract_files:write_file [INFO] 478:Writing: TX-HB0007-1776-Y2021.txt
Jun 17 20:21:49 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:38 extract_files:write_file [INFO] 478:Writing: TX-HB0008-1776-Y2021.txt
Jun 17 20:21:49 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:38 extract_files:write_file [INFO] 478:Writing: TX-HB0009-1776-Y2021.txt
Jun 17 20:21:49 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:39 extract_files:write_file [INFO] 478:Writing: TX-HB0010-1776-Y2021.txt
Jun 17 20:21:49 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:39 extract_files:write_file [INFO] 478:Writing: TX-HB0011-1776-Y2021.txt
Jun 17 20:21:49 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:40 extract_files:write_file [INFO] 478:Writing: TX-HB0012-1776-Y2021.txt
Jun 17 20:21:49 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 03:21:40 extract_files:write_file [INFO] 478:Writing: TX-HB0013-1776-Y2021.txt

CA

Jun 17 19:22:04 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 02:21:58 extract_files:write_file [INFO] 478:Writing: CA-SB1308-1624-Y2020.txt
Jun 17 19:22:04 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 02:21:59 extract_files:write_file [INFO] 478:Writing: CA-SB1309-1624-Y2020.txt
Jun 17 19:22:05 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 02:22:00 extract_files:write_file [INFO] 478:Writing: CA-SB1310-1624-Y2020.txt
Jun 17 19:22:05 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 02:22:01 extract_files:write_file [INFO] 478:Writing: CA-SB1311-1624-Y2020.txt
Jun 17 19:22:05 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 02:22:02 extract_files:write_file [INFO] 478:Writing: CA-SB1312-1624-Y2020.txt
Jun 17 19:22:05 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 02:22:02 extract_files:write_file [INFO] 478:Writing: CA-SB1313-1624-Y2020.txt
Jun 17 19:22:05 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 02:22:03 extract_files:write_file [INFO] 478:Writing: CA-SB1314-1624-Y2020.txt
Jun 17 19:22:05 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 02:22:04 extract_files:write_file [INFO] 478:Writing: CA-SB1315-1624-Y2020.txt
Jun 17 19:22:15 Code Engine legit-info-job-extractfiles-jobrun-gcjhm-0-0 06-18 02:22:06 extract_files:write_file [INFO] 478:Writing: CA-SB1316-1624-Y2020.txt
upkarlidder commented 3 years ago

The extractjob is still running. I will check for the rest of the states and then run analyzejob after.

github-actions[bot] commented 3 years ago

:wave: Hi! This issue has been marked stale due to inactivity. If no further activity occurs, it will automatically be closed.