Refactor presences script

fgrehm commented 7 years ago

This builds on top of a pairing session I had with @cuducos and it is made of:

The extraction of some utility methods to be shared with other scripts
Lots of cleanup
More information extracted / normalized for the Deputies dataset:
- registration was renamed to congressperson_document
- name was renamed to congressperson_name
- New fields: picture_url, gender, phone_number, email
More information extracted / normalized for the Presences dataset:
- registration was renamed to congressperson_document
- presence was translated to english
- New fields: term, present_on_day, justification
Logic for fetching presences made more robust by issuing retries and dealing with the API "flakyness"
On the SessionStartTimes, the session field was renamed to description

fgrehm commented 7 years ago

Almost forgot, here's da data 😋

And the execution logs

fgrehm commented 7 years ago

@cuducos @Irio @jtemporal I've finally had a chance to get back to this 😅

Please take a look at the latest changes and LMK what u think about them. I've also fetched a new version of the datasets that would be nice to be uploaded to S3:

2017-04-19-session-start-times.xz.zip
2017-04-19-presences.xz.zip
2017-04-19-deputies.xz.zip
~~2017-04-19-speeches.xz.zip~~ Only fetched the data for 2015
2017-04-21-speeches.xz.zip (fixed)
Execution logs
Execution logs for fixed speeches

(as usual, github does not let me upload .xz files so I had to zip them in order to upload, make sure to unzip and upload the xz only 😄 )

cuducos commented 7 years ago

Just in case I just uploaded these datasets to S3:

2017-04-19-session-start-times.xz.zip
2017-04-19-presences.xz.zip
2017-04-19-deputies.xz.zip
2017-04-21-speeches.xz.zip

fgrehm commented 7 years ago

Here's another round of updates based on the new code:

fgrehm commented 7 years ago

@cuducos I added some integration tests for the new datasets and some unit tests for the helper methods introduced. Do u think we need some additional testing for the datasets or the ~~sanity checks~~ integration tests I introduced are enough?

cuducos commented 7 years ago

As landscape.io is temporally not running in CI mode, I ran prospector locally, this is the report if it helps:

Messages
========

chamber_of_deputies/deputies_dataset.py
  Line: 12
    pep8: E302 / expected 2 blank lines, found 1 (col 1)
  Line: 71
    pep8: E302 / expected 2 blank lines, found 1 (col 1)

chamber_of_deputies/presences_dataset.py
  Line: 9
    pyflakes: F401 / 'serenata_toolbox.datasets.helpers.xml_extract_date' imported but unused (col 1)
  Line: 17
    pep8: E302 / expected 2 blank lines, found 1 (col 1)
  Line: 20
    pep8: E501 / line too long (94 > 79 characters) (col 80)
  Line: 33
    pep8: E501 / line too long (108 > 79 characters) (col 80)
  Line: 55
    pep8: E501 / line too long (84 > 79 characters) (col 80)
  Line: 56
    pep8: E501 / line too long (87 > 79 characters) (col 80)
  Line: 59
    pep8: E711 / comparison to None should be 'if cond is None:' (col 20)
  Line: 100
    pep8: E303 / too many blank lines (2) (col 5)
  Line: 170
    pep8: E501 / line too long (89 > 79 characters) (col 80)
  Line: 171
    pep8: E501 / line too long (89 > 79 characters) (col 80)

chamber_of_deputies/session_start_times_dataset.py
  Line: 7
    pyflakes: F401 / 'serenata_toolbox.datasets.helpers.translate_column' imported but unused (col 1)
  Line: 14
    pep8: E302 / expected 2 blank lines, found 1 (col 1)
  Line: 16
    pep8: E501 / line too long (86 > 79 characters) (col 80)
  Line: 24
    pep8: E501 / line too long (93 > 79 characters) (col 80)
  Line: 25
    pep8: E501 / line too long (82 > 79 characters) (col 80)
  Line: 39
    pep8: E501 / line too long (92 > 79 characters) (col 80)
  Line: 48
    pep8: E302 / expected 2 blank lines, found 1 (col 1)
  Line: 51
    pep8: E501 / line too long (87 > 79 characters) (col 80)
  Line: 59
    pep8: E501 / line too long (83 > 79 characters) (col 80)

chamber_of_deputies/speeches_dataset.py
  Line: 4
    pyflakes: F401 / 'datetime.datetime' imported but unused (col 1)
  Line: 15
    pep8: E302 / expected 2 blank lines, found 1 (col 1)
  Line: 18
    pep8: E501 / line too long (91 > 79 characters) (col 80)
  Line: 58
    pep8: E221 / multiple spaces before operator (col 24)
  Line: 63
    pep8: E221 / multiple spaces before operator (col 39)
    pep8: E501 / line too long (84 > 79 characters) (col 80)
  Line: 64
    pep8: E221 / multiple spaces before operator (col 40)
    pep8: E501 / line too long (82 > 79 characters) (col 80)
  Line: 65
    pep8: E501 / line too long (85 > 79 characters) (col 80)
  Line: 66
    pep8: E501 / line too long (80 > 79 characters) (col 80)
  Line: 69
    pep8: E501 / line too long (94 > 79 characters) (col 80)
  Line: 71
    pep8: E501 / line too long (105 > 79 characters) (col 80)
  Line: 72
    pep8: E126 / continuation line over-indented for hanging indent (col 33)
  Line: 79
    pep8: E221 / multiple spaces before operator (col 36)
    pep8: E501 / line too long (83 > 79 characters) (col 80)
  Line: 80
    pep8: E501 / line too long (85 > 79 characters) (col 80)
  Line: 97
    pep8: E302 / expected 2 blank lines, found 1 (col 1)

datasets/helpers.py
  Line: 13
    pep8: E302 / expected 2 blank lines, found 1 (col 1)
  Line: 19
    pep8: E711 / comparison to None should be 'if cond is not None:' (col 13)
  Line: 23
    pep8: E302 / expected 2 blank lines, found 1 (col 1)
  Line: 30
    pep8: E302 / expected 2 blank lines, found 1 (col 1)
  Line: 39
    pep8: E302 / expected 2 blank lines, found 1 (col 1)
  Line: 43
    pep8: E501 / line too long (92 > 79 characters) (col 80)
  Line: 47
    pep8: E128 / continuation line under-indented for visual indent (col 20)
  Line: 50
    pep8: E302 / expected 2 blank lines, found 1 (col 1)

fgrehm commented 7 years ago

@cuducos all done 😄

cuducos commented 7 years ago

🎉 Yay! Many tks @fgrehm

okfn-brasil / serenata-toolbox

Refactor presences script #20