Fix issues flake8 flags

[ormiret]

Was looking at adding flake8 run as part of test action but will need to fix the issues it's currently flagging first:

ormiret@dunkoked:~/Documents/ctc/the_od_bods$ flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
./web-scrapers/aberdeenshire_council_scraper.py:110:83: F821 undefined name 'title'
                "car parks" or "cycle" or "grit" or "harbours" or "core paths" in title
                                                                                  ^
1     F821 undefined name 'title'
1
ormiret@dunkoked:~/Documents/ctc/the_od_bods$ flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
./dcat.py:4:1: E722 do not use bare 'except'
./dcat.py:59:5: E722 do not use bare 'except'
./processor.py:1:1: F401 'urllib.parse' imported but unused
./processor.py:69:9: E722 do not use bare 'except'
./arcgis.py:4:1: E722 do not use bare 'except'
./merge_data.py:1:1: E266 too many leading '#' for block comment
./merge_data.py:4:1: F401 'datetime as dt' imported but unused
./merge_data.py:7:1: C901 'merge_data' is too complex (13)
./merge_data.py:8:5: E266 too many leading '#' for block comment
./merge_data.py:10:5: E266 too many leading '#' for block comment
./merge_data.py:14:5: E266 too many leading '#' for block comment
./merge_data.py:30:5: E266 too many leading '#' for block comment
./merge_data.py:46:5: E266 too many leading '#' for block comment
./merge_data.py:65:5: E266 too many leading '#' for block comment
./merge_data.py:82:5: E266 too many leading '#' for block comment
./merge_data.py:98:5: E266 too many leading '#' for block comment
./merge_data.py:111:5: E266 too many leading '#' for block comment
./merge_data.py:114:5: E266 too many leading '#' for block comment
./merge_data.py:117:5: E266 too many leading '#' for block comment
./merge_data.py:123:1: C901 'clean_data' is too complex (15)
./merge_data.py:132:5: E266 too many leading '#' for block comment
./merge_data.py:135:5: E266 too many leading '#' for block comment
./merge_data.py:149:5: E266 too many leading '#' for block comment
./merge_data.py:153:5: E266 too many leading '#' for block comment
./merge_data.py:155:5: E266 too many leading '#' for block comment
./merge_data.py:158:5: E266 too many leading '#' for block comment
./merge_data.py:175:5: E266 too many leading '#' for block comment
./merge_data.py:195:5: E266 too many leading '#' for block comment
./merge_data.py:204:9: E266 too many leading '#' for block comment
./merge_data.py:595:9: E266 too many leading '#' for block comment
./merge_data.py:602:9: E266 too many leading '#' for block comment
./merge_data.py:611:5: E266 too many leading '#' for block comment
./merge_data.py:614:5: E266 too many leading '#' for block comment
./usmart.py:3:1: E722 do not use bare 'except'
./export2jkan.py:4:1: F401 'math.isnan' imported but unused
./export2jkan.py:6:1: F401 're' imported but unused
./export2jkan.py:75:5: E722 do not use bare 'except'
./export2jkan.py:79:5: E722 do not use bare 'except'
./export2jkan.py:118:18: E741 ambiguous variable name 'l'
./export2jkan.py:130:8: E741 ambiguous variable name 'l'
./export2jkan.py:132:8: E741 ambiguous variable name 'l'
./export2jkan.py:134:8: E741 ambiguous variable name 'l'
./export2jkan.py:137:8: E713 test for membership should be 'not in'
./export2jkan.py:145:1: E266 too many leading '#' for block comment
./tools/alive.py:3:1: F401 'github.GithubIntegration' imported but unused
./tools/alive.py:14:24: E711 comparison to None should be 'if cond is None:'
./tools/alive.py:18:25: E711 comparison to None should be 'if cond is None:'
./tools/alive.py:22:1: C901 'TryExcept 22' is too complex (13)
./tools/alive.py:48:13: F841 local variable 'e' is assigned to but never used
./tools/alive.py:62:27: E712 comparison to False should be 'if cond is False:' or 'if not cond:'
./tests/conftest.py:2:1: F401 'pytest' imported but unused
./web-scrapers/east_ayrshire_scraper.py:13:1: E302 expected 2 blank lines, found 1
./web-scrapers/.#aberdeenshire_council_scraper.py:0:1: E902 FileNotFoundError: [Errno 2] No such file or directory: './web-scrapers/.#aberdeenshire_council_scraper.py'
./web-scrapers/aberdeenshire_council_scraper.py:8:1: E302 expected 2 blank lines, found 1
./web-scrapers/aberdeenshire_council_scraper.py:87:1: C901 'parse_feeds' is too complex (16)
./web-scrapers/aberdeenshire_council_scraper.py:110:83: F821 undefined name 'title'
./web-scrapers/aberdeenshire_council_scraper.py:179:5: E266 too many leading '#' for block comment
./web-scrapers/nls_scraper.py:163:24: E711 comparison to None should be 'if cond is None:'
./web-scrapers/nls_scraper.py:170:1: C901 'fetch_file_size' is too complex (13)
./web-scrapers/nls_scraper.py:233:20: E711 comparison to None should be 'if cond is None:'
./web-scrapers/nls_scraper.py:257:20: E711 comparison to None should be 'if cond is None:'
./web-scrapers/nls_scraper.py:388:13: E266 too many leading '#' for block comment
./web-scrapers/nls_scraper.py:389:45: E231 missing whitespace after ','
./web-scrapers/nls_scraper.py:389:51: E261 at least two spaces before inline comment
./web-scrapers/nls_scraper.py:389:52: E262 inline comment should start with '# '
./web-scrapers/nls_scraper.py:389:75: W291 trailing whitespace
5     C901 'merge_data' is too complex (13)
1     E231 missing whitespace after ','
1     E261 at least two spaces before inline comment
1     E262 inline comment should start with '# '
28    E266 too many leading '#' for block comment
2     E302 expected 2 blank lines, found 1
5     E711 comparison to None should be 'if cond is None:'
1     E712 comparison to False should be 'if cond is False:' or 'if not cond:'
1     E713 test for membership should be 'not in'
7     E722 do not use bare 'except'
4     E741 ambiguous variable name 'l'
1     E902 FileNotFoundError: [Errno 2] No such file or directory: './web-scrapers/.#aberdeenshire_council_scraper.py'
6     F401 'math.isnan' imported but unused
1     F821 undefined name 'title'
1     F841 local variable 'e' is assigned to but never used
1     W291 trailing whitespace
66
ormiret@dunkoked:~/Documents/ctc/the_od_bods$ 

[kernelguardian]

@JackGilmore is this up for grabs?

[JackGilmore]

@ormiret are you still working on this or are you happy for @scrapgists to have a go at it?

[ormiret]

I'm definitely happy for other people to take things off my TODO list :) It generally increases the chances that the things actually get done.

The 5 things flagged C901 "too complex" might need fairly major reworking (and I'd like tests for that code before refactoring it...) so maybe just go with flagging them so flake8 will pass for now and file other tickets for the refactoring there.