search

tilezen / vector-datasource

Tilezen vector tile service - OpenStreetMap data in several formats
https://www.nextzen.org/
Other
509 stars 119 forks source link

Add python linters #2027

Closed peitili closed 2 years ago

peitili commented 2 years ago

This is the 3rd wave of linters for vector-datasource.

In this PR we introduce 3 linters

  1. flake8 (this is not really an introduce since the repo used it before, this change just moves it to be managed and triggered by pre-commit manager and thus we remove the standalone flake8 step on the Circle CI.
  2. autopep8: auto fixes non PEP8 codes
  3. reorder_python_imports: removes unused imports and reorders import lines to alphabetic order and make multi-import line to multiple lines, this is for better readability and reduce chance of merge conflicts.
  4. yesqa: remove unnecessary python comments such as unnecessary # noqa etc.
  5. sort requirements file
  6. enforce single quote python string for consistency.

It also fixes some detected violations in yaml/places.yaml

The following are the output of the initial linters run:

Check docstring is first.................................................Passed
Check Yaml...............................................................Failed
- hook id: check-yaml
- exit code: 1

Traceback (most recent call last):
  File "/Users/pli2/.cache/pre-commit/repobzzzmhye/py_env-python2.7/bin/check-yaml", line 8, in <module>
    sys.exit(main())
  File "/Users/pli2/.cache/pre-commit/repobzzzmhye/py_env-python2.7/lib/python2.7/site-packages/pre_commit_hooks/check_yaml.py", line 63, in main
    load_fn(f)
  File "/Users/pli2/.cache/pre-commit/repobzzzmhye/py_env-python2.7/lib/python2.7/site-packages/ruamel/yaml/main.py", line 343, in load
    return constructor.get_single_data()
  File "/Users/pli2/.cache/pre-commit/repobzzzmhye/py_env-python2.7/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 113, in get_single_data
    return self.construct_document(node)
  File "/Users/pli2/.cache/pre-commit/repobzzzmhye/py_env-python2.7/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 123, in construct_document
    for _dummy in generator:
  File "/Users/pli2/.cache/pre-commit/repobzzzmhye/py_env-python2.7/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 723, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/Users/pli2/.cache/pre-commit/repobzzzmhye/py_env-python2.7/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 439, in construct_mapping
    self.flatten_mapping(node)
  File "/Users/pli2/.cache/pre-commit/repobzzzmhye/py_env-python2.7/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 396, in flatten_mapping
    raise DuplicateKeyError(*args)
ruamel.yaml.constructor.DuplicateKeyError: while constructing a mapping
  in "yaml/places.yaml", line 162, column 7
found duplicate key "<<"
  in "yaml/places.yaml", line 163, column 7

To suppress this check see:
   http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys

Duplicate keys will become an error in future releases, and are errors
by default when using the new API.

Debug Statements (Python)................................................Passed
Check for case conflicts.................................................Passed
Check that executables have shebangs.....................................Passed
Check JSON...............................................................Passed
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing queries.yaml

Reorder python imports...................................................Failed
- hook id: reorder-python-imports
- exit code: 1
- files were modified by this hook

Reordering imports in integration-test/844-normalize-poi-kind.py
Reordering imports in integration-test/806-building-height.py
Reordering imports in integration-test/605-corridor.py
Reordering imports in scripts/test_server.py
Reordering imports in integration-test/976-fractional-pois.py
Reordering imports in integration-test/546-road-sort-keys-aerialway.py
Reordering imports in integration-test/1993-estimate-null-populations.py
Reordering imports in vectordatasource/transform.py
Reordering imports in integration-test/1273-roads-access.py
Reordering imports in integration-test/469-transit-features.py
Reordering imports in integration-test/990-add-art-galleries.py
Reordering imports in integration-test/480-rest_area-services.py
Reordering imports in integration-test/845-buildings-z13.py
Reordering imports in integration-test/1353-ne-min-zoom-roads.py
Reordering imports in integration-test/664-raceway.py
Reordering imports in integration-test/342-winter-sports-pistes.py
Reordering imports in integration-test/657-natural-man_made.py
Reordering imports in integration-test/510-funicular.py
Reordering imports in integration-test/588-funicular-monorail.py
Reordering imports in integration-test/875-camp-grounds-zoom.py
Reordering imports in integration-test/744-remove-osm-neighbourhoods.py
Reordering imports in integration-test/471-categorize-trains.py
Reordering imports in integration-test/890-normalize-ne-roads.py
Reordering imports in integration-test/244-railway-plaforms.py
Reordering imports in integration-test/484-include-state-pois.py
Reordering imports in integration-test/1178-earlier-piers.py
Reordering imports in integration-test/479-barrier-toll_booth.py
Reordering imports in integration-test/344-winter-sports-pois.py
Reordering imports in integration-test/931-locality-changes-places.py
Reordering imports in integration-test/675-man_made-outdoor-landmarks.py
Reordering imports in integration-test/440-zoos-and-other-attractions-barrier.py
Reordering imports in test/test_meta.py
Reordering imports in integration-test/1185-garden-min-zoom.py
Reordering imports in integration-test/1190-mz-colours.py
Reordering imports in integration-test/1982-covered-water.py
Reordering imports in integration-test/549-subways.py
Reordering imports in integration-test/1103-no-natural-pois.py
Reordering imports in integration-test/987-national-forests.py
Reordering imports in integration-test/1910-underground-water.py
Reordering imports in integration-test/232-power-lines.py
Reordering imports in integration-test/1972-big-parking-lots.py
Reordering imports in integration-test/674-outdoor-shops.py
Reordering imports in integration-test/774-cycleway-no.py
Reordering imports in integration-test/662-basic-outdoor-pois.py
Reordering imports in integration-test/1406-include-all-name-variants.py
Reordering imports in integration-test/1695-turning-circles-turning-loops.py
Reordering imports in integration-test/1417-cranes.py
Reordering imports in integration-test/558-default-brunnel-sort-keys.py
Reordering imports in integration-test/843-normalize-underscore.py
Reordering imports in integration-test/1250-early-unclassified-roads.py
Reordering imports in integration-test/922-source-in-POI.py
Reordering imports in integration-test/566-landuse-line.py
Reordering imports in integration-test/970-kind-detail-for-all-roads.py
Reordering imports in integration-test/546-road-sort-keys-roads.py
Reordering imports in integration-test/1186-the-pois-with-no-name.py
Reordering imports in vectordatasource/colour.py
Reordering imports in integration-test/440-zoos-and-other-attractions.py
Reordering imports in integration-test/742-predictable-layers-pois.py
Reordering imports in integration-test/1147-bicycle-ramps.py
Reordering imports in integration-test/__init__.py
Reordering imports in integration-test/599-whitewater.py
Reordering imports in integration-test/981-remove-unstyled-ne-places.py
Reordering imports in data/wof_snapshot.py
Reordering imports in integration-test/291-483-suppress-historical-closed.py
Reordering imports in integration-test/557-missing-buildings.py
Reordering imports in integration-test/832-pedestrian-paths-bicycle.py
Reordering imports in integration-test/148-sea-ocean-labels-water-layer.py
Reordering imports in integration-test/768-multiline-encoded.py
Reordering imports in integration-test/1337-roads-surface-cobblestone-value-transformed.py
Reordering imports in integration-test/840-normalize-place-kind.py
Reordering imports in integration-test/dsl.py
Reordering imports in integration-test/546-road-sort-keys-aeroway.py
Reordering imports in integration-test/1403-fences-walls-landuse.py
Reordering imports in integration-test/192-shield-text-ref.py
Reordering imports in integration-test/343-winter-sports-resorts.py
Reordering imports in integration-test/552-water-boundary-sort-key.py
Reordering imports in integration-test/546-road-sort-keys-tunnel.py
Reordering imports in vectordatasource/meta/sql.py
Reordering imports in integration-test/895-extract-airport-lines-minor_road.py
Reordering imports in integration-test/912-missing-building-part.py
Reordering imports in integration-test/447-ice-cream-shops.py
Reordering imports in integration-test/605-crosswalk-sidewalk.py
Reordering imports in integration-test/860-lighthouse.py
Reordering imports in integration-test/370-prisons.py
Reordering imports in integration-test/830-windmill-zoom.py
Reordering imports in integration-test/592-add-adjust-bicycle-pois.py
Reordering imports in integration-test/404-toys-not-found.py
Reordering imports in integration-test/369-subway-stations-z12.py
Reordering imports in integration-test/398-airport-iata-codes.py
Reordering imports in integration-test/368-disused-railway-stations.py
Reordering imports in integration-test/713-urban-areas.py
Reordering imports in integration-test/594-trailhead.py
Reordering imports in integration-test/628-standardize-water-kinds.py
Reordering imports in integration-test/776-duplicate-footway.py
Reordering imports in integration-test/661-historic-transit-stops.py
Reordering imports in integration-test/488-motorway_link-z11.py
Reordering imports in integration-test/1016-missing-localized-names.py
Reordering imports in integration-test/1995-medium-sized-parks.py
Reordering imports in integration-test/655-landuse-scree.py
Reordering imports in integration-test/1020-roads-surface.py
Reordering imports in integration-test/896-ne-shield-enums.py
Reordering imports in integration-test/399-add-island-labels.py
Reordering imports in integration-test/546-road-sort-keys-bridges.py
Reordering imports in integration-test/857-move_barriers_to_landuse.py
Reordering imports in vectordatasource/collision.py
Reordering imports in vectordatasource/meta/python.py
Reordering imports in integration-test/935-source-in-transit.py
Reordering imports in vectordatasource/meta/function.py
Reordering imports in integration-test/704-exclude-null-values-for-buildings.py
Reordering imports in integration-test/829-garden-pois.py
Reordering imports in integration-test/601-cliff-arete-ridge-valley.py
Reordering imports in integration-test/421-zoos-z13.py
Reordering imports in integration-test/197-clip-buildings.py
Reordering imports in integration-test/1030-invalid-wkb-polygons.py
Reordering imports in integration-test/522-hotels.py
Reordering imports in integration-test/418-wof-l10n_name.py
Reordering imports in integration-test/677-waterfall.py
Reordering imports in integration-test/1218-poni-whitelist.py
Reordering imports in integration-test/859-add-bridleway.py
Reordering imports in integration-test/400-bay-water.py
Reordering imports in integration-test/580-kind-csv.py
Reordering imports in integration-test/1408-building-material.py
Reordering imports in integration-test/993-remove-props-road-merge.py
Reordering imports in integration-test/546-road-sort-keys-railways.py
Reordering imports in setup.py
Reordering imports in integration-test/546-road-sort-keys-layers.py
Reordering imports in integration-test/663-combo-outdoor-landuse-pois.py
Reordering imports in integration-test/1091-missing-name-short.py
Reordering imports in integration-test/507-routes-via-stop-positions.py
Reordering imports in integration-test/982-remove-unstyled-localities.py
Reordering imports in integration-test/852-remove-landuse_labels.py
Reordering imports in integration-test/668-intermittent-water.py
Reordering imports in integration-test/671-ranger-station.py
Reordering imports in integration-test/1955-chinese-parser.py
Reordering imports in scripts/update-integration-test-coordinates.py
Reordering imports in scripts/csv_colours.py
Reordering imports in integration-test/766-dont-merge-z16-roads.py
Reordering imports in integration-test/524-peak-kind-tile-rank.py
Reordering imports in integration-test/382-pier-lines.py
Reordering imports in integration-test/927-normalize-operator-values.py
Reordering imports in integration-test/1171-bicycle-yes-designated-roads.py
Reordering imports in integration-test/448-wine-and-alcohol-shops.py
Reordering imports in integration-test/679-less-landuse-building-label-placements.py
Reordering imports in integration-test/896-ne-shield-enums-2.py
Reordering imports in integration-test/842-normalize-building-kind.py
Reordering imports in integration-test/834-park-building.py
Reordering imports in integration-test/719-add-kind-detail-for-pois.py
Reordering imports in data/bootstrap.py
Reordering imports in integration-test/367-military-landuse.py
Reordering imports in data/tile-shapefile.py
Reordering imports in integration-test/602-add-boat-rental.py
Reordering imports in integration-test/647-cycle-route.py
Reordering imports in integration-test/1224-earlier-bike-properties.py
Reordering imports in integration-test/797-add-missing-boundaries.py
Reordering imports in integration-test/1194-bus-route-refs.py
Reordering imports in integration-test/366-beaches.py
Reordering imports in integration-test/596-add-hiking-routes.py
Reordering imports in integration-test/443-swimming-pools.py
Reordering imports in integration-test/1251-early-track-roads.py
Reordering imports in integration-test/454-aeroway-gates.py
Reordering imports in integration-test/160-motorway-junctions.py
Reordering imports in integration-test/729-route-name.py
Reordering imports in scripts/mktest.py
Reordering imports in integration-test/1979-shield-text-length.py
Reordering imports in integration-test/919-gates-line-geometry.py
Reordering imports in integration-test/593-early-footway.py

Strip unnecessary `# noqa`s..............................................Failed
- hook id: yesqa
- exit code: 1
- files were modified by this hook

Rewriting integration-test/844-normalize-poi-kind.py
Rewriting integration-test/806-building-height.py
Rewriting integration-test/605-corridor.py
Rewriting integration-test/1983-china-south-sea.py
Rewriting integration-test/976-fractional-pois.py
Rewriting integration-test/546-road-sort-keys-aerialway.py
Rewriting vectordatasource/transform.py
Rewriting integration-test/1273-roads-access.py
Rewriting integration-test/469-transit-features.py
Rewriting integration-test/990-add-art-galleries.py
Rewriting integration-test/480-rest_area-services.py
Rewriting integration-test/845-buildings-z13.py
Rewriting integration-test/1353-ne-min-zoom-roads.py
Rewriting integration-test/664-raceway.py
Rewriting integration-test/342-winter-sports-pistes.py
Rewriting integration-test/657-natural-man_made.py
Rewriting integration-test/510-funicular.py
Rewriting integration-test/588-funicular-monorail.py
Rewriting integration-test/875-camp-grounds-zoom.py
Rewriting integration-test/744-remove-osm-neighbourhoods.py
Rewriting integration-test/471-categorize-trains.py
Rewriting integration-test/890-normalize-ne-roads.py
Rewriting integration-test/244-railway-plaforms.py
Rewriting integration-test/484-include-state-pois.py
Rewriting integration-test/1178-earlier-piers.py
Rewriting integration-test/479-barrier-toll_booth.py
Rewriting integration-test/344-winter-sports-pois.py
Rewriting integration-test/931-locality-changes-places.py
Rewriting integration-test/675-man_made-outdoor-landmarks.py
Rewriting integration-test/440-zoos-and-other-attractions-barrier.py
Rewriting integration-test/1185-garden-min-zoom.py
Rewriting integration-test/1190-mz-colours.py
Rewriting integration-test/549-subways.py
Rewriting integration-test/1103-no-natural-pois.py
Rewriting integration-test/987-national-forests.py
Rewriting integration-test/232-power-lines.py
Rewriting integration-test/674-outdoor-shops.py
Rewriting integration-test/774-cycleway-no.py
Rewriting integration-test/662-basic-outdoor-pois.py
Rewriting integration-test/1406-include-all-name-variants.py
Rewriting integration-test/1417-cranes.py
Rewriting integration-test/558-default-brunnel-sort-keys.py
Rewriting integration-test/843-normalize-underscore.py
Rewriting integration-test/1250-early-unclassified-roads.py
Rewriting integration-test/922-source-in-POI.py
Rewriting integration-test/566-landuse-line.py
Rewriting integration-test/970-kind-detail-for-all-roads.py
Rewriting integration-test/546-road-sort-keys-roads.py
Rewriting integration-test/1186-the-pois-with-no-name.py
Rewriting integration-test/440-zoos-and-other-attractions.py
Rewriting integration-test/742-predictable-layers-pois.py
Rewriting integration-test/1147-bicycle-ramps.py
Rewriting integration-test/599-whitewater.py
Rewriting integration-test/981-remove-unstyled-ne-places.py
Rewriting integration-test/291-483-suppress-historical-closed.py
Rewriting integration-test/557-missing-buildings.py
Rewriting integration-test/832-pedestrian-paths-bicycle.py
Rewriting integration-test/148-sea-ocean-labels-water-layer.py
Rewriting integration-test/768-multiline-encoded.py
Rewriting integration-test/1337-roads-surface-cobblestone-value-transformed.py
Rewriting integration-test/840-normalize-place-kind.py
Rewriting integration-test/546-road-sort-keys-aeroway.py
Rewriting integration-test/1403-fences-walls-landuse.py
Rewriting integration-test/1810-alternate-viewpoints.py
Rewriting integration-test/343-winter-sports-resorts.py
Rewriting integration-test/552-water-boundary-sort-key.py
Rewriting integration-test/546-road-sort-keys-tunnel.py
Rewriting integration-test/895-extract-airport-lines-minor_road.py
Rewriting integration-test/912-missing-building-part.py
Rewriting integration-test/447-ice-cream-shops.py
Rewriting integration-test/605-crosswalk-sidewalk.py
Rewriting integration-test/860-lighthouse.py
Rewriting integration-test/370-prisons.py
Rewriting integration-test/830-windmill-zoom.py
Rewriting integration-test/592-add-adjust-bicycle-pois.py
Rewriting integration-test/369-subway-stations-z12.py
Rewriting integration-test/398-airport-iata-codes.py
Rewriting integration-test/368-disused-railway-stations.py
Rewriting integration-test/713-urban-areas.py
Rewriting integration-test/594-trailhead.py
Rewriting integration-test/628-standardize-water-kinds.py
Rewriting integration-test/776-duplicate-footway.py
Rewriting integration-test/661-historic-transit-stops.py
Rewriting integration-test/488-motorway_link-z11.py
Rewriting integration-test/655-landuse-scree.py
Rewriting integration-test/1020-roads-surface.py
Rewriting integration-test/896-ne-shield-enums.py
Rewriting integration-test/546-road-sort-keys-bridges.py
Rewriting integration-test/857-move_barriers_to_landuse.py
Rewriting integration-test/935-source-in-transit.py
Rewriting integration-test/704-exclude-null-values-for-buildings.py
Rewriting integration-test/829-garden-pois.py
Rewriting integration-test/601-cliff-arete-ridge-valley.py
Rewriting integration-test/421-zoos-z13.py
Rewriting integration-test/197-clip-buildings.py
Rewriting integration-test/522-hotels.py
Rewriting integration-test/418-wof-l10n_name.py
Rewriting integration-test/677-waterfall.py
Rewriting integration-test/1218-poni-whitelist.py
Rewriting integration-test/859-add-bridleway.py
Rewriting integration-test/580-kind-csv.py
Rewriting integration-test/1408-building-material.py
Rewriting integration-test/993-remove-props-road-merge.py
Rewriting integration-test/546-road-sort-keys-railways.py
Rewriting integration-test/546-road-sort-keys-layers.py
Rewriting integration-test/663-combo-outdoor-landuse-pois.py
Rewriting integration-test/1091-missing-name-short.py
Rewriting integration-test/982-remove-unstyled-localities.py
Rewriting integration-test/852-remove-landuse_labels.py
Rewriting integration-test/668-intermittent-water.py
Rewriting integration-test/671-ranger-station.py
Rewriting integration-test/1955-chinese-parser.py
Rewriting integration-test/766-dont-merge-z16-roads.py
Rewriting integration-test/524-peak-kind-tile-rank.py
Rewriting integration-test/382-pier-lines.py
Rewriting integration-test/927-normalize-operator-values.py
Rewriting integration-test/1171-bicycle-yes-designated-roads.py
Rewriting integration-test/448-wine-and-alcohol-shops.py
Rewriting integration-test/679-less-landuse-building-label-placements.py
Rewriting integration-test/896-ne-shield-enums-2.py
Rewriting integration-test/842-normalize-building-kind.py
Rewriting integration-test/834-park-building.py
Rewriting integration-test/719-add-kind-detail-for-pois.py
Rewriting integration-test/367-military-landuse.py
Rewriting integration-test/602-add-boat-rental.py
Rewriting integration-test/1224-earlier-bike-properties.py
Rewriting integration-test/797-add-missing-boundaries.py
Rewriting integration-test/366-beaches.py
Rewriting integration-test/443-swimming-pools.py
Rewriting integration-test/1970-split-ne-boundaries-filter.py
Rewriting integration-test/1251-early-track-roads.py
Rewriting integration-test/454-aeroway-gates.py
Rewriting integration-test/160-motorway-junctions.py
Rewriting integration-test/729-route-name.py
Rewriting integration-test/919-gates-line-geometry.py

flake8...................................................................Failed
- hook id: flake8
- exit code: 1

data/wof_snapshot.py:31:1: E302 expected 2 blank lines, found 1

https://github.com/tilezen/vector-datasource/issues/2004