search

marcell-ferencz-databricks / dash-hackathon-0424

0 stars 0 forks source link

NY_flood/00 Download data - ROADS #3

Closed Miche11e8 closed 7 months ago

Miche11e8 commented 7 months ago

Error

0 --> 'tl_rd22_36001_roads.zip' SSLError: HTTPSConnectionPool(host='www2.census.gov', port=443): Max retries exceeded with url: /geo/tiger/TIGER_RD18/LAYER/ROADS/tl_rd22_36001_roads.zip (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))

from

import pathlib import requests

fuse_path = pathlib.Path(f"{ROOT_PATH}/road/shapefile") fuse_path.mkdir(parents=True, exist_ok=True)

for idx,f in enumerate(ny_files): idx_str = str(idx).rjust(4) fuse_file = fuse_path / f if not fuse_file.exists(): print(f"{idx_str} --> '{f}'") req = requests.get(f'https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/ROADS/{f}') with open(fuse_file, 'wb') as f: f.write(req.content) else: print(f"{idx_str} --> '{f}' exists...skipping")

Needed to use --no-check-certificate with wget, is there an equivalent here

marcell-ferencz-databricks commented 7 months ago

Hey @Miche11e8

Can you try with verify=False flag in the requests.get call please, i.e.:

req = requests.get(f'https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/ROADS/{f}', verify=False)

(docs for reference)