Error 500 when calling us_census()

[juancalvof]

• Commit SHA: commit c4af4d71d781ac9fd204713b41cd79650519497c (HEAD -> master, origin/master, origin/HEAD) Merge: f23824d 3de7d90 Author: Manuel Alvarez Campo manuel@pythiac.com Date: Sat Apr 4 13:48:16 2020 +0200

  Merge pull request #35 from shaikh-raj/master

  Adding metadata for CDS datasource

• Python version: 3.7
• Operating System: Windows
• Trello card: https://trello.com/c/AtA819Yd

  Description

  I got the error 500 when calling us_census()

Error

data_us_census = ds.us_census.us_census() Traceback (most recent call last): File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in data_us_census = ds.us_census.us_census() File "c:\users\juanc\desktop\01 juan calvo\01 big data\01 proyectos\covid-19\code\repositories\task-geo\task_geo\data_sources\demographics\us_census\us_census.py", line 31, in us_census df = us_census_connector() File "c:\users\juanc\desktop\01 juan calvo\01 big data\01 proyectos\covid-19\code\repositories\task-geo\task_geo\data_sources\demographics\us_census\us_census.py", line 47, in us_census_connector urllib.request.urlretrieve(url, "uscensus.zip") File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 247, in urlretrieve with contextlib.closing(urlopen(url, data)) as fp: File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 222, in urlopen return opener.open(url, data, timeout) File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 531, in open response = meth(req, response) File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 569, in error return self._call_chain(args) File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 503, in _call_chain result = func(args) File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 500: Internal Server Error

[ManuelAlvarezC]

Hi @JuanCalvoFerrandiz, and thanks for reporting.

I have been able to replicate it and will start fixing it soon. I will keep you updated.

[DanielRobertNicoud]

The .zip file that the program wants to download is this one: https://data.census.gov/cedsci/table?q=demographic%20data%20by%20county&hidePreview=false&tid=ACSCP5Y2018.CP05&vintage=2018&g=0100000US.050000

[cgomez9]

What is causing the problem is that the dataset URL was obtained getting the link address for the downloaded file in the web browser. This URL has the following form:

https://data.census.gov/api/access/table/download?download_id=TOKEN

The problem is that this TOKEN is changed in each download, making it useless when trying to download the file multiple times. We need to find a URL using the API directly.