search

openclimatedata / national-inventory-submisions

Downloader for UNFCCC National Inventory Submissions
7 stars 3 forks source link

download CRF fails #2

Closed JGuetschow closed 5 years ago

JGuetschow commented 5 years ago

first run of

make download-crf-2018

crashed with

./venv/bin/python scripts/download.py CRF 2018 CRF 2018 Australia CRF 2018 Traceback (most recent call last): File "scripts/download.py", line 49, in with open(local_filename, 'wb') as f: TypeError: invalid file: PosixPath('archive/2018/aus-2018-crf-13apr18.zip') Makefile:5: recipe for target 'download-crf-2018' failed make: *** [download-crf-2018] Error 1

second attempt gives

./venv/bin/python scripts/download.py CRF 2018 CRF 2018 Traceback (most recent call last): File "scripts/download.py", line 34, in shutil.rmtree(download_path) File "/usr/lib/python3.5/shutil.py", line 465, in rmtree onerror(os.lstat, path, sys.exc_info()) File "/usr/lib/python3.5/shutil.py", line 463, in rmtree orig_st = os.lstat(path) TypeError: lstat: illegal type for path parameter Makefile:5: recipe for target 'download-crf-2018' failed make: *** [download-crf-2018] Error 1

I used a newly generated / updated submissions-2018.csv

rgieseke commented 5 years ago

Seems to work for me. Can you trying clearing the archive folder?

JGuetschow commented 5 years ago

It was with an empty archive folder (except for .gitkeep). Running

make download-crf-2018 for the first time created an empty 2018 folder both on downloads and in archive. If I remove them I get the same behavior as described above. Running it for 2017 gives the same errors.

rgieseke commented 5 years ago

Can you try with a fresh clone (if you haven't already) and/or fresh virtual env?

JGuetschow commented 5 years ago

fresh clone and fresh virtualenv give the same result

rgieseke commented 5 years ago

Python version?

JGuetschow commented 5 years ago

2.7.12.1 and 3.5.1

rgieseke commented 5 years ago

Ok, it should only use Python3 ...

I guess this is the problem: https://stackoverflow.com/questions/42694112/when-using-pathlib-getting-error-typeerror-invalid-file-posixpathexample-t

Just pushed a workaround, can you try again?

rgieseke commented 5 years ago

(I'm on 3.6)

JGuetschow commented 5 years ago

Now I get an new error. Maybe I should just download the few changed files by hand and wait for Python 3.6 to come to my linux mint version...

Here's the error message

Download => archive/2018/aus-2018-crf-13apr18.zip Traceback (most recent call last): File "scripts/download.py", line 56, in zipped_file = zipfile.ZipFile(local_filename, 'r') File "/usr/lib/python3.5/zipfile.py", line 1026, in init self._RealGetContents() File "/usr/lib/python3.5/zipfile.py", line 1089, in _RealGetContents endrec = _EndRecData(fp) File "/usr/lib/python3.5/zipfile.py", line 241, in _EndRecData fpin.seek(0, 2) AttributeError: 'PosixPath' object has no attribute 'seek' Makefile:5: recipe for target 'download-crf-2018' failed make: *** [download-crf-2018] Error 1

rgieseke commented 5 years ago

The same "str" workaround should work their, too. I can also upload the data somewhere.

JGuetschow commented 5 years ago

OK, I'll take a look and see if I can fix it

JGuetschow commented 5 years ago

After trowing a few more str(XXX) at the code it runs fine. In case it finished without error, how should I integrate the changes? Branch or direct commit?

rgieseke commented 5 years ago

After trowing a few more str(XXX) at the code it runs fine. In case it finished without error, how should I integrate the changes? Branch or direct commit?

Direct commit should be fine!

JGuetschow commented 5 years ago

I don't have the necessary access rights

rgieseke commented 5 years ago

Added you as a collaborator, probably best to just wrap the path definitions download_path and local_filename with str

JGuetschow commented 5 years ago

If I didn't miss any occurrence then download_path and local_filename are always used with str(). On my machine it works now.

rgieseke commented 5 years ago

Alrighty, thanks!