search

tjvandal / deepsd

DeepSD Super-resolution for Climate Downscaling in KDD 2017
https://www.kdd.org/kdd2017/papers/view/deepsd-generating-high-resolution-climate-change-projections-through-single
MIT License
88 stars 25 forks source link

ValueError("New shape is shitty") #3

Closed zeeem closed 4 years ago

zeeem commented 4 years ago

Not running in python 3.7 so I used python 2.7 instead. This is the error I am getting. (changed years to 2017 & 2018 in the config.ini)

ERROR:

2018-12-31 00:00:00 ('making temp dir', '/tmp/tmppHxmZ0') Year: 2017 ('data dir', u'data/prism/ppt/raw') ('highres file', u'data/prism/ppt/raw/prism_ppt_4km2_2017.nc') ('Making patches or year:', 2017) scratch/ppt_004_008/train_2017.tfrecords trying to make patches tcmalloc: large alloc 2541805568 bytes == 0x55940c214000 @ 0x7f74314b81e7 0x7f74160b1771 0x7f741611628e 0x7f7416116847 0x7f74161b0810 0x5592ca1816da 0x5592ca17ed3a 0x5592ca18683e 0x5592ca186452 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca17e7b9 0x5592ca1af4df 0x5592ca1aa2d2 0x5592ca1a974d 0x5592ca158de4 0x7f74310b5b97 0x5592ca15869a Traceback (most recent call last): File "prism.py", line 364, in main_prism_tf(config) File "prism.py", line 334, in main_prism_tf d.make_patches(tf_file, size=patch_size, stride=20, scale1=scale1, scale2=scale2) File "prism.py", line 230, in make_patches mask, da1, da2, elev = self.resolve_data(scale1, scale2) File "prism.py", line 213, in resolve_data Y_interp = utils.interp_da(Y, scale1) File "/gdrive/My Drive/Colab Notebooks/deepsd/utils.py", line 53, in interp_da raise ValueError("New shape is shitty") ValueError: New shape is shitty

zeeem commented 4 years ago

daily/ppt/2018/PRISM_ppt_stable_4kmD2_20181231_bil.zip 2018-12-31 00:00:00 ('making temp dir', '/tmp/tmppHxmZ0') Year: 2017 ('data dir', u'data/prism/ppt/raw') ('highres file', u'data/prism/ppt/raw/prism_ppt_4km2_2017.nc') ('Making patches or year:', 2017) scratch/ppt_004_008/train_2017.tfrecords trying to make patches tcmalloc: large alloc 2541805568 bytes == 0x55940c214000 @ 0x7f74314b81e7 0x7f74160b1771 0x7f741611628e 0x7f7416116847 0x7f74161b0810 0x5592ca1816da 0x5592ca17ed3a 0x5592ca18683e 0x5592ca186452 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca17e7b9 0x5592ca1af4df 0x5592ca1aa2d2 0x5592ca1a974d 0x5592ca158de4 0x7f74310b5b97 0x5592ca15869a Traceback (most recent call last): File "prism.py", line 364, in main_prism_tf(config) File "prism.py", line 334, in main_prism_tf d.make_patches(tf_file, size=patch_size, stride=20, scale1=scale1, scale2=scale2) File "prism.py", line 230, in make_patches mask, da1, da2, elev = self.resolve_data(scale1, scale2) File "prism.py", line 213, in resolve_data Y_interp = utils.interp_da(Y, scale1) File "/gdrive/My Drive/Colab Notebooks/deepsd/utils.py", line 53, in interp_da raise ValueError("New shape is shitty") ValueError: New shape is shitty

sanket-p commented 4 years ago

@zeeem thanx for raising this , I am also facing the same issue

zeeem commented 4 years ago

@tjvandal , I hope you can help us.

tjvandal commented 4 years ago

Hi @zeem @sanket-p , I'm looking into it, haven't touched this code in a while but that ValueError I added in there is fairly amusing. Did daily/ppt/2018/PRISM_ppt_stable_4kmD2_20181231_bil.zip download completely? I suspect the binary is too short and cannot be reshaped.

zeeem commented 4 years ago

Hello @tjvandal , Yes it downloaded completely. I tried changing the years to 2016, 2017 and 2018 everytime after downloading and it raises this error.

('Making patches or year:', 2017) scratch/ppt_004_008/train_2017.tfrecords trying to make patches tcmalloc: large alloc 2541805568 bytes == 0x55940c214000 @ 0x7f74314b81e7 0x7f74160b1771 0x7f741611628e 0x7f7416116847 0x7f74161b0810 0x5592ca1816da 0x5592ca17ed3a 0x5592ca18683e 0x5592ca186452 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca17e7b9 0x5592ca1af4df 0x5592ca1aa2d2 0x5592ca1a974d 0x5592ca158de4 0x7f74310b5b97 0x5592ca15869a Traceback (most recent call last):

tjvandal commented 4 years ago

I reproduced that error and committed a fix. https://github.com/tjvandal/deepsd/commit/0d29487064d7513fb521e928b372fa3b1e9b8147

zeeem commented 4 years ago

@tjvandal thanks a lot.. let me try running the code, will let you know if that works. thanks again.

@sanket-p hope you will find the fix useful as well.

sanket-p commented 4 years ago

Thank you for your help @tjvandal In my case I am not sure about the data as I had few errors related to packages as I was switching from python 3 to 2. Though, your recent update will do. I will be running this right away. Also, thank you for assistance @zeeem

sanket-p commented 4 years ago

Yepppp... the codes are working and I got those patches ready.

zeeem commented 4 years ago

@tjvandal , the first step worked fine.

but at the 2nd step when I am trying the "bash run_job.sh", it's showing this error.

Traceback (most recent call last): File "train.py", line 25, in FLAGS._parse_flags() File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/platform/flags.py", line 85, in getattr return wrapped.getattr(name) File "/usr/local/lib/python2.7/dist-packages/absl/flags/_flagvalues.py", line 473, in getattr raise AttributeError(name) AttributeError: _parse_flags Traceback (most recent call last): File "train.py", line 25, in FLAGS._parse_flags() File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/platform/flags.py", line 85, in getattr return wrapped.getattr(name) File "/usr/local/lib/python2.7/dist-packages/absl/flags/_flagvalues.py", line 473, in getattr raise AttributeError(name) AttributeError: _parse_flags

Although I am still trying to figure the solution to this, I would highly appreciate it if you could look into the issue.

**Besides, Upon going through all the codes, I understood that we will be getting the res.pdf as the final result after running the inference.py. So I was wondering how can I get the result like below example: image

Also, I was wondering how you prepared the elevation for this project. really appreciate your assistance.** Thanks

tjvandal commented 4 years ago

The codes available will not generate that figure. Use Tensorflow version 1.5 for FLAGS to parse. That object is deprecated in recent versions.

zeeem commented 4 years ago

@tjvandal , sorry to bother again. But I tried tensorflow 1.5, 1.15 and also 1.14 but i am getting the same error:

Traceback (most recent call last): File "train.py", line 25, in FLAGS._parse_flags() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/flags.py", line 85, in getattr return wrapped.getattr(name) File "/usr/local/lib/python2.7/dist-packages/absl/flags/_flagvalues.py", line 473, in getattr raise AttributeError(name) AttributeError: _parse_flags Traceback (most recent call last): File "train.py", line 25, in FLAGS._parse_flags() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/flags.py", line 85, in getattr return wrapped.getattr(name) File "/usr/local/lib/python2.7/dist-packages/absl/flags/_flagvalues.py", line 473, in getattr raise AttributeError(name) AttributeError: _parse_flags

zeeem commented 4 years ago

also is there any other git repo where I can produce the image results like the paper? thanks in advance.