search

wangzhecheng / DeepSolar

Nationwide houseshold-level solar panel identification with deep learning
MIT License
247 stars 73 forks source link

Libraries mismatch : Python / requirements versions ? #7

Open raphaeljolivet opened 5 years ago

raphaeljolivet commented 5 years ago

Hi,

I am trying to run it locally, but I seem to get some version mismatch of numpy vs skimage. I get this, when testing classification :

File "DeepSolar/.venv3/lib/python3.5/site-packages/skimage/util/arraycrop.py", line 8, in from numpy.lib.arraypad import _validate_lengths ImportError: cannot import name '_validate_lengths'

requirements.txt has floating versions (and also was missing pandas) Could you please provide the versions of each dependency ? Also, which version of python are you using ? 2.7 or 3.5 ? I guess it's 3.5 ?

Thanks in advance for your help.

raphaeljolivet commented 5 years ago

Ok, so finally, I am able to have it work. In summary :

For information here is a frozen requirement.txt :

numpy==1.15.4 tensorflow==1.12.0 scikit-image==0.14.1 matplotlib==2.2.3 pandas==0.23.4

raphaeljolivet commented 5 years ago

Please update requirement.txt and README accordingly.

CoolSciGuy commented 5 years ago

Hi @raphaeljolivet, were you able to download training and eval sets?

raphaeljolivet commented 5 years ago

Yes I did.

-------- Message d'origine -------- On 8 févr. 2019 15:37, CoolSciGuy a écrit :

Hi @raphaeljolivet, were you able to download training and eval sets?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

CoolSciGuy commented 5 years ago

Yes I did. … -------- Message d'origine -------- On 8 févr. 2019 15:37, CoolSciGuy a écrit : Hi @.***(https://github.com/raphaeljolivet), were you able to download training and eval sets? — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#7 (comment)), or mute the thread.

Thanks for responding, did you follow instructions as suggested by developer? I am having no luck.

sassej commented 5 years ago

I am getting this error: $ python train_classification.py --fine_tune=True Training set built. Size: 366467 Traceback (most recent call last): File "train_classification.py", line 262, in train() File "train_classification.py", line 117, in train scope=None) File "/Users/sassej/DeepSolar/inception_model.py", line 84, in inference with slim.arg_scope([slim.ops.conv2d, slim.ops.fc], weight_decay=0.00004): AttributeError: 'module' object has no attribute 'arg_scope'

raphaeljolivet commented 5 years ago

Hi,

Look at my fork here: https://github.com/raphaeljolivet/DeepSolar

I have been able to run the classification script, by integrating directly the inception library and with the dependencies listed in requirements.txt

Please note that I have simplified the usage of test_classification here, it does not require the generate_data_list.py anymore. Please refer to the header of test_classificiation.py for usage.

namgivu commented 3 years ago

hi @raphaeljolivet

I tried your fork and cannot get install -r requirements.txt working I'm using python 2.7.18 virtualenv thru pyenv

namgivu@localhost:~/code/b/deepsonar/zref/raphaeljolivet-DeepSolar
$ ./.venv/bin/python -m pip install -r requirements.txt
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting numpy==1.15.4
  Using cached numpy-1.15.4-cp27-cp27mu-manylinux1_x86_64.whl (13.8 MB)
Collecting tensorflow==1.12.0
  Using cached tensorflow-1.12.0-cp27-cp27mu-manylinux1_x86_64.whl (83.1 MB)
Collecting scikit-image==0.14.1
  Using cached scikit_image-0.14.1-cp27-cp27mu-manylinux1_x86_64.whl (25.4 MB)
Collecting matplotlib==2.2.3
  Using cached matplotlib-2.2.3-cp27-cp27mu-manylinux1_x86_64.whl (12.6 MB)
Collecting pandas==0.23.4
  Using cached pandas-0.23.4-cp27-cp27mu-manylinux1_x86_64.whl (8.9 MB)
Collecting fiona
  Downloading Fiona-1.8.20.tar.gz (1.3 MB)
     |████████████████████████████████| 1.3 MB 1.6 MB/s 
    ERROR: Command errored out with exit status 1:
     command: /home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-okL5YE/fiona/setup.py'"'"'; __file__='"'"'/tmp/pip-install-okL5YE/fiona/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-Codbjh
         cwd: /tmp/pip-install-okL5YE/fiona/
    Complete output (2 lines):
    Failed to get options via gdal-config: [Errno 2] No such file or directory
    A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

removing two lines in the requirements.txt below to get it thru

fiona opencv-python

numpy==1.15.4
tensorflow==1.12.0
scikit-image==0.14.1
matplotlib==2.2.3
pandas==0.23.4
# fiona
# opencv-python
shapely
pyproj
progressbar2==3.39.2
namgivu commented 3 years ago

I have succeeded running the train classification with your fork version - great thanks for the fix for inception

./.venv/bin/python train_classification.py --fine_tune=False

Though I got the error

ValueError: Dimension 3 in both shapes must be equal, but are 96 and 32. Shapes are [32,35,35,96] and [32,35,35,32]. From merging shape 2 with other shapes. for 'inception_v3/mixed_35x35x256a/concat/concat_dim' (op: 'Pack') with input shapes: [32,35,35,64], [32,35,35,64], [32,35,35,96], [32,35,35,32].

Would be higly appreciated as you would help to have a look.

In full as below 

namgivu@localhost:~/code/b/deepsonar/DeepSolar/zref/raphaeljolivet-DeepSolar
$ ll
total 88
drwxrwxr-x 6 namgivu namgivu  4096 Jun  4 12:45 ./
drwxrwxr-x 4 namgivu namgivu  4096 Jun  4 09:56 ../
-rw-rw-r-- 1 namgivu namgivu  8196 Jun  4 09:56 .DS_Store
-rwxrwxr-x 1 namgivu namgivu  2725 Jun  4 09:56 generate_data_list.py*
drwxrwxr-x 8 namgivu namgivu  4096 Jun  4 09:56 .git/
drwxrwxr-x 3 namgivu namgivu  4096 Jun  4 10:36 .idea/
drwxrwxr-x 4 namgivu namgivu  4096 Jun  4 09:56 inception/
-rwxrwxr-x 1 namgivu namgivu     0 Jun  4 09:56 __init__.py*
-rw-rw-r-- 1 namgivu namgivu     7 Jun  4 09:56 .python-version
-rw-rw-r-- 1 namgivu namgivu  2893 Jun  4 09:56 README.md
-rw-rw-r-- 1 namgivu namgivu   147 Jun  4 10:06 requirements.txt
lrwxrwxrwx 1 namgivu namgivu    42 Jun  4 12:45 SPI_eval -> /media/namgivu/Elements/deepsonar/SPI_eval
-rwxrwxr-x 1 namgivu namgivu  3004 Jun  4 09:56 test_classification.py*
-rwxrwxr-x 1 namgivu namgivu 10541 Jun  4 09:56 test_segmentation.py*
-rwxrwxr-x 1 namgivu namgivu 10551 Jun  4 09:56 train_classification.py*
-rwxrwxr-x 1 namgivu namgivu  8087 Jun  4 09:56 train_segmentation.py*
drwxrwxr-x 6 namgivu namgivu  4096 Jun  4 10:07 .venv/

namgivu@localhost:~/code/b/deepsonar/DeepSolar/zref/raphaeljolivet-DeepSolar
$ ll
total 88
drwxrwxr-x 6 namgivu namgivu  4096 Jun  4 12:45 ./
drwxrwxr-x 4 namgivu namgivu  4096 Jun  4 09:56 ../
-rw-rw-r-- 1 namgivu namgivu  8196 Jun  4 09:56 .DS_Store
-rwxrwxr-x 1 namgivu namgivu  2725 Jun  4 09:56 generate_data_list.py*
drwxrwxr-x 8 namgivu namgivu  4096 Jun  4 09:56 .git/
drwxrwxr-x 3 namgivu namgivu  4096 Jun  4 10:36 .idea/
drwxrwxr-x 4 namgivu namgivu  4096 Jun  4 09:56 inception/
-rwxrwxr-x 1 namgivu namgivu     0 Jun  4 09:56 __init__.py*
-rw-rw-r-- 1 namgivu namgivu     7 Jun  4 09:56 .python-version
-rw-rw-r-- 1 namgivu namgivu  2893 Jun  4 09:56 README.md
-rw-rw-r-- 1 namgivu namgivu   147 Jun  4 10:06 requirements.txt
lrwxrwxrwx 1 namgivu namgivu    42 Jun  4 12:45 SPI_eval -> /media/namgivu/Elements/deepsonar/SPI_eval/
-rwxrwxr-x 1 namgivu namgivu  3004 Jun  4 09:56 test_classification.py*
-rwxrwxr-x 1 namgivu namgivu 10541 Jun  4 09:56 test_segmentation.py*
-rwxrwxr-x 1 namgivu namgivu 10551 Jun  4 09:56 train_classification.py*
-rwxrwxr-x 1 namgivu namgivu  8087 Jun  4 09:56 train_segmentation.py*
drwxrwxr-x 6 namgivu namgivu  4096 Jun  4 10:07 .venv/

namgivu@localhost:~/code/b/deepsonar/DeepSolar/zref/raphaeljolivet-DeepSolar
$ ./.venv/bin/python train_classification.py --fine_tune=False
Traceback (most recent call last):
  File "train_classification.py", line 264, in <module>
    train()
  File "train_classification.py", line 83, in train
    raise EnvironmentError('Data list not existed. Please run generate_data_list.py first.')
EnvironmentError: Data list not existed. Please run generate_data_list.py first.

namgivu@localhost:~/code/b/deepsonar/DeepSolar/zref/raphaeljolivet-DeepSolar
$ ./.venv/bin/python generate_data_list.py
Train set list done. # positive samples: 0 # negative samples: 0
Validation set list done. # positive samples: 0 # negative samples: 0
Test set list done. # positive samples: 1221 # negative samples: 92279

namgivu@localhost:~/code/b/deepsonar/DeepSolar/zref/raphaeljolivet-DeepSolar
$ ./.venv/bin/python train_classification.py --fine_tune=False
Training set built. Size: 0
Traceback (most recent call last):
  File "train_classification.py", line 264, in <module>
    train()
  File "train_classification.py", line 119, in train
    scope=None)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/inception/inception_model.py", line 95, in inference
    scope=scope)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/inception/slim/inception_model.py", line 126, in inception_v3
    net = tf.concat(3, [branch1x1, branch5x5, branch3x3dbl, branch_pool])
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1121, in concat
    dtype=dtypes.int32).get_shape().assert_is_compatible_with(
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1050, in convert_to_tensor
    as_ref=False)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1146, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 971, in _autopacking_conversion_function
    return _autopacking_helper(v, dtype, name or "packed")
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 923, in _autopacking_helper
    return gen_array_ops.pack(elems_as_tensors, name=scope)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 4875, in pack
    "Pack", values=values, axis=axis, name=name)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1792, in __init__
    control_input_ops)
  File "/home/namgivu/code/b/deepsonar/zref/raphaeljolivet-DeepSolar/.venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1631, in _create_c_op
    raise ValueError(str(e))
ValueError: Dimension 3 in both shapes must be equal, but are 96 and 32. Shapes are [32,35,35,96] and [32,35,35,32].
        From merging shape 2 with other shapes. for 'inception_v3/mixed_35x35x256a/concat/concat_dim' (op: 'Pack') with input shapes: [32,35,35,64], [32,35,35,64], [32,35,35,96], [32,35,35,32].
namgivu commented 3 years ago

Hi @raphaeljolivet, May you have free time, please have a look at the error here[(https://github.com/wangzhecheng/DeepSolar/issues/7#issuecomment-854381647). Thanks a lot!