search

mikeyEcology / MLWIC

Machine Learning for Wildlife Image Classification
69 stars 16 forks source link

model_predictions.txt does not exist #6

Closed thom1253 closed 5 years ago

thom1253 commented 5 years ago

Hi! After some struggling with tensorflow etc, I think everything is working correctly (I'm running this on a windows machine). When I run classify(), I get this message, which looks like it ran correctly.... but when I look in the L1 folder "model_predictions.txt" does not exist (I've hunted around and triple checked that my working directory is what I think it is...). Also, the ~ 2 second run time seems really fast, so I wonder if it is just not working quite right? Any thoughts you have would be super helpful! Thank you, Sarah

Message: C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\h5py__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Namespace(architecture='resnet', batch_size=128, crop_size=[224, 224], data_info='data_info.csv', delimiter=',', depth=18, load_size=[256, 256], log_dir='USDA182', num_batches=16, num_channels=3, num_classes=28, num_samples=1934, num_threads=20, path_prefix='C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/images', save_predictions='model_predictions.txt', top_n=5) Traceback (most recent call last): File "eval.py", line 129, in main() File "eval.py", line 125, in main evaluate(args) File "eval.py", line 25, in evaluate images, labels, urls = data_loader.read_inputs(False, args) File "C:\Users\sthompson\Desktop\Sarah_Image_Processing\Sarah_Image_Processing\L1\data_loader.py", line 23, in read_inputs filepaths, labels = _read_label_file(args.data_info, args.delimiter) File "C:\Users\sthompson\Desktop\Sarah_Image_Processing\Sarah_Image_Processing\L1\data_loader.py", line 19, in _read_label_file labels.append(int(tokens[1])) ValueError: invalid literal for int() with base 10: 'pic_id' [1] "evaluation of images took 2.04373002052307 secs. The results are stored in C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/L1/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

mikeyEcology commented 5 years ago

Hi, Classify did not work on your machine and it looks like it is because setup did not work right either. Can you please post the functions as you specified them when you tried to run. Thanks, Mikey

thom1253 commented 5 years ago

Hi Mikey, I was hopeful that it was closer to working, as it previously gave me a message that it took 0.0 seconds, so this seemed like progress! The only error that I get with this is after running setup (Error: Error 127 occurred creating conda environment r-reticulate), but the documentation indicated that may be ok:

library(MLWIC) library(tensorflow) library(keras) library(cloudml) library(Rcpp)

devtools::install_github("rstudio/reticulate") library(reticulate)

'# test tensorflow, this seems to be working use_condaenv("r-tensorflow") sess <- tf$Session() hello <- tf$constant('Hello, TensorFlow!') sess$run(hello)

MLWIC::setup(python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/python", conda_loc = "C:/Users/sthompson/AppData/Local/conda")

MLWIC::classify( path_prefix = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/images",

data_info = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/image_labels.csv",

model_dir = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing",

python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/" ,

save_predictions = "model_predictions.txt" )

mikeyEcology commented 5 years ago

Can you try posting the error message you receive when you run setup? The conda_loc looks funny. Also, maybe try setting conda_loc="auto"

thom1253 commented 5 years ago

Hi! Sorry about the slow reply, I was out of town for a few days. Thanks for your help on this :) I ran your conda_loc suggestion, then ran setup again Still get the same error: Error: Error 127 occurred creating conda environment r-reticulate And get the same stuff when I then run classify.

Let me know if I can do anything else to help troubleshoot! Thanks, Sarah

mikeyEcology commented 5 years ago

Can you please provide the input and the output that you have when you run setup?

thom1253 commented 5 years ago

SETUP CODE MLWIC::setup(python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/python", conda_loc = "C:/Users/sthompson/AppData/Local/conda")

ERROR OUTPUT Error: Error 127 occurred creating conda environment r-reticulate

thom1253 commented 5 years ago

oh hey, I realized I'm re-telling it where conda is.. :] -- rerunning now

thom1253 commented 5 years ago

MLWIC::setup(python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/python") Solving environment: ...working... done

'## Package Plan ##

environment location: C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\envs\r-reticulate

added / updated specs:

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
pip-10.0.1                 |           py37_0         1.7 MB
wincertstore-0.2           |           py37_0          13 KB
wheel-0.32.2               |           py37_0          52 KB
setuptools-40.4.3          |           py37_0         576 KB
certifi-2018.10.15         |           py37_0         138 KB
python-3.7.1               |       h33f27b4_3        19.5 MB
------------------------------------------------------------
                                       Total:        22.0 MB

The following NEW packages will be INSTALLED:

certifi:        2018.10.15-py37_0     
pip:            10.0.1-py37_0         
python:         3.7.1-h33f27b4_3      
setuptools:     40.4.3-py37_0         
vc:             14.1-h0510ff6_4       
vs2015_runtime: 14.15.26706-h3a45250_0
wheel:          0.32.2-py37_0         
wincertstore:   0.2-py37_0

Downloading and Extracting Packages pip-10.0.1 | 1.7 MB | ########## | 100% wincertstore-0.2 | 13 KB | ########## | 100% wheel-0.32.2 | 52 KB | ########## | 100% setuptools-40.4.3 | 576 KB | ########## | 100% certifi-2018.10.15 | 138 KB | ########## | 100% python-3.7.1 | 19.5 MB | ########## | 100% Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done '# '# To activate this environment, use: '# > activate r-reticulate '# '# To deactivate an active environment, use: '# > deactivate '# '# * for power-users using bash, you must source '#

Solving environment: ...working... failed

UnsatisfiableError: The following specifications were found to be in conflict:

Error: Error 1 occurred installing packages into conda environment r-reticulate

mikeyEcology commented 5 years ago

Thank you. It looks like setup is working properly. Can you please paste your classify input and output.

thom1253 commented 5 years ago

MLWIC::classify( path_prefix = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/images",

data_info = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/image_labels.csv",

model_dir = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing",

python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/" ,

save_predictions = "model_predictions.txt" )

C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\h5py__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Namespace(architecture='resnet', batch_size=128, crop_size=[224, 224], data_info='data_info.csv', delimiter=',', depth=18, load_size=[256, 256], log_dir='USDA182', num_batches=16, num_channels=3, num_classes=28, num_samples=1934, num_threads=20, path_prefix='C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/images', save_predictions='model_predictions.txt', top_n=5) Traceback (most recent call last): File "eval.py", line 129, in main() File "eval.py", line 125, in main evaluate(args) File "eval.py", line 25, in evaluate images, labels, urls = data_loader.read_inputs(False, args) File "C:\Users\sthompson\Desktop\Sarah_Image_Processing\Sarah_Image_Processing\L1\data_loader.py", line 23, in read_inputs filepaths, labels = _read_label_file(args.data_info, args.delimiter) File "C:\Users\sthompson\Desktop\Sarah_Image_Processing\Sarah_Image_Processing\L1\data_loader.py", line 19, in _read_label_file labels.append(int(tokens[1])) ValueError: invalid literal for int() with base 10: 'pic_id' [1] "evaluation of images took 2.04849195480347 secs. The results are stored in C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/L1/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

thom1253 commented 5 years ago

if I leave out the "python_loc = ", I get this:

[1] "evaluation of images took 0 secs. The results are stored in C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/L1/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"**

mikeyEcology commented 5 years ago

It looks like the function isn't finding the images on your computer. Are you using the example from MLWIC_examples. If so, you should try downloading the example files again and make sure that you don't re-save the image_labels csv. Then make sure that your images really are in the location of path_prefix and that the csv really is in data_info. Otherwise, I'm not sure what is going wrong.

thom1253 commented 5 years ago

Yep, I am using the MLWIC example. I started totally over, in a different directory/folder, re-downloading the MLWIC stuff. Code is exactly the same (well, other than re-doing file paths). However, I got a different (more promising?) message with classify. It took longer to run (56 seconds), and did output model_predictions.txt as promised (super exciting :), but the .txt is blank.

Messages after classify() :

C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\h5py__init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters WARNING:tensorflow:From C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\tensorflow\python\training\input.py:187: QueueRunner.init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. WARNING:tensorflow:From C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\tensorflow\python\training\input.py:187: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. 2018-10-24 14:22:31.380926: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2018-10-24 14:22:31.541008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties: name: Quadro P3200 with Max-Q Design major: 6 minor: 1 memoryClockRate(GHz): 1.404 pciBusID: 0000:01:00.0 totalMemory: 6.00GiB freeMemory: 4.94GiB 2018-10-24 14:22:31.541333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0 2018-10-24 14:22:32.279461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-10-24 14:22:32.279638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] 0 2018-10-24 14:22:32.279738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0: N 2018-10-24 14:22:32.279966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4697 MB memory) -> physical GPU (device: 0, name: Quadro P3200 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 6.1) WARNING:tensorflow:From eval.py:65: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. 2018-10-24 14:22:47.162986: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-10-24 14:22:50.690340: E tensorflow/stream_executor/cuda/cuda_dnn.cc:353] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED [1] "evaluation of images took 56.1054491996765 secs. The results are stored in S:/Wildlife/Thompson/Sarah_Image_Processing/L1/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

mikeyEcology commented 5 years ago

It looks like something funny is going on with your computer drives. Do you have the L1 folder here on your S drive: S:/Wildlife/Thompson/Sarah_Image_Processing/L1? You're specifying everything as being on your C drive.

thom1253 commented 5 years ago

image

thom1253 commented 5 years ago

MLWIC::setup(python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/python", conda_loc = "auto")

MLWIC::classify( path_prefix = "S:/Wildlife/Thompson/Sarah_Image_Processing/MLWIC_examples-master/images",

data_info = "S:/Wildlife/Thompson/Sarah_Image_Processing/MLWIC_examples-master/image_labels.csv",

model_dir = "S:/Wildlife/Thompson/Sarah_Image_Processing", # assuming this is where you stored the L1 folder in Step 3 of the instructions: github.com/mikeyEcology/MLWIC/blob/master/README

python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/" , save_predictions = "model_predictions.txt" )

thom1253 commented 5 years ago

I moved all of the code from my c drive to a different drive... I do work for the state gov, so maybe something is happening with that. I don't usually have a lot of issues with it though. But everything appears to be in the correct locations and correctly specified in the paths.

thom1253 commented 5 years ago

I honestly have no idea what changed, but this seems to be working now! Super interesting... hahaha. I get output that looks as expected. Thanks for your help... :)

mikeyEcology commented 5 years ago

Ok. It sounds like this might have been a file path issue as you went between your C and S drives. I'm glad you got it to work.

tundraboyce commented 5 years ago

Hi I am having the same issue as originally outline in this thread. Everything seems to be working though classify takes no time at all and there is no model_predictions.txt file created. I have checked the paths and folder locations a bunch of times and can't figure out why its not finding the images?

classify( path_prefix ="C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/MLWIC_examples-master/images", data_info = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/MLWIC_examples-master/image_labels.csv", model_dir = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/", # assuming this is where you stored the L1 folder in Step 3 of the instructions: github.com/mikeyEcology/MLWIC/blob/master/README python_loc = "C:/Users/Paul/Anaconda3", = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/model_predictions.txt" )

[1] "evaluation of images took 0.000499963760375977 secs. The results are stored in C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example//L1/C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

Is there something I'm missing with the //L1/C location where the output says the code is?

Thanks in advance, Paul

thom1253 commented 5 years ago

I did manage to get this working, however I ended up having to move it all around to different folders... for some reason, it acted like the location was unknown when I used my desktop (which seemed like the simplest location), and then it worked when I put it all on a different network drive & that simple step got it working on the sample images. I also could not direct classify() to look for pictures in a folder elsewhere (which you don't appear to be doing; but the folder had to be in the same place as the L1 folder -- so when I switched to try running it on my own pictures, that took more picture-moving than I had been hoping). I work for a state gov, so I just sort of assumed it was something weird with our network.

On Thu, Jan 3, 2019 at 1:26 PM tundraboyce notifications@github.com wrote:

Hi I am having the same issue as originally outline in this thread. Everything seems to be working though classify takes no time at all and there is no model_predictions.txt file created. I have checked the paths and folder locations a bunch of times and can't figure out why its not finding the images?

classify( path_prefix ="C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/MLWIC_examples-master/images", data_info = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/MLWIC_examples-master/image_labels.csv", model_dir = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/", # assuming this is where you stored the L1 folder in Step 3 of the instructions: github.com/mikeyEcology/MLWIC/blob/master/README python_loc = "C:/Users/Paul/Anaconda3", = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/model_predictions.txt" )

[1] "evaluation of images took 0.000499963760375977 secs. The results are stored in C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example//L1/C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

Is there something I'm missing with the //L1/C location where the output says the code is?

Thanks in advance, Paul

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/mikeyEcology/MLWIC/issues/6#issuecomment-451266619, or mute the thread https://github.com/notifications/unsubscribe-auth/AGDIfrnwUIfyqaZ9d1mpBsVo9AAQduZsks5u_mdmgaJpZM4Xelml .

tundraboyce commented 5 years ago

Okay great I will try and move things around I guess and put it on the desktop. I followed your outline above too and checked that everything was working/installed properly but still a sub zero second classification situation and no file. Hopefully!

Thanks

tundraboyce commented 5 years ago

No luck!

I tried again through setup: got the packages in conflict error you had which I understand is okay?

setup(python_loc = "C:/Users/Paul/Anaconda3/", conda_loc = "auto",

  • r_reticulate = FALSE)

Remove all packages in environment C:\Users\Paul\ANACON~1\envs\r-reticulate:

Package Plan

environment location: C:\Users\Paul\ANACON~1\envs\r-reticulate

The following packages will be REMOVED:

ca-certificates: 2018.03.07-0          
certifi:         2018.11.29-py37_0     
openssl:         1.1.1a-he774522_0     
pip:             18.1-py37_0           
python:          3.7.2-h8c8aaf0_0      
setuptools:      40.6.3-py37_0         
sqlite:          3.26.0-he774522_0     
vc:              14.1-h0510ff6_4       
vs2015_runtime:  14.15.26706-h3a45250_0
wheel:           0.32.3-py37_0         
wincertstore:    0.2-py37_0

Solving environment: ...working... done

Package Plan

environment location: C:\Users\Paul\ANACON~1\envs\r-reticulate

added / updated specs:

The following NEW packages will be INSTALLED:

ca-certificates: 2018.03.07-0          
certifi:         2018.11.29-py37_0     
openssl:         1.1.1a-he774522_0     
pip:             18.1-py37_0           
python:          3.7.2-h8c8aaf0_0      
setuptools:      40.6.3-py37_0         
sqlite:          3.26.0-he774522_0     
vc:              14.1-h0510ff6_4       
vs2015_runtime:  14.15.26706-h3a45250_0
wheel:           0.32.3-py37_0         
wincertstore:    0.2-py37_0

Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done #

To activate this environment, use:

> activate r-reticulate

#

To deactivate an active environment, use:

> deactivate

#

* for power-users using bash, you must source

#

Solving environment: ...working... failed

UnsatisfiableError: The following specifications were found to be in conflict:

Error: Error 1 occurred installing packages into conda environment r-reticulate

Then tried Classify again with the same result in a new drive. Stumped!

classify(path_prefix ="E:/Example/MLWIC_examples-master/images", # this is the absolute path to the images.

  • data_info = "E:/Example/MLWIC_examples-master/image_labels.csv", # this is the location of the csv containing image information. It has Unix linebreaks and no headers.
  • model_dir = "E:/Example/", # assuming this is where you stored the L1 folder in Step 3 of the instructions: github.com/mikeyEcology/MLWIC/blob/master/README
  • python_loc = "C:/Users/Paul/Anaconda3", # the location of Python on your computer.
  • save_predictions = "E:/Example/model_predictions.txt" # this is the default and you should use it unless you have reason otherwise.
  • )

[1] "evaluation of images took 0.000501155853271484 secs. The results are stored in

E:/Example//L1/E:/Example/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

tundraboyce commented 5 years ago

Have tried a bunch of different drives and re-downloaded etc a few times. If anyone can think of why it wouldn't be finding the images let me know. e.g., if python, or tensorflow wasn't working (though setup seems to be okay, and I've checked those packages in R).

Thanks in advance!

fabstp commented 5 years ago

It appears that we have exactly the same problem tundraboyce. I tried everything on this thread and it didn't work. I also understand that the setup conflict between argparse and tenserflow is expected. However, theres something different between youroutput and mne compared to OP one. We both don't have this part : Downloading and Extracting Packages pip-10.0.1 | 1.7 MB | ########## | 100% wincertstore-0.2 | 13 KB | ########## | 100% wheel-0.32.2 | 52 KB | ########## | 100% setuptools-40.4.3 | 576 KB | ########## | 100% certifi-2018.10.15 | 138 KB | ########## | 100% python-3.7.1 | 19.5 MB | ########## | 100%

Maybe it is the problem.. I'll try a new thread to reach MikeyEcology

Nova-Scotia commented 5 years ago

@tundraboyce try using a folder that doesn't have special characters (a hyphen).