model_predictions.txt does not exist

[thom1253]

Hi! After some struggling with tensorflow etc, I think everything is working correctly (I'm running this on a windows machine). When I run classify(), I get this message, which looks like it ran correctly.... but when I look in the L1 folder "model_predictions.txt" does not exist (I've hunted around and triple checked that my working directory is what I think it is...). Also, the ~ 2 second run time seems really fast, so I wonder if it is just not working quite right? Any thoughts you have would be super helpful! Thank you, Sarah

Message: C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\h5py__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Namespace(architecture='resnet', batch_size=128, crop_size=[224, 224], data_info='data_info.csv', delimiter=',', depth=18, load_size=[256, 256], log_dir='USDA182', num_batches=16, num_channels=3, num_classes=28, num_samples=1934, num_threads=20, path_prefix='C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/images', save_predictions='model_predictions.txt', top_n=5) Traceback (most recent call last): File "eval.py", line 129, in main() File "eval.py", line 125, in main evaluate(args) File "eval.py", line 25, in evaluate images, labels, urls = data_loader.read_inputs(False, args) File "C:\Users\sthompson\Desktop\Sarah_Image_Processing\Sarah_Image_Processing\L1\data_loader.py", line 23, in read_inputs filepaths, labels = _read_label_file(args.data_info, args.delimiter) File "C:\Users\sthompson\Desktop\Sarah_Image_Processing\Sarah_Image_Processing\L1\data_loader.py", line 19, in _read_label_file labels.append(int(tokens[1])) ValueError: invalid literal for int() with base 10: 'pic_id' [1] "evaluation of images took 2.04373002052307 secs. The results are stored in C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/L1/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

[mikeyEcology]

Hi, Classify did not work on your machine and it looks like it is because setup did not work right either. Can you please post the functions as you specified them when you tried to run. Thanks, Mikey

[thom1253]

Hi Mikey, I was hopeful that it was closer to working, as it previously gave me a message that it took 0.0 seconds, so this seemed like progress! The only error that I get with this is after running setup (Error: Error 127 occurred creating conda environment r-reticulate), but the documentation indicated that may be ok:

library(MLWIC) library(tensorflow) library(keras) library(cloudml) library(Rcpp)

devtools::install_github("rstudio/reticulate") library(reticulate)

'# test tensorflow, this seems to be working use_condaenv("r-tensorflow") sess <- tf$Session() hello <- tf$constant('Hello, TensorFlow!') sess$run(hello)

MLWIC::setup(python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/python", conda_loc = "C:/Users/sthompson/AppData/Local/conda")

MLWIC::classify( path_prefix = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/images",

data_info = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/image_labels.csv",

model_dir = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing",

python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/" ,

save_predictions = "model_predictions.txt" )

[mikeyEcology]

Can you try posting the error message you receive when you run setup? The conda_loc looks funny. Also, maybe try setting conda_loc="auto"

[thom1253]

Hi! Sorry about the slow reply, I was out of town for a few days. Thanks for your help on this :) I ran your conda_loc suggestion, then ran setup again Still get the same error: Error: Error 127 occurred creating conda environment r-reticulate And get the same stuff when I then run classify.

Let me know if I can do anything else to help troubleshoot! Thanks, Sarah

[mikeyEcology]

Can you please provide the input and the output that you have when you run setup?

[thom1253]

SETUP CODE MLWIC::setup(python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/python", conda_loc = "C:/Users/sthompson/AppData/Local/conda")

ERROR OUTPUT Error: Error 127 occurred creating conda environment r-reticulate

[thom1253]

oh hey, I realized I'm re-telling it where conda is.. :] -- rerunning now

[thom1253]

MLWIC::setup(python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/python") Solving environment: ...working... done

'## Package Plan ##

environment location: C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\envs\r-reticulate

added / updated specs:

• python

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
pip-10.0.1                 |           py37_0         1.7 MB
wincertstore-0.2           |           py37_0          13 KB
wheel-0.32.2               |           py37_0          52 KB
setuptools-40.4.3          |           py37_0         576 KB
certifi-2018.10.15         |           py37_0         138 KB
python-3.7.1               |       h33f27b4_3        19.5 MB
------------------------------------------------------------
                                       Total:        22.0 MB

The following NEW packages will be INSTALLED:

certifi:        2018.10.15-py37_0     
pip:            10.0.1-py37_0         
python:         3.7.1-h33f27b4_3      
setuptools:     40.4.3-py37_0         
vc:             14.1-h0510ff6_4       
vs2015_runtime: 14.15.26706-h3a45250_0
wheel:          0.32.2-py37_0         
wincertstore:   0.2-py37_0            

Downloading and Extracting Packages pip-10.0.1 | 1.7 MB | ########## | 100% wincertstore-0.2 | 13 KB | ########## | 100% wheel-0.32.2 | 52 KB | ########## | 100% setuptools-40.4.3 | 576 KB | ########## | 100% certifi-2018.10.15 | 138 KB | ########## | 100% python-3.7.1 | 19.5 MB | ########## | 100% Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done '# '# To activate this environment, use: '# > activate r-reticulate '# '# To deactivate an active environment, use: '# > deactivate '# '# * for power-users using bash, you must source '#

Solving environment: ...working... failed

UnsatisfiableError: The following specifications were found to be in conflict:

• argparse
• tensorflow Use "conda info " to see the dependencies for each package.

Error: Error 1 occurred installing packages into conda environment r-reticulate

[mikeyEcology]

Thank you. It looks like setup is working properly. Can you please paste your classify input and output.

[thom1253]

MLWIC::classify( path_prefix = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/images",

data_info = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/image_labels.csv",

model_dir = "C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing",

python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/" ,

save_predictions = "model_predictions.txt" )

C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\h5py__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Namespace(architecture='resnet', batch_size=128, crop_size=[224, 224], data_info='data_info.csv', delimiter=',', depth=18, load_size=[256, 256], log_dir='USDA182', num_batches=16, num_channels=3, num_classes=28, num_samples=1934, num_threads=20, path_prefix='C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/images', save_predictions='model_predictions.txt', top_n=5) Traceback (most recent call last): File "eval.py", line 129, in main() File "eval.py", line 125, in main evaluate(args) File "eval.py", line 25, in evaluate images, labels, urls = data_loader.read_inputs(False, args) File "C:\Users\sthompson\Desktop\Sarah_Image_Processing\Sarah_Image_Processing\L1\data_loader.py", line 23, in read_inputs filepaths, labels = _read_label_file(args.data_info, args.delimiter) File "C:\Users\sthompson\Desktop\Sarah_Image_Processing\Sarah_Image_Processing\L1\data_loader.py", line 19, in _read_label_file labels.append(int(tokens[1])) ValueError: invalid literal for int() with base 10: 'pic_id' [1] "evaluation of images took 2.04849195480347 secs. The results are stored in C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/L1/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

[thom1253]

if I leave out the "python_loc = ", I get this:

[1] "evaluation of images took 0 secs. The results are stored in C:/Users/sthompson/Desktop/Sarah_Image_Processing/Sarah_Image_Processing/L1/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"**

[mikeyEcology]

It looks like the function isn't finding the images on your computer. Are you using the example from MLWIC_examples. If so, you should try downloading the example files again and make sure that you don't re-save the image_labels csv. Then make sure that your images really are in the location of path_prefix and that the csv really is in data_info. Otherwise, I'm not sure what is going wrong.

[thom1253]

Yep, I am using the MLWIC example. I started totally over, in a different directory/folder, re-downloading the MLWIC stuff. Code is exactly the same (well, other than re-doing file paths). However, I got a different (more promising?) message with classify. It took longer to run (56 seconds), and did output model_predictions.txt as promised (super exciting :), but the .txt is blank.

Messages after classify() :

C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\h5py__init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters WARNING:tensorflow:From C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\tensorflow\python\training\input.py:187: QueueRunner.init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. WARNING:tensorflow:From C:\Users\STHOMP~1\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\tensorflow\python\training\input.py:187: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. 2018-10-24 14:22:31.380926: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2018-10-24 14:22:31.541008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties: name: Quadro P3200 with Max-Q Design major: 6 minor: 1 memoryClockRate(GHz): 1.404 pciBusID: 0000:01:00.0 totalMemory: 6.00GiB freeMemory: 4.94GiB 2018-10-24 14:22:31.541333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0 2018-10-24 14:22:32.279461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-10-24 14:22:32.279638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] 0 2018-10-24 14:22:32.279738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0: N 2018-10-24 14:22:32.279966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4697 MB memory) -> physical GPU (device: 0, name: Quadro P3200 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 6.1) WARNING:tensorflow:From eval.py:65: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. 2018-10-24 14:22:47.162986: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-10-24 14:22:50.690340: E tensorflow/stream_executor/cuda/cuda_dnn.cc:353] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED [1] "evaluation of images took 56.1054491996765 secs. The results are stored in S:/Wildlife/Thompson/Sarah_Image_Processing/L1/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

[mikeyEcology]

It looks like something funny is going on with your computer drives. Do you have the L1 folder here on your S drive: S:/Wildlife/Thompson/Sarah_Image_Processing/L1? You're specifying everything as being on your C drive.

[thom1253]

[thom1253]

MLWIC::setup(python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/python", conda_loc = "auto")

MLWIC::classify( path_prefix = "S:/Wildlife/Thompson/Sarah_Image_Processing/MLWIC_examples-master/images",

data_info = "S:/Wildlife/Thompson/Sarah_Image_Processing/MLWIC_examples-master/image_labels.csv",

model_dir = "S:/Wildlife/Thompson/Sarah_Image_Processing", # assuming this is where you stored the L1 folder in Step 3 of the instructions: github.com/mikeyEcology/MLWIC/blob/master/README

python_loc = "C:/Users/sthompson/AppData/Local/Continuum/anaconda3/" , save_predictions = "model_predictions.txt" )

[thom1253]

I moved all of the code from my c drive to a different drive... I do work for the state gov, so maybe something is happening with that. I don't usually have a lot of issues with it though. But everything appears to be in the correct locations and correctly specified in the paths.

[thom1253]

I honestly have no idea what changed, but this seems to be working now! Super interesting... hahaha. I get output that looks as expected. Thanks for your help... :)

[mikeyEcology]

Ok. It sounds like this might have been a file path issue as you went between your C and S drives. I'm glad you got it to work.

[tundraboyce]

Hi I am having the same issue as originally outline in this thread. Everything seems to be working though classify takes no time at all and there is no model_predictions.txt file created. I have checked the paths and folder locations a bunch of times and can't figure out why its not finding the images?

classify( path_prefix ="C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/MLWIC_examples-master/images", data_info = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/MLWIC_examples-master/image_labels.csv", model_dir = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/", # assuming this is where you stored the L1 folder in Step 3 of the instructions: github.com/mikeyEcology/MLWIC/blob/master/README python_loc = "C:/Users/Paul/Anaconda3", = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/model_predictions.txt" )

[1] "evaluation of images took 0.000499963760375977 secs. The results are stored in C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example//L1/C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

Is there something I'm missing with the //L1/C location where the output says the code is?

Thanks in advance, Paul

[thom1253]

I did manage to get this working, however I ended up having to move it all around to different folders... for some reason, it acted like the location was unknown when I used my desktop (which seemed like the simplest location), and then it worked when I put it all on a different network drive & that simple step got it working on the sample images. I also could not direct classify() to look for pictures in a folder elsewhere (which you don't appear to be doing; but the folder had to be in the same place as the L1 folder -- so when I switched to try running it on my own pictures, that took more picture-moving than I had been hoping). I work for a state gov, so I just sort of assumed it was something weird with our network.

On Thu, Jan 3, 2019 at 1:26 PM tundraboyce notifications@github.com wrote:

Hi I am having the same issue as originally outline in this thread. Everything seems to be working though classify takes no time at all and there is no model_predictions.txt file created. I have checked the paths and folder locations a bunch of times and can't figure out why its not finding the images?

classify( path_prefix ="C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/MLWIC_examples-master/images", data_info = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/MLWIC_examples-master/image_labels.csv", model_dir = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/", # assuming this is where you stored the L1 folder in Step 3 of the instructions: github.com/mikeyEcology/MLWIC/blob/master/README python_loc = "C:/Users/Paul/Anaconda3", = "C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/model_predictions.txt" )

[1] "evaluation of images took 0.000499963760375977 secs. The results are stored in C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example//L1/C:/Users/Paul/Desktop/PhD/Horses/Data/Photo_processing/Example/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

Is there something I'm missing with the //L1/C location where the output says the code is?

Thanks in advance, Paul

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/mikeyEcology/MLWIC/issues/6#issuecomment-451266619, or mute the thread https://github.com/notifications/unsubscribe-auth/AGDIfrnwUIfyqaZ9d1mpBsVo9AAQduZsks5u_mdmgaJpZM4Xelml .

[tundraboyce]

Okay great I will try and move things around I guess and put it on the desktop. I followed your outline above too and checked that everything was working/installed properly but still a sub zero second classification situation and no file. Hopefully!

Thanks

[tundraboyce]

No luck!

I tried again through setup: got the packages in conflict error you had which I understand is okay?

setup(python_loc = "C:/Users/Paul/Anaconda3/", conda_loc = "auto",

• r_reticulate = FALSE)

Remove all packages in environment C:\Users\Paul\ANACON~1\envs\r-reticulate:

Package Plan

environment location: C:\Users\Paul\ANACON~1\envs\r-reticulate

The following packages will be REMOVED:

ca-certificates: 2018.03.07-0          
certifi:         2018.11.29-py37_0     
openssl:         1.1.1a-he774522_0     
pip:             18.1-py37_0           
python:          3.7.2-h8c8aaf0_0      
setuptools:      40.6.3-py37_0         
sqlite:          3.26.0-he774522_0     
vc:              14.1-h0510ff6_4       
vs2015_runtime:  14.15.26706-h3a45250_0
wheel:           0.32.3-py37_0         
wincertstore:    0.2-py37_0            

Solving environment: ...working... done

Package Plan

environment location: C:\Users\Paul\ANACON~1\envs\r-reticulate

added / updated specs:

• python

The following NEW packages will be INSTALLED:

ca-certificates: 2018.03.07-0          
certifi:         2018.11.29-py37_0     
openssl:         1.1.1a-he774522_0     
pip:             18.1-py37_0           
python:          3.7.2-h8c8aaf0_0      
setuptools:      40.6.3-py37_0         
sqlite:          3.26.0-he774522_0     
vc:              14.1-h0510ff6_4       
vs2015_runtime:  14.15.26706-h3a45250_0
wheel:           0.32.3-py37_0         
wincertstore:    0.2-py37_0            

Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done #

To activate this environment, use:

> activate r-reticulate

#

To deactivate an active environment, use:

> deactivate

#

* for power-users using bash, you must source

#

Solving environment: ...working... failed

UnsatisfiableError: The following specifications were found to be in conflict:

• argparse
• tensorflow Use "conda info " to see the dependencies for each package.

Error: Error 1 occurred installing packages into conda environment r-reticulate

Then tried Classify again with the same result in a new drive. Stumped!

classify(path_prefix ="E:/Example/MLWIC_examples-master/images", # this is the absolute path to the images.

• data_info = "E:/Example/MLWIC_examples-master/image_labels.csv", # this is the location of the csv containing image information. It has Unix linebreaks and no headers.
• model_dir = "E:/Example/", # assuming this is where you stored the L1 folder in Step 3 of the instructions: github.com/mikeyEcology/MLWIC/blob/master/README
• python_loc = "C:/Users/Paul/Anaconda3", # the location of Python on your computer.
• save_predictions = "E:/Example/model_predictions.txt" # this is the default and you should use it unless you have reason otherwise.
• )

[1] "evaluation of images took 0.000501155853271484 secs. The results are stored in

E:/Example//L1/E:/Example/model_predictions.txt. To view the results in a viewer-friendly format, please use the function make_output"

[tundraboyce]

Have tried a bunch of different drives and re-downloaded etc a few times. If anyone can think of why it wouldn't be finding the images let me know. e.g., if python, or tensorflow wasn't working (though setup seems to be okay, and I've checked those packages in R).

Thanks in advance!

[fabstp]

It appears that we have exactly the same problem tundraboyce. I tried everything on this thread and it didn't work. I also understand that the setup conflict between argparse and tenserflow is expected. However, theres something different between youroutput and mne compared to OP one. We both don't have this part : Downloading and Extracting Packages pip-10.0.1 | 1.7 MB | ########## | 100% wincertstore-0.2 | 13 KB | ########## | 100% wheel-0.32.2 | 52 KB | ########## | 100% setuptools-40.4.3 | 576 KB | ########## | 100% certifi-2018.10.15 | 138 KB | ########## | 100% python-3.7.1 | 19.5 MB | ########## | 100%

Maybe it is the problem.. I'll try a new thread to reach MikeyEcology

[Nova-Scotia]

@tundraboyce try using a folder that doesn't have special characters (a hyphen).