search

matterport / Mask_RCNN

Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow
Other
24.58k stars 11.69k forks source link

Training balloon model on gpu: AttributeError: 'Model' object has no attribute 'metrics_tensors' #1754

Open HjalteMann opened 5 years ago

HjalteMann commented 5 years ago

I am trying to train a model with the balloon example. I can get the training running on my local cpu, but when I attempt to run the balloon.py on my gpu (and actually also from the Jupyter Notebook) I get the error below. Everything seemed to install fine, bot requirements and the setup.py.

python balloon.py train dataset=/home/Mask_RCNN/samples/balloon/balloon/ --weights=mask_rcnn_coco.h5

Traceback (most recent call last): File "/home/bitcue/Dropbox/mrcnn2/Mask_RCNN/samples/balloon/balloon.py", line 367, in <module> train(model) File "/home/bitcue/Dropbox/mrcnn2/Mask_RCNN/samples/balloon/balloon.py", line 201, in train layers='heads') File "/home/bitcue/anaconda3/envs/mrcnn2/lib/python3.6/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py", line 2354, in train File "/home/bitcue/anaconda3/envs/mrcnn2/lib/python3.6/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py", line 2199, in compile AttributeError: 'Model' object has no attribute 'metrics_tensors

I have the following packages installed in an anaconda environment:

Name Version Build Channel _libgcc_mutex 0.1 main _tflow_select 2.1.0 gpu absl-py 0.7.1 py36_0 astor 0.8.0 py36_0 backports 1.0 py_2 backports.weakref 1.0rc1 py36_0 blas 1.0 mkl bleach 1.5.0 py36_0 c-ares 1.15.0 h7b6447c_1001 ca-certificates 2019.5.15 1 anaconda certifi 2019.6.16 py36_1 anaconda cudatoolkit 9.0 h13b8566_0 cudnn 7.6.0 cuda9.0_0 cupti 9.0.176 0 expat 2.2.6 he6710b0_0 gast 0.2.2 py36_0 git 2.20.1 pl526hacde149_0 grpcio 1.16.1 py36hf8bcb03_1 h5py 2.9.0 py36h7918eee_0 hdf5 1.10.4 hb1b8bf9_0 html5lib 0.9999999 py36_0 intel-openmp 2019.5 281 keras-applications 1.0.8 py_0 keras-gpu 2.0.8 py36h0585f72_0 anaconda keras-preprocessing 1.1.0 py_1 krb5 1.16.1 h173b8e3_7 libcurl 7.65.3 h20c2e04_0 libedit 3.1.20181209 hc058e9b_0 libffi 3.2.1 hd88cf55_4 libgcc 7.2.0 h69d50b8_2 libgcc-ng 9.1.0 hdf63c60_0 libgfortran-ng 7.3.0 hdf63c60_0 libprotobuf 3.8.0 hd408876_0 libssh2 1.8.2 h1ba5d50_0 libstdcxx-ng 9.1.0 hdf63c60_0 markdown 3.1.1 py36_0 mkl 2019.5 281 mkl-service 2.3.0 py36he904b0f_0 mkl_fft 1.0.14 py36ha843d7b_0 mkl_random 1.0.2 py36hd81dba3_0 mock 3.0.5 py36_0 ncurses 6.1 he6710b0_1 numpy 1.16.5 py36h7e9f1db_0 numpy-base 1.16.5 py36hde5b4d6_0 openssl 1.1.1 h7b6447c_0 anaconda perl 5.26.2 h14c3975_0 pip 19.2.2 py36_0 protobuf 3.8.0 py36he6710b0_0 python 3.6.9 h265db76_0 pyyaml 5.1.2 py36h7b6447c_0 readline 7.0 h7b6447c_5 scipy 1.3.1 py36h7c811a0_0 setuptools 41.0.1 py36_0 six 1.12.0 py36_0 sqlite 3.29.0 h7b6447c_0 tensorboard 1.13.1 py36hf484d3e_0 tensorflow 1.13.1 gpu_py36h26cf82e_0 tensorflow-base 1.13.1 gpu_py36h8f37b9b_0 tensorflow-estimator 1.13.0 py_0 tensorflow-gpu 1.13.1 h0d30ee6_0 tensorflow-gpu-base 1.7.0 py36hcdda91b_1 tensorflow-tensorboard 1.5.1 py36hf484d3e_1 termcolor 1.1.0 py36_1 tk 8.6.8 hbc83047_0 werkzeug 0.15.5 py_0 wheel 0.33.4 py36_0 xz 5.2.4 h14c3975_4 yaml 0.1.7 had09818_2 zlib 1.2.11 h7b6447c_3

yangdeqieyingli commented 5 years ago

i meet the same question,How to solve it?

msrkp commented 5 years ago

Initialise self.keras_model.metrics_tensors = [] then you are good to go...

Hunain-Gillani commented 5 years ago

If anyone is looking for the solution, downgrade keras to 2.1.0. It started working for me.

HjalteMann commented 5 years ago

Thank you for the reply, @51r1u5 ! Where in the scripts should I do this?

@Hunain-G, I'm using keras 2.0.8 as advised in the readme. Do you propose to upgrade?

Hunain-Gillani commented 5 years ago

@HjalteMann , if you're using keras 2.0.8 then there shouldn't have been any issue. I was using keras 2.3.0 and by downgrading keras to 2.1.0 made the issue resolved.

mashuai191 commented 5 years ago

since Keras 2.3.0 there is such issue, when i downgrade to 2.2.5 the issue resolved so i believe any version <= 2.2.5 would work, and of course should be >= 2.0.8 as requiremets.txt suggested.

mffigueroa commented 5 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name)

JepsonNomad commented 5 years ago

I got the same error but when trying to run train_shapes (as described here). Fixed by installing keras version 2.2.5 and tensorflow version 1.15.0rc2 (pip3 install keras==2.2.5 and the same formatting for TF). Keras version 2.1.0 did not work for me, nor did version 2.3.0.

freitaucher commented 5 years ago

Did someone succeed to run any example with tensorflow 1.13 ? This is the oldest version which is compatible with cuda10.0. Older versions require older cuda's, but installing several cuda's is "highly" inconvenient

julian-vidal commented 5 years ago

I was having the same error, but after installing keras 2.0.8 as you said above. Now I have other error: File "/Users/julianvidal/anaconda3/envs/MaskRCNN/lib/python3.6/site-packages/keras/callbacks/callbacks.py", line 23, in <module> from ..engine.training_utils import standardize_input_data ModuleNotFoundError: No module named 'keras.engine.training_utils'

Can you help please? Thanks!

julian-vidal commented 5 years ago

Initialise self.keras_model.metrics_tensors = [] then you are good to go...

Can you give more details about how to do this? Thanks!

rteuwens commented 4 years ago

since Keras 2.3.0 there is such issue, when i downgrade to 2.2.5 the issue resolved so i believe any version <= 2.2.5 would work, and of course should be >= 2.0.8 as requiremets.txt suggested.

ran tf.keras.__version__ in the console in a Kaggle kernel and got '2.2.4-tf'. I'm struggling with the same issue, though, and I'm not sure how to downgrade on Kaggle.

trialzuki commented 4 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name)

It's solve the problem for 2.3.0

HjalteMann commented 4 years ago

My issue was solved by installing all dependencies in a new conda environment, with pip, not conda.

dfy888 commented 4 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name)

it is work! 非常感谢

pipun-gif commented 4 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name)

it is work! 非常感谢

hi excuse me @dfy888 how to edit model.py inside egg file? i can open it by rename the .egg into .zip and modify all file inside that egg, but unfortunately i don't know package it back as a python egg file after my modification. thank you

ZOUYIyi commented 4 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name)

it is work! 非常感谢

我应该在哪里修改这一句呢？

pipun-gif commented 4 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name) it is work! 非常感谢

我应该在哪里修改这一句呢？

around here : anaconda3/envs/mrcnn2/lib/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py

oktemerbay commented 4 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name) it is work! 非常感谢

我应该在哪里修改这一句呢？

around here : anaconda3/envs/mrcnn2/lib/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py

hi , first change model.py file's line 2199 , which is "self.keras_model.metrics_tensors.append(loss)" to "self.keras_model.add_metric(loss, name)" and then with this command "python .\setup.py install" generate new egg file. You cannot directly change the content of the egg file.

Also , I do not recommend downgrading tensorflow because in return , it generates performance issuses.

tourist-C commented 4 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name) it is work! 非常感谢

我应该在哪里修改这一句呢？

mrcnn\model.py line2199

abhishekpundir22 commented 4 years ago

i have keras version 2.2.0 and tf 1.16.0 python 3.6.10 and having this issue.... many said it will work on this... anyone has its fix? I tried multiple combinations of version till now.. nothing worked. thanks!

rteuwens commented 4 years ago

i have keras version 2.2.0 and tf 1.16.0 python 3.6.10 and having this issue.... many said it will work on this... anyone has its fix? I tried multiple combinations of version till now.. nothing worked. thanks!

Hey Abhishek, check out @mffigueroa 's answer:

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name)

As @tourist-C points out, it's on line 2199 in mcrnn/model.py As here: https://github.com/rteuwens/Mask_RCNN/blob/master/mrcnn/model.py#L2199

banafsh89 commented 4 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name) it is work! 非常感谢

我应该在哪里修改这一句呢？

around here : anaconda3/envs/mrcnn2/lib/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py

This path doesn't exist for me. How can I find where the model.py is or where my mrcnn package is saved in anaconda?

Independent-Laboratory commented 4 years ago

In my case, this works.

cd Mask_RCNN # Root directory of Mask RCNN emacs mrcnn/model.py # Edit model.py

--------------------------------------

At line 2199

self.keras_model.metrics_tensors.append(loss) # <- comment out

self.keras_model.add_metric(loss, name) # <- Add

--------------------------------------

sudo python3 setup.py install # <- mask_rcnn-2.1-py3.6.egg is automaticalliy replaced

Then, balloon training script will work.

Mstfakts commented 4 years ago

If anyone is looking for the solution, downgrade keras to 2.1.0. It started working for me.

It did not work for me, I got another error about keras.

Hunain-Gillani commented 4 years ago

If anyone is looking for the solution, downgrade keras to 2.1.0. It started working for me.

It did not work for me, I got another error about keras.

Can you specify the error? What the error is now?

Mstfakts commented 4 years ago

If anyone is looking for the solution, downgrade keras to 2.1.0. It started working for me.

It did not work for me, I got another error about keras.

Can you specify the error? What the error is now?

Actually, I do not exactly remember the error but @mffigueroa 's solution worked well.

gethubwy commented 4 years ago

Initialise self.keras_model.metrics_tensors = [] then you are good to go... well,I use keras version=2.3.1,it is not work for me.

abhishekpundir22 commented 4 years ago

I have solved this issue by changing keras version to 2.1.0 and tensorflow version to 1.15.0

abhishekpundir22 commented 4 years ago

i have keras version 2.2.0 and tf 1.16.0 python 3.6.10 and having this issue.... many said it will work on this... anyone has its fix? I tried multiple combinations of version till now.. nothing worked. thanks!

Hey Abhishek, check out @mffigueroa 's answer:

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name)

As @tourist-C points out, it's on line 2199 in mcrnn/model.py As here: https://github.com/rteuwens/Mask_RCNN/blob/master/mrcnn/model.py#L2199

This works too. I can confirm. Thanks.

gethubwy commented 4 years ago

i have keras version 2.2.0 and tf 1.16.0 python 3.6.10 and having this issue.... many said it will work on this... anyone has its fix? I tried multiple combinations of version till now.. nothing worked. thanks!

Hey Abhishek, check out @mffigueroa 's answer:

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name)

As @tourist-C points out, it's on line 2199 in mcrnn/model.py As here: https://github.com/rteuwens/Mask_RCNN/blob/master/mrcnn/model.py#L2199

This works too. I can confirm. Thanks.

"name“ is what?

abhishekpundir22 commented 4 years ago

Maybe its a parameter having name of the Metric function.. I am not fully sure as this was used in earlier versions. Above solution is a just a tweak. Can you give me your keras version and tensorflow version details?

gethubwy commented 4 years ago

In my case, this works.

cd Mask_RCNN # Root directory of Mask RCNN emacs mrcnn/model.py # Edit model.py

--------------------------------------

At line 2199

self.keras_model.metrics_tensors.append(loss) # <- comment out

self.keras_model.add_metric(loss, name) # <- Add

--------------------------------------

sudo python3 setup.py install # <- mask_rcnn-2.1-py3.6.egg is automaticalliy replaced

Then, balloon training script will work.

Hello,bro, I wonder how do you distinguish between a instances_train2017.json file and instances_val2017.json file. I have labled data file, I want to train a new model starting from pre-trained COCO weights, but I don't know how to name the json file,and the vaild json file should like what format? Thank you.

gethubwy commented 4 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name) Hello,I want to train a new model starting from pre-trained COCO weights, but I don't know how to name the json file,and the vaild json file should like what format? Thank you. When you train a new model ,I wonder how do you distinguish between a instances_train2017.json file and instances_val2017.json file. I have labled data file,

abhishekpundir22 commented 4 years ago

As per the version I used keras 2.1.0 and for the format I recommend you to use VGG annotation tool. It produces the exact format that is required to feed the model. Its available online.

gethubwy commented 4 years ago

Maybe its a parameter having name of the Metric function.. I am not fully sure as this was used in earlier versions. Above solution is a just a tweak. Can you give me your keras version and tensorflow version details?

Excuse me, I use the software named "labelme" to label my data, I want to train a new model starting from pre-trained COCO weights, but I don't know how to write the annotation file, are there tools to help me? Thank you.

gethubwy commented 4 years ago

In my case, this works.

cd Mask_RCNN # Root directory of Mask RCNN emacs mrcnn/model.py # Edit model.py

--------------------------------------

At line 2199

self.keras_model.metrics_tensors.append(loss) # <- comment out

self.keras_model.add_metric(loss, name) # <- Add

--------------------------------------

sudo python3 setup.py install # <- mask_rcnn-2.1-py3.6.egg is automaticalliy replaced

Then, balloon training script will work. Excuse me, I want to know how to train a new mode, I use the software named "labelme" to label my data, I want to train a new model starting from pre-trained COCO weights, but I don't know how to write the annotation file, are there tools to help me? Thank you.

gethubwy commented 4 years ago

In my case, this works.

cd Mask_RCNN # Root directory of Mask RCNN emacs mrcnn/model.py # Edit model.py

--------------------------------------

At line 2199

self.keras_model.metrics_tensors.append(loss) # <- comment out

self.keras_model.add_metric(loss, name) # <- Add

--------------------------------------

sudo python3 setup.py install # <- mask_rcnn-2.1-py3.6.egg is automaticalliy replaced

Then, balloon training script will work. Excuse me, I want to know how to train a new mode, I use the software named "labelme" to label my data, I want to train a new model starting from pre-trained COCO weights, but I don't know how to write the annotation file, are there tools to help me? Thank you.

sandipan commented 3 years ago

it works for me.

Herant commented 3 years ago

This worked for me in colab, run these lines before importing packages and initializing ROOT_DIR

!pip uninstall tensorflow !pip install tensorflow-gpu==1.13.1

%tensorflow_version 1.x import tensorflow print(tensorflow.version)

!pip uninstall keras-nightly !pip uninstall keras !pip install h5py==2.10.0 !pip install q keras==2.1.0

XTiNCT-7 commented 2 years ago

I was having the same error, but after installing keras 2.0.8 as you said above. Now I have other error: File "/Users/julianvidal/anaconda3/envs/MaskRCNN/lib/python3.6/site-packages/keras/callbacks/callbacks.py", line 23, in <module> from ..engine.training_utils import standardize_input_data ModuleNotFoundError: No module named 'keras.engine.training_utils'

Can you help please? Thanks!

use keras version 2.2.5

Behnam72 commented 2 years ago

I had the same issue caused by incompatibility of packages. I've seen many solutions that contradict each other in different websites. What I came up with is this solution using anaconda:

conda create --name mrcnnGPU python=3.6 tensorflow-gpu==1.14.0 keras==2.2.4 opencv imgaug tqdm tifffile imagecodecs pandas hdf5 h5py jupyter notebook

keras==2.1.0 works as well, but there are fewer warnings with 2.2.4

XTiNCT-7 commented 2 years ago

https://github.com/akTwelve/Mask_RCNN/tree/master/mrcnn u can copy paste this all file in mrcnn folder. check if this helps

aggelos-michael-papadopoulos commented 2 years ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name)

THANK YOU !!!

dhruvgoel99 commented 2 years ago

hi, replace line: self.keras_model.metrics_tensors.append(loss) to line: self.keras_model.add_loss(loss)

otherwise it will give error for aggregation parameter

gbinduo commented 11 months ago

What about changing: self.keras_model.metrics_tensors.append(loss) to: self.keras_model.add_metric(loss, name)

Perfect!

MSOHAPPY commented 4 months ago

Initialise self.keras_model.metrics_tensors = [] then you are good to go...

veeeeeeery thank you!After changing to keras2.3.0, I encountered this problem, and the solution you provided solved it perfectly. By the way, it also solved the problem of GPU utilization being 0 that I encountered before!