google / aiyprojects-raspbian

API libraries, samples, and system images for AIY Projects (Voice Kit and Vision Kit)
https://aiyprojects.withgoogle.com/
Apache License 2.0
1.62k stars 694 forks source link

Model 'object_detection' is not loaded. #488

Closed RickZLLiao closed 5 years ago

RickZLLiao commented 6 years ago

Hi

I retrain with MobileNet SSD V1 with 256x256 DM=0.125,but I got an issue as follw Could anybody solve the issue ? pi@raspberrypi:~/AIY-voice-kit-python/src/examples/vision $ ./object_detection.py --input glassskew_Resized.png --output glassskew_detected.png Traceback (most recent call last): File "/opt/aiy/projects-python/src/aiy/_drivers/_spicomm.py", line 128, in transact fcntl.ioctl(self._dev, SPICOMM_IOCTL_TRANSACT, self._tbuf) OSError: [Errno 14] Bad address

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./object_detection.py", line 39, in main result = inference.run(image_center) File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 100, in run return self._engine.image_inference(self._key, image, params) File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 327, in image_inference return self._communicate(request).inference_result File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 191, in _communicate response.ParseFromString(self._transport.send(request.SerializeToString())) File "/opt/aiy/projects-python/src/aiy/_drivers/_transport.py", line 33, in send return self._spicomm.transact(request) File "/opt/aiy/projects-python/src/aiy/_drivers/_spicomm.py", line 143, in transact raise SpicommInternalError aiy._drivers._spicomm.SpicommInternalError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./object_detection.py", line 50, in main() File "./object_detection.py", line 46, in main image.save(args.output) File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 110, in exit self.close() File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 103, in close self._engine.unload_model(self._key) File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 251, in unload_model self._communicate(request) File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 194, in _communicate raise InferenceException(response.status.message) aiy.vision.inference.InferenceException: UnloadModel: Model 'object_detection' is not loaded.

RickZLLiao commented 6 years ago

My Config was as follow:

SSD with Mobilenet v1, configured for Oxford-IIIT Pets Dataset.

Users should configure the fine_tune_checkpoint field in the train config as

well as the label_map_path and input_path fields in the train_input_reader and

eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that

should be configured.

model { ssd { num_classes: 1 box_coder { faster_rcnn_box_coder { y_scale: 10.0 x_scale: 10.0 height_scale: 5.0 width_scale: 5.0 } } matcher { argmax_matcher { matched_threshold: 0.5 unmatched_threshold: 0.5 ignore_thresholds: false negatives_lower_than_unmatched: true force_match_for_each_row: true } } similarity_calculator { iou_similarity { } } anchor_generator { ssd_anchor_generator { num_layers: 6 min_scale: 0.2 max_scale: 0.95 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 aspect_ratios: 3.0 aspect_ratios: 0.3333 } } image_resizer { fixed_shape_resizer { height: 256 width: 256 } } box_predictor { convolutional_box_predictor { min_depth: 0 max_depth: 0 num_layers_before_predictor: 0 use_dropout: false dropout_keep_probability: 0.8 kernel_size: 1 box_code_size: 4 apply_sigmoid_to_scores: false conv_hyperparams { activation: RELU_6, regularizer { l2_regularizer { weight: 0.00004 } } initializer { truncated_normal_initializer { stddev: 0.03 mean: 0.0 } } batch_norm { train: true, scale: true, center: true, decay: 0.9997, epsilon: 0.001, } } } } feature_extractor { type: 'ssd_mobilenet_v1' min_depth: 16 depth_multiplier: 0.125 conv_hyperparams { activation: RELU_6, regularizer { l2_regularizer { weight: 0.00004 } } initializer { truncated_normal_initializer { stddev: 0.03 mean: 0.0 } } batch_norm { train: true, scale: true, center: true, decay: 0.9997, epsilon: 0.001, } } } loss { classification_loss { weighted_sigmoid { } } localization_loss { weighted_smooth_l1 { } } hard_example_miner { num_hard_examples: 3000 iou_threshold: 0.99 loss_type: CLASSIFICATION max_negatives_per_positive: 3 min_negatives_per_image: 0 } classification_weight: 1.0 localization_weight: 1.0 } normalize_loss_by_num_matches: true post_processing { batch_non_max_suppression { score_threshold: 1e-8 iou_threshold: 0.6 max_detections_per_class: 100 max_total_detections: 100 } score_converter: SIGMOID } } }

train_config: { batch_size: 1 optimizer { rms_prop_optimizer: { learning_rate: { exponential_decay_learning_rate { initial_learning_rate: 0.004 decay_steps: 800720 decay_factor: 0.95 } } momentum_optimizer_value: 0.9 decay: 0.9 epsilon: 1.0 } } fine_tune_checkpoint: "" from_detection_checkpoint: true

Note: The below line limits the training process to 200K steps, which we

empirically found to be sufficient enough to train the pets dataset. This

effectively bypasses the learning rate schedule (the learning rate will

never decay). Remove the below line to train indefinitely.

num_steps: 20000 data_augmentation_options { random_horizontal_flip { } } data_augmentation_options { ssd_random_crop { } } }

train_input_reader: { tf_record_input_reader { input_path: "I:/work/Other/L3C_Glass_skew/Model_256_256_DM_P125/data/train.record" } label_map_path: "I:/work/Other/L3C_Glass_skew/Label.pbtxt" }

eval_config: { num_examples: 2000

Note: The below line limits the evaluation process to 10 evaluations.

Remove the below line to evaluate indefinitely.

max_evals: 10 }

eval_input_reader: { tf_record_input_reader { input_path: "I:/work/Other/skew/Model_256_256_DM_P125/data/train.record" } label_map_path: "I:/work/Other/skew/Label.pbtxt" shuffle: false num_readers: 1 }

kissste commented 6 years ago

I have had this issue before, something is wrong with your model. The chip is not able to handle. Based on my web searching, nobody knows what is wrong. It simply fails during execution.

Solution - use a model that works to start with and retrain that one.

RickZLLiao commented 6 years ago

Could you please provide the check point file & Mobile SSD V1 config file to me, then I could train it myself. thanks.

kissste commented 6 years ago

You will find what you are looking for here: https://github.com/google/aiyprojects-raspbian/issues/402 and https://github.com/google/aiyprojects-raspbian/issues/314

RickZLLiao commented 6 years ago

Today i retrain Embedded MobileNet V1 SSD in the tensorflow 1.10,compiled successful. But i got same problem - Model 'object_detection' is not loaded. How to get rid of it ?

RickZLLiao commented 6 years ago

my transfer command was as follow : ./bonnet_model_compiler.par --frozen_graph_path=./Glass_skewEmbeded_MobileNet_SSD_Res256_DM_P125/Glassskew_Embeded_Model_256_256_DM_P125_20180914_1857.pb --output_graph_path=./Glass_skewEmbeded_MobileNet_SSD_Res256_DM_P125/Glassskew_Embeded_Model_256_256_DM_P125_20180914_1857.binaryproto --input_tensor_name="Preprocessor/sub" --output_tensor_names="concat,concat_1" --input_tensor_size=256 --debug

weiran-work commented 6 years ago

@RickZLLiao @kissste

VisionKit has very limited HW resources and it can only support particular architecture with specific configuration.

When you configure the training pipeline, start with embedded_ssd_mobilenet_v1_coco.config, https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/embedded_ssd_mobilenet_v1_coco.config That one is VisionKit friendly.

We provided more details about your use case, search How to train and deploy a customized object detection model at https://aiyprojects.withgoogle.com/vision/#makers-guide

zhoujustin@ provided a pretrained checkpoint in issue 314

Let me know if this solves the issue or not. Thanks!

RickZLLiao commented 5 years ago

@sainttelant Thanks for your help! Would I have to reflesh Image first ?

RickZLLiao commented 5 years ago

@sainttelant I test 4 case, (1)python3 test_run_model_on_bonnet.py --model_path /opt/aiy/models/mobilenet_ssd_256res_0.125_person_cat_dog.binaryproto --input_height 256 --input_width 256 ==>Default Model,It's ok. (2)python3 test_run_model_on_bonnet.py --model_path /opt/aiy/models/Glassskew_Embeded_Model_256_256_DM_P125_20180914_1857.binaryproto ==> It's failed. (My training model - Embeded Mobile Model,256x256, depth Multiplier: 0.125 ),Error Messafe as follow... (3)python3 test_run_model_on_bonnet.py --model_path /opt/aiy/models/pikachu_detector.binaryproto --input_height 256 --input_width 256 ==> It's failed. (.binaryproto was from https://github.com/giacomobartoli/vision-kit/tree/master/pikachu-detector) (4)python3 test_run_model_on_bonnet.py --model_path /opt/aiy/models/cat_detector_cwh_180602.binaryproto --input_height 256 --input_width 256 ==>It's failed.(.binaryproto was from chadwallacehart's github ,https://github.com/chadwallacehart/aiy_custom_cat_detector)

Case (2)~(4),has same error , as follow Traceback (most recent call last): File "/opt/aiy/projects-python/src/aiy/_drivers/_spicomm.py", line 128, in transact fcntl.ioctl(self._dev, SPICOMM_IOCTL_TRANSACT, self._tbuf) OSError: [Errno 14] Bad address

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "test_run_model_on_bonnet.py", line 66, in main for i, result in enumerate(camera_inference.run()): File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 76, in run yield self._engine.camera_inference() File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 267, in camera_inference return self._communicate(request).inference_result File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 191, in _communicate response.ParseFromString(self._transport.send(request.SerializeToString())) File "/opt/aiy/projects-python/src/aiy/_drivers/_transport.py", line 33, in send return self._spicomm.transact(request) File "/opt/aiy/projects-python/src/aiy/_drivers/_spicomm.py", line 143, in transact raise SpicommInternalError aiy._drivers._spicomm.SpicommInternalError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "test_run_model_on_bonnet.py", line 77, in main (i, fps, ','.join(output_tensor_str))) File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 87, in exit self.close() File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 79, in close self._engine.stop_camera_inference() File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 273, in stop_camera_inference self._communicate(request) File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 194, in _communicate raise InferenceException(response.status.message) aiy.vision.inference.InferenceException: StopCameraInference: Camera inference is not running.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "test_run_model_on_bonnet.py", line 81, in main() File "test_run_model_on_bonnet.py", line 77, in main (i, fps, ','.join(output_tensor_str))) File "/usr/lib/python3/dist-packages/picamera/camera.py", line 758, in exit self.close() File "/usr/lib/python3/dist-packages/picamera/camera.py", line 752, in close raise exc File "/usr/lib/python3/dist-packages/picamera/camera.py", line 1995, in _control_callback "No data recevied from sensor. Check all connections, " picamera.exc.PiCameraRuntimeError: No data recevied from sensor. Check all connections, including the SUNNY chip on the camera board

RickZLLiao commented 5 years ago

Sorry sainttelant,I attach wrong person。 I need your help,weiranzhao.

RickZLLiao commented 5 years ago

@sainttelant & @weiranzhao,I need your help.

weiran-work commented 5 years ago

@RickZLLiao Quick thing to check, are you on the latest SD card image?

RickZLLiao commented 5 years ago

@weiranzhao,I'm not update to latestest SD card image. I'll try today, and let you know the result, thanks.

RickZLLiao commented 5 years ago

@weiranzhao,After i update the latestest SD card image (aiyprojects-2018-08-03.img),i got errors as follow : (No such file or directory: '/dev/vision_spicomm')

pi@raspberrypi:~/AIY-projects-python/src/examples/vision $ python3 any_model_camera.py --model_path /opt/aiy/models/Glassskew_Embeded_Model_256_256_DM_P125_20180914_1857.binaryproto --input_height 256 --input_width 256 Traceback (most recent call last): File "any_model_camera.py", line 63, in main() File "any_model_camera.py", line 56, in main with CameraInference(model) as inference: File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 94, in init self._engine = self._stack.enter_context(InferenceEngine()) File "/opt/aiy/projects-python/src/aiy/vision/inference.py", line 227, in init self._transport = make_transport() File "/opt/aiy/projects-python/src/aiy/_drivers/_transport.py", line 89, in make_transport return _SpiTransport() File "/opt/aiy/projects-python/src/aiy/_drivers/_transport.py", line 29, in init self._spicomm = _spicomm.Spicomm() File "/opt/aiy/projects-python/src/aiy/_drivers/_spicomm.py", line 289, in init self._dev = os.open(SPICOMM_DEV, os.O_RDWR) FileNotFoundError: [Errno 2] No such file or directory: '/dev/vision_spicomm'

RickZLLiao commented 5 years ago

@weiranzhao,I try commands as follw, but i got same error ==> sudo apt-get update; sudo apt-get upgrade; sudo reboot ==> dpkg -l aiy* from : https://aiyprojects.withgoogle.com/help/#updating-your-sd-card

weiran-work commented 5 years ago

@RickZLLiao

That error seems like system issue. Could you try to reflash the sd card with latest image?

weiran-work commented 5 years ago

@dmitriykovalev

Any idea on the error message after reflashing?

RickZLLiao commented 5 years ago

@weiranzhao,After i update the latestest SD card image (aiyprojects-2018-08-03.img) & finishing update,I also found message : ' Failed to start load kernel modules ' , when rebooting in the first line.

manoj7410 commented 5 years ago

@RickZLLiao There is some issue going on with apt-get upgrade. Download the sdcard image from : https://github.com/google/aiyprojects-raspbian/releases/tag/v20180803 and you need not to perform any upgrade on it.

RickZLLiao commented 5 years ago

@manoj7410,thanks your responding, i'll try it. And let you know the result.

RickZLLiao commented 5 years ago

@weiranzhao @manoj7410 @dmitriykovalev After download the sdcard image from: https://github.com/google/aiyprojects-raspbian/releases/tag/v20180803,i could excute my embedded_mobilenet_ssd model successfully。 I excute cat detection and pikachu detection also successfully. Thanks all your help.

abhinavrawat27 commented 5 years ago

@RickZLLiao Can you share the command which you used to retrain the embedded_mobilnet model please. Thanks