Open haimat opened 3 years ago
I would like to help, could you please provide colab copy of the code. As I am not able to find the training code.
I would like to help, could you please provide colab copy of the code. As I am not able to find the training code.
Thanks for the offer to help. But frankly there is no Colab, since I am solely using standard tools from the research object detection API for TF2. Following the docs I did the following:
.tfrecord
file with all my imagespipeline.config
file from that modelmodel_main_tf2.py
script from the object detection API:python model_main_tf2.py --model_dir=/path/to/efficientdet_d3_coco17_tpu-32 --pipeline_config_path=/path/to/efficientdet_d3_coco17_tpu-32/pipeline.config
That's it, no custom steps involved, everything according the official docs and using the official API and tools. Hope that helps!
I'm interesting at this topic too. I wondering also if there's a way to add some callbacks (or estimators) such as Early Stopping during training model, obviously with the latest and official Object Detection API.
@haimat I have solved this problem while making as few changes as possible to the original object detection api using the ff. steps:
with open(os.path.join(top_checkpoints_path,"checkpoint"), "w") as checkpoint_file:
checkpoint_file.write("model_checkpoint_path: \"ckpt-{}\"".format(best_checkpoints[0]))
@jartantupjar Thanks for your ideas. I will give that a try - but frankly, an automated solution would still be great ;-)
@hiamat I have automated this step. Its just one line script that automatically selects the checkpoint, freezes the checkpoint and converts it to tflite. But yeah, it would be great if it was something that was implemented internally.
@haimat I might have not understood the issue accurately so I was just curious to know, how save_best_only
feature of tensorflow.keras.callbacks.ModelCheckpoint
not solve the problem?
@iamarchisha Well, maybe it would, but I don't know - how can one integrate that with the model_main_tf2.py
script?
inside model_lib.py i changed this parameters for getting best ckpt, still its not working https://www.tensorflow.org/api_docs/python/tf/estimator/BestExporter
exporter = tf.estimator.BestExporter(export_to_keep=10,compare_fn=_loss_smaller, name=exporter_name, serving_input_receiver_fn=None) eval_specs.append( tf.estimator.EvalSpec( name=eval_spec_name, input_fn=eval_input_fn, steps=None, exporters=exporter))
I would like to help, could you please provide colab copy of the code. As I am not able to find the training code.
Thanks for the offer to help. But frankly there is no Colab, since I am solely using standard tools from the research object detection API for TF2. Following the docs I did the following:
- Installed the object detection API
- Created a
.tfrecord
file with all my images- Downloaded the EfficientDet D3 model
- Copied over and modified the
pipeline.config
file from that model- Run the
model_main_tf2.py
script from the object detection API:python model_main_tf2.py --model_dir=/path/to/efficientdet_d3_coco17_tpu-32 --pipeline_config_path=/path/to/efficientdet_d3_coco17_tpu-32/pipeline.config
That's it, no custom steps involved, everything according the official docs and using the official API and tools. Hope that helps!
Helloo i have errors while running model_main_tf2.py. " _python model_main_tf2.py --alsologtostderr --model_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2pets.config "
I dont have any custom steps i did everything from here
But my tensorflow version is 2.5 and i did all setups according to 2.5
So the errors are =
File "C:\tensorflow1\models\research\object_detection\model_main_tf2.py", line 116, in
I searched for 4 days to find a solution. Im so new to this topic and couldnt solve the problems. I would be so appreciate if you help me.
@PelinSuK Chances are you trying to run a TF 1 model on the TF 2 version. Try to run a tf2 model you can get from tf2 model zoo because load_all_detection_checkpoint_vars variable in your pipeline.config file indicates its a TF1 model
@PelinSuK Chances are you trying to run a TF 1 model on the TF 2 version. Try to run a tf2 model you can get from tf2 model zoo because load_all_detection_checkpoint_vars variable in your pipeline.config file indicates its a TF1 model
reallyy? but i dont know how to change this is the mine pipeline.config and what should i change or do i need to change ? is it enough to download Faster R-CNN Inception ResNet V2 1024x1024 from your link (faster_rcnn_inception_resnet_v2_1024x1024_coco17_tpu-8.tar) (btw i download that one after you linked to me) thank you soo much for your reply !
@PelinSuK Chances are you trying to run a TF 1 model on the TF 2 version. Try to run a tf2 model you can get from tf2 model zoo because load_all_detection_checkpoint_vars variable in your pipeline.config file indicates its a TF1 model
reallyy? but i dont know how to change this is the mine pipeline.config and what should i change or do i need to change ? is it enough to download Faster R-CNN Inception ResNet V2 1024x1024 from your link (faster_rcnn_inception_resnet_v2_1024x1024_coco17_tpu-8.tar) (btw i download that one after you linked to me) thank you soo much for your reply !
the model zoo download should have both the model files and the pipeline.config file. You would need to update the same stuff on the pipeline.config file [fine_tune_checkpoint, train_input_reader, eval_input_reader]
@PelinSuK Chances are you trying to run a TF 1 model on the TF 2 version. Try to run a tf2 model you can get from tf2 model zoo because load_all_detection_checkpoint_vars variable in your pipeline.config file indicates its a TF1 model
reallyy? but i dont know how to change this is the mine pipeline.config and what should i change or do i need to change ? is it enough to download Faster R-CNN Inception ResNet V2 1024x1024 from your link (faster_rcnn_inception_resnet_v2_1024x1024_coco17_tpu-8.tar) (btw i download that one after you linked to me) thank you soo much for your reply !
the model zoo download should have both the model files and the pipeline.config file. You would need to update the same stuff on the pipeline.config file [fine_tune_checkpoint, train_input_reader, eval_input_reader]
sad news that i tried (_python model_main_tf2.py --logtostderr --model_dir=training/ --pipeline_config_path=training/faster_rcnn_resnet152pets.config)
and dont know whats wrong with it , put all my relative files here still having the errors :(
These are = File "C:\tensorflow1\models\research\object_detection\model_main_tf2.py", line 116, in
I have a quick fix for this. It might not be ideal but it works! In the model_lib_v2.py script, add these few lines of code,
#keep best_mAP record
best_mAP = 0.0
on top of
for latest_checkpoint in tf.train.checkpoints_iterator(
checkpoint_dir, timeout=timeout, min_interval_secs=wait_interval):
ckpt = tf.compat.v2.train.Checkpoint(
step=global_step, model=detection_model, optimizer=optimizer)
Insert this function somewhere
def find_ckpt_num(string):
for i in range(len(string)-1, -1 ,-1):
if string[i] == '/':
return(string[i+1:])
and add this modification
with summary_writer.as_default():
e_metrics = eager_eval_loop(
detection_model,
configs,
eval_input,
use_tpu=use_tpu,
postprocess_on_cpu=postprocess_on_cpu,
global_step=global_step,
)
if e_metrics['DetectionBoxes_Precision/mAP'] > best_mAP:
tf.logging.info('latest checkpoint' + latest_checkpoint)
ckpt_idx = latest_checkpoint + '.index'
ckpt_data = latest_checkpoint + '.data-00000-of-00001'
ckpt_name = find_ckpt_num(latest_checkpoint)
new_idx = 'path-to-training-dir/best_ckpt/' + ckpt_name + '.index'
new_data = 'path-to-training-dir/best_ckpt/' + ckpt_name + '.data-00000-of-00001'
shutil.copyfile(ckpt_idx, new_idx)
shutil.copyfile(ckpt_data, new_data)
best_mAP = e_metrics['DetectionBoxes_Precision/mAP']
tf.logging.info('current best mAP' + str(best_mAP))
The idea is to keep a record of the best mAP score at each evaluation run and then copy that checkpoint to a file called 'best_ckpt' if the current checkpoint has a better mAP score then the current best. You could do the same for every metric by modifying which metric you're comparing for. Also note that this can be done only at the evaluation loop.
Hope this helps! P.S. forgot to mention but rmb to import shutil in the script
I have a quick fix for this. It might not be ideal but it works! In the model_lib_v2.py script, add these few lines of code,
#keep best_mAP record best_mAP = 0.0
on top of
for latest_checkpoint in tf.train.checkpoints_iterator( checkpoint_dir, timeout=timeout, min_interval_secs=wait_interval): ckpt = tf.compat.v2.train.Checkpoint( step=global_step, model=detection_model, optimizer=optimizer)
Insert this function somewhere
def find_ckpt_num(string): for i in range(len(string)-1, -1 ,-1): if string[i] == '/': return(string[i+1:])
and add this modification
with summary_writer.as_default(): e_metrics = eager_eval_loop( detection_model, configs, eval_input, use_tpu=use_tpu, postprocess_on_cpu=postprocess_on_cpu, global_step=global_step, ) if e_metrics['DetectionBoxes_Precision/mAP'] > best_mAP: tf.logging.info('latest checkpoint' + latest_checkpoint) ckpt_idx = latest_checkpoint + '.index' ckpt_data = latest_checkpoint + '.data-00000-of-00001' ckpt_name = find_ckpt_num(latest_checkpoint) new_idx = 'path-to-training-dir/best_ckpt/' + ckpt_name + '.index' new_data = 'path-to-training-dir/best_ckpt/' + ckpt_name + '.data-00000-of-00001' shutil.copyfile(ckpt_idx, new_idx) shutil.copyfile(ckpt_data, new_data) best_mAP = e_metrics['DetectionBoxes_Precision/mAP'] tf.logging.info('current best mAP' + str(best_mAP))
The idea is to keep a record of the best mAP score at each evaluation run and then copy that checkpoint to a file called 'best_ckpt' if the current checkpoint has a better mAP score then the current best. You could do the same for every metric by modifying which metric you're comparing for. Also note that this can be done only at the evaluation loop.
Hope this helps! P.S. forgot to mention but rmb to import shutil in the script
Thank you so much ^^
with summary_writer.as_default()
Hello, Thanks for the hack. :-)
this part
with summary_writer.as_default():
e_metrics = eager_eval_loop(
detection_model,
configs,
eval_input,
use_tpu=use_tpu,
postprocess_on_cpu=postprocess_on_cpu,
global_step=global_step,
)
What part of the code does one add it to please?
can you share your version of the code please. Thanks
with summary_writer.as_default()
Hello, Thanks for the hack. :-)
this part
with summary_writer.as_default(): e_metrics = eager_eval_loop( detection_model, configs, eval_input, use_tpu=use_tpu, postprocess_on_cpu=postprocess_on_cpu, global_step=global_step, )
What part of the code does one add it to please?
can you share your version of the code please. Thanks
#keep best_mAP record
best_mAP = 0.5317
for latest_checkpoint in tf.train.checkpoints_iterator(
checkpoint_dir, timeout=timeout, min_interval_secs=wait_interval):
ckpt = tf.compat.v2.train.Checkpoint(
step=global_step, model=detection_model, optimizer=optimizer)
# We run the detection_model on dummy inputs in order to ensure that the
# model and all its variables have been properly constructed. Specifically,
# this is currently necessary prior to (potentially) creating shadow copies
# of the model variables for the EMA optimizer.
if eval_config.use_moving_averages:
unpad_groundtruth_tensors = (eval_config.batch_size == 1 and not use_tpu)
_ensure_model_is_built(detection_model, eval_input,
unpad_groundtruth_tensors)
optimizer.shadow_copy(detection_model)
ckpt.restore(latest_checkpoint).expect_partial()
if eval_config.use_moving_averages:
optimizer.swap_weights()
summary_writer = tf.compat.v2.summary.create_file_writer(
os.path.join(model_dir, 'eval', eval_input_config.name))
with summary_writer.as_default():
e_metrics = eager_eval_loop(
detection_model,
configs,
eval_input,
use_tpu=use_tpu,
postprocess_on_cpu=postprocess_on_cpu,
global_step=global_step,
)
if e_metrics['DetectionBoxes_Precision/mAP'] > best_mAP:
tf.logging.info('latest checkpoint' + latest_checkpoint)
ckpt_idx = latest_checkpoint + '.index'
ckpt_data = latest_checkpoint + '.data-00000-of-00001'
ckpt_file = '/content/drive/MyDrive/<model_file_name>/transfer_learning/checkpoint'
ckpt_name = find_ckpt_num(latest_checkpoint)
new_idx = '/content/drive/MyDrive/<model_file_name>/transfer_learning/best_ckpt/' + ckpt_name + '.index'
new_data = '/content/drive/MyDrive/<model_file_name>/transfer_learning/best_ckpt/' + ckpt_name + '.data-00000-of-00001'
new_ckpt = '/content/drive/MyDrive/<model_file_name>/transfer_learning/best_ckpt/checkpoint'
shutil.copyfile(ckpt_idx, new_idx)
shutil.copyfile(ckpt_data, new_data)
shutil.copyfile(ckpt_file, new_ckpt)
best_mAP = e_metrics['DetectionBoxes_Precision/mAP']
tf.logging.info('current best mAP' + str(best_mAP))
This is how the end of my model_lib_v2.py script looks like. Hope it helps!
Hello. I understand correctly that there is still no solution to this issue, and the only thing we can do is edit the eval_continuously
function model_lib_v2.py
. And run an assessment in parallel with the training? In 1 console model_main_tf2.py --model_dir=<DIR> --pipeline_config_path=pipeline.config
in 2 model_main_tf2.py --model_dir=<DIR> --pipeline_config_path=pipeline.config --checkpoint_dir=<DIR>
.
Prerequisites
Please answer the following question for yourself before submitting an issue.
1. The entire URL of the file you are using
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md
2. Describe the feature you request
During training model checkpoints are being saved, also the last version of the trained model, which might not be the best model. So there should be an option to save the best model as well. I know of the
tf.estimator.BestExporter
estimator, but how can I use that with the latest and officialmodel_main_tf2.py
script of the object detection API?Moreover, until there is such a solution, how can one get the best model during or after the training via the object detection API?
3. Additional context
I am following the official object detection API docs, thus using the
model_main_tf2.py
script, so a solution should be based on that script too, if possible.4. Are you willing to contribute it? (Yes or No)
In theory yes, but I am afraid that I don't know the TF object detection API well enough to be able to help.