Instructions for reproducing the paper's results

[achiyaj]

Hi,

Could you please add some instructions to test the trained models, and reproduce the results that were reported in the paper?

Thanks!

[yekeren]

Sorry, I cannot understand the question. What do you mean by testing? Do you mean to generate the xml or json files to submit to the website?

On Fri, Mar 13, 2020 at 7:41 AM achiyaj notifications@github.com wrote:

Hi,

Could you please add some instructions to test the trained models, and reproduce the results that were reported in the paper?

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/yekeren/Cap2Det/issues/20, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6CPAPEN36OR7DIFPUNWPDRHILXLANCNFSM4LHAWIQA .

-- Thanks, best regards.

Keren

[achiyaj]

Sorry for the inclarity. In the paper you have reported the accuracy of some different models on the VOC 2007/2012 test sets. Is there some script I can run to get this accuracy after I've trained a model? Or could you send some general instruction to get these numbers?

[yekeren]

You can check the predict.py'' in thetrain'' directory. For example, you could use the following command to evaluate on the VOC07 test set.

python "train/predict.py" \

--alsologtostderr \

--run_once \

--pipeline_proto="${PIPELINE_CONFIG_PATH}" \

--model_dir="${MODEL_DIR}" \

--eval_log_dir="${TMP}/${NAME}.eval_det" \

--saved_ckpts_dir="${TMP}/${NAME}.saved_ckpts" \

--visl_file_path="${TMP}/${NAME}.html" \

--results_dir="${TMP}" \

--max_eval_examples=50000 \

--evaluator="pascal" \

--label_file="data/voc_label.txt" \

|| exit -1

The ``predict.py'' also exports the results to an intermediate file. You can then convert the intermediate file to the XML or JSON format required by the challenge website.

python "train/predict.py" \

--alsologtostderr \

--run_once \

--input_pattern="output/coco17_testdev.record*" \

--shard_indicator="0/1" \

--pipeline_proto="${PIPELINE_CONFIG_PATH}" \

--model_dir="${MODEL_DIR}" \

--eval_log_dir="tmp/eval_det2" \

--saved_ckpts_dir="${MODEL_DIR}/saved_ckpts" \

--max_eval_examples=500000 \

--evaluator="coco" \

--detection_result_dir="${DETECTION_RESULT_DIR}" \

--vocabulary_file="${VOCAB_FILE}"

On Sat, Mar 14, 2020 at 3:51 AM achiyaj notifications@github.com wrote:

Sorry for the inclarity. In the paper you have reported the accuracy of some different models on the VOC 2007/2012 test sets. Is there some script I can run to get this accuracy after I've trained a model? Or could you send some general instruction to get these numbers?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yekeren/Cap2Det/issues/20#issuecomment-599024247, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6CPALEMAR2BAYMYYKLPWTRHMZO7ANCNFSM4LHAWIQA .

-- Thanks, best regards.

Keren

[achiyaj]

Hi,

Thanks for your quick reply. When running the script as you specified, I got a MAp score of 0.0 for all of the 20 VOC's relevant classes. After diving into it, I see that the feature map extracted contains only -nan values (specifically, I checked the value of features_to_crop at line 135 of this file. Trying to increase the batch size to more than 1 as suggested here causes the trainer.predict generator (line 345 here) to produce an immediate StopIteration exception.

Have you ever encounter this problem?

[yekeren]

Yes, though I did not dive into the details, I remember that I try to reduce the number of the training process and it works. You can also try to lower the learning rate. Do you use the script with multi-processes? I suggest you reduce the number of processes. Or, you can just use a single process, for example:

export CUDA_VISIBLE_DEVICES="1"

python "train/trainer_main.py" \

--alsologtostderr \

--pipeline_proto=${PIPELINE_CONFIG_PATH} \

--model_dir="${MODEL_DIR}" \

"log/${NAME}.train.log" 2>&1 &

Let me know if the solution works.

On Tue, Mar 17, 2020 at 4:38 AM achiyaj notifications@github.com wrote:

Hi,

Thanks for your quick reply. When running the script as you specified, I got a MAp score of 0.0 for all of the 20 VOC's relevant classes. After diving into it, I see that the feature map extracted contains only -nan values (specifically, I checked the value of features_to_crop at line 135 of this file https://github.com/yekeren/Cap2Det/blob/cap2det/models/utils.py. Trying to increase the batch size to more than 1 as suggested here https://github.com/tensorflow/models/issues/3870 causes the trainer.predict generator (line 345 here https://github.com/yekeren/Cap2Det/blob/cap2det/train/predict.py) to produce an immediate StopIteration exception.

Have you ever encounter this problem?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yekeren/Cap2Det/issues/20#issuecomment-599945680, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6CPAKMCHFVY5XP6TKFVG3RH4ZHZANCNFSM4LHAWIQA .

-- Thanks, best regards.

Keren

[achiyaj]

Lowering the number of workers worked for me (lowering the LR did not, though...).

Thanks very much for your help!

[chencjGene]

Hi achiyaj,

I came across the same error when i tried to reproduce the results (got nan values in extracted feature map). Could you share your modified codes? Thanks.

[achiyaj]

Hi,

This is the modified training script to train with a single training process and one online evaluation process. I did not experiment with more than one training processes.

train.sh.txt