Evaluate PASCAL VOC 2012

[InstantWindy]

Hi! I want to test the segmentation result on the VOC 2012 test set, but I need to upload the test results to the official website. I don't know what the format of the uploaded file is.So how do you evaluate the VOC test set? How did you test the segmentation results of VOC2012 test data set on PASCAL VOC official website? Looking forward to your reply,Thanks!

[qiulesun]

@InstantWindy The voc server gives the directory structure of submission and also provides an example submission structure for the VOC2010 challenge.

[InstantWindy]

Yeah, I know,but when I upload the folder, it goes wrong ，The foler format is "\results\VOC2012\Segmentation\comp6_test_cls" . Evaluation errors：Error while evaluating challenge 6. Results were not generated properly. Have your ever had the same problem? Thanks!

[qiulesun]

I didn't meet that error. Your folder fomat looks correct. The directory structure of submission is compressed by a single archive file (xxx.tar.gz). The evaluation log is as follow.

Evaluation log Checking submission Submission file: /var/voc_challenges/VOC2012/submissions/24758/input/results-encnet-resnet50.tar.gz Working directory: /var/voc_challenges/VOC2012/submissions/24758/input Extracting submission Submission MD5 sum is: 5fe0fbb34b011132b502835bd41fe073 Extraction complete. Checking submission. Challenge 1. No results present. Challenge 2. No results present. Challenge 3. No results present. Challenge 4. No results present. Challenge 5. Not submitted (looking for folder results/VOC2012/Segmentation/comp5_test_cls) Challenge 6.+++ Submitted. All required images are present. Challenge 7. Not submitted (looking for results/VOC2012/Layout/comp7_layout_test.xml) Challenge 8. Not submitted (looking for results/VOC2012/Layout/comp8_layout_test.xml) Challenge 9. No results present. Challenge 10. No results present. Challenge 11. No results present. Challenge 12. No results present. Submission check complete. Task is ready for evaluation.

[RajatGarg45]

After running eval.py and vis.py, two folders are formed raw_segmentation_results and segmentation_results but both are empty. I have trained the model successfully on VOC dataset with 30000 steps. But eval.py and vis.py gets stuck at Waiting for new checkpoint:

How do I resolve this?

[zhanghang1989]

use test.py

[SafaMessaoud]

I am having the same error message when submitting to the server. Were you able to solve it? Thanks!

[SteveJunGao]

I managed to submit my result to the server, but I have been kept in the status: "Evaluation is in progress" for two days, is it typical to be so slow? or some problem in my side?

[StacyYang]

just ping the dataset organizer

[SteveJunGao]

Thanks a lot!

[sreesindhu-sabbineni]

I saw the directory structure but somehow couldn't understand, where do we submit the model we trained?? I saw the example in PASCAL VOC 2010 but looks like they are generated results for each class. Can someone help me by giving the details of submission. I need to test one model for classification and one model for segmentation. I have two models in .h5 files and how should I submit this model for evaluation?

[Seohyeong]

@sreesindhu-sabbineni You don't submit the trained model. Segmentation task for example, You need to generate segmentation masks one for each test image running inference on your model and submit .png format masks zipped into "\results\VOC2012\Segmentation\comp6_test_cls\" this form. More details are found here (http://host.robots.ox.ac.uk/pascal/VOC/voc2012/htmldoc/devkit_doc.html#sec:exampleseg)

[zhanghang1989]

The images can be generated by running test.py. For example:

python test.py --dataset PContext --model-zoo Encnet_ResNet50_PContext

[drcege]

@qiulesun @SteveJunGao
Hey, guys, have your results submitted to the PASCAL server been evaluated?

My submission has been kept ”Evaluation in progress“ for almost two days. I emailed yusuf@robots.ox.ac.uk, but got no reply.

[drcege]

@andrewsonga Yes, my submission was finally evaluated. Sorry it’s been too long; I don’t remember how long it took. Maybe a few days or a week?

He didn't seem to reply. I think maybe there are too many submissions in the waiting queue. You can't help but wait patiently.

[vail-xjl]

I am having the same error message when submitting to the server. Were you able to solve it? Thanks!

Hello, may I ask about the issue you encountered on the Pascal VOC dataset's test set at the time? [Error while evaluating challenge 6 .Results were not generated properly.] Have you solved it

[wangjialipp]

是的，我知道，但是当我上传文件夹时，它出错了，文件夹格式是“\results\VOC2012\Segmentation\comp6_test_cls”。评估错误：评估挑战6时出错。结果未正确生成。你也 遇遇到同样的问题吗？谢谢！

请问您的问题解决了吗

[wangjialipp]

我在提交到服务器时收到相同的错误消息。你能解决吗？谢谢！

您好，请问您当时在 Pascal VOC 数据集的测试集中遇到了问题吗？[评估挑战 6 时错误。结果正确未生成。] 您解决了吗？

请问您的问题解决了吗，我也遇到了同样的问题

[wangjialipp]

我在提交到服务器时收到相同的错误消息。你能解决吗？谢谢！

您好，请问您当时在Pascal VOC数据集的测试集上遇到的问题吗？[评估挑战 6 时出错。结果未正确生成。] 你解决了吗

您好，我遇到了同样的错误，请问您解决了吗