evaluation

[Muzi010]

我有几个关于评价指标的问题想问一下： 1.请问作者是否用BIPDE的训练集测试过指标，如果测试过，可否说一下，想验证一下我的测试结果。 2.关于NMS的参数问题，不知道这些参数为何这么设置，不知作者BIPDE和UDED进行评价的时候用的参数是什么？（(tmp_edge,O,1,5,1.03,8)？）还有就是参数的不同对结果的影响大不大？ 3.关于evaluation过程，有没有什么需要特别注意的？

[Muzi010]

I have a few questions about the evaluation metrics to ask:

1. may I ask the authors if they have tested the metrics with the training set of BIPDE, if so, can you say something, I would like to verify my test results.
2. about the parameter issue of NMS, I don't know why these parameters are set so, I don't know what parameters the authors BIPDE and UDED used for evaluation? ((tmp_edge,O,1,5,1.03,8)?) And how much does the difference in parameters affect the results?
3. Is there anything about the evaluation process that needs special attention?

[xavysp]

Hi, thanks for using TEED.

I have a few questions about the evaluation metrics to ask:

1. may I ask the authors if they have tested the metrics with the training set of BIPDE, if so, can you say something, I would like to verify my test results.

We are evaluating with the Test set of BIPED, we will report this result soon, please wait. But it does not reach the SOTA result but it beats in downstream tasks.

2. about the parameter issue of NMS, I don't know why these parameters are set so, I don't know what parameters the authors BIPDE and UDED used for evaluation? ((tmp_edge,O,1,5,1.03,8)?) And how much does the difference in parameters affect the results?

There is a light difference since, as you know, if we use the same setting as for BSDS even for DexiNed, you will find quite different ODS. Because, the edge predicted from TEED is thinner than DexiNed and tiniest than other models like BSDS, UAED EDTER and so on. So, We should leave it as (tmp_edge, O, 1, 5, 1.04, 8).

3. Is there anything about the evaluation process that needs special attention? No, just the previous answer.

Hope I help you.

Xavier

[Muzi010]

Thank you very much for your answer, it feels like the code in the evaluation metrics section is running very slowly. It is even slower than the calculation of the BSDS500 dataset 5-layer truth overlay, didn't figure out what is the reason. Have you updated the evaluation metrics and if so can you share.

[xavysp]

You know, bsds images are small, and most of uded are bigger. And uded GTs have more annotation so it will take time. I suppose, in a normal PC, it will take 3 hours.

[Muzi010]

I'm not sure what my fault is, the BIPED test set I ran for over 10 hours on top of a 4090ti

[xavysp]

I'm not sure what my fault is, the BIPED test set I ran for over 10 hours on top of a 4090ti

Yo mean the evaluation process, in matlab?

[Muzi010]

Yes, using matlab, the model I trained is not perfect, the ted model I tested has a metric of 0.857 on the biped test set.

[xavysp]

And what about Uded dataset?

[xavysp]

I'm not sure what my fault is, the BIPED test set I ran for over 10 hours on top of a 4090ti

Its ok, the code in Matlab is not using your gpu to run your edge evaluator.

[Muzi010]

Thank you very much for your answer, there is another question that has been bothering me for days now and I never know what the problem is!I replaced the TED model with my own and there is only one output from the model, so I did the BCEloss. not sure why the training came up with a black line. Then I took out only the last output of the TED model to do the loss, also using only BCEloss, and the result came out still with black lines, don't understand why this happens

[Muzi010]

I know why the above happens, the sigmoid and image_normalisation functions were performed during testing. It feels so complicated, does it have a big impact on the metrics to output the test results after this processing and directly output the test results?

[xavysp]

I know why the above happens, the sigmoid and image_normalisation functions were performed during testing. It feels so complicated, does it have a big impact on the metrics to output the test results after this processing and directly output the test results?

Sorry, I don't follow you. Maybe we can have a zoom meeting, just emil me. Or show me the training and testing post procesing.

Cheers

[ZhouJin0854]

我有几个关于评价指标的问题想问一下： 1.请问作者是否用BIPDE的训练集测试过指标，如果测试过，可否说一下，想验证一下我的测试结果。 2.关于NMS的参数问题，不知道这些参数为何这么设置，不知作者BIPDE和UDED进行评价的时候用的参数是什么？（(tmp_edge,O,1,5,1.03,8)？）还有就是参数的不同对结果的影响大不大？ 3.关于evaluation过程，有没有什么需要特别注意的？

兄弟，可以给个Wechat或者QQ号交流一下吗？

[Muzi010]

我有几个关于评价指标的问题想问一下： 1.请问作者是否用BIPDE的训练集测试过指标，如果测试过，可否说一下，想验证一下我的测试结果。 2.关于NMS的参数问题，不知道这些参数为何这么设置，不知作者BIPDE和UDED进行评价的时候用的参数是什么？（(tmp_edge,O,1,5,1.03,8)？）还有就是参数的不同对结果的影响大不大？ 3.关于evaluation过程，有没有什么需要特别注意的？

兄弟，可以给个Wechat或者QQ号交流一下吗？

2635339989

[KeyWu]

我有几个关于评价指标的问题想问一下： 1.请问作者是否用BIPDE的训练集测试过指标，如果测试过，可否说一下，想验证一下我的测试结果。 2.关于NMS的参数问题，不知道这些参数为何这么设置，不知作者BIPDE和UDED进行评价的时候用的参数是什么？（(tmp_edge,O,1,5,1.03,8)？）还有就是参数的不同对结果的影响大不大？ 3.关于evaluation过程，有没有什么需要特别注意的？

你好，我想问一下这篇文章你跑通了吗？

[ZhouJin0854]

通了

On Sat, Sep 21, 2024 at 15:46 KeyWu @.***> wrote:

我有几个关于评价指标的问题想问一下： 1.请问作者是否用BIPDE的训练集测试过指标，如果测试过，可否说一下，想验证一下我的测试结果。 2.关于NMS的参数问题，不知道这些参数为何这么设置，不知作者BIPDE和UDED进行评价的时候用的参数是什么？（(tmp_edge,O,1,5,1.03,8)？）还有就是参数的不同对结果的影响大不大？ 3.关于evaluation过程，有没有什么需要特别注意的？

你好，我想问一下这篇文章你跑通了吗？

— Reply to this email directly, view it on GitHub https://github.com/xavysp/TEED/issues/21#issuecomment-2365047769, or unsubscribe https://github.com/notifications/unsubscribe-auth/A75HPBCOI5JCI54DHQHUALTZXUP4PAVCNFSM6AAAAABHMG56DGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRVGA2DONZWHE . You are receiving this because you commented.Message ID: @.***>