search

ACANETS / NetML-Competition2020

BSD 2-Clause "Simplified" License
22 stars 11 forks source link

Details of the competition reg: #5

Open Narayan2407 opened 3 years ago

Narayan2407 commented 3 years ago

Hello again @onurbarut I am Narayan, a representative of Team01_5C_2020_SoCSE_KLETech. I have some doubts related to the competition results.

These lines are from the challenge overview page(hosted on EvalAI.org), "Three winners will be announced for each track. In order to win the prize, for example for non-vpn2016 toplevel track, a team must submit their results for both non-vpn2016_toplevel_dev and vpn2016_toplevel_challenge phases. The final score for each track will be calculated by averaging the 'overall' scores of both dev and challenge phases."

As per this, 3 winners will be announced for each of 7 tracks but as of now there is announcement of only 2 winners per track, so we wanna know a update on this matter. And also as said earlier, the final score will be calculated by averaging the overall scores of both phases(dev and challenge), for example, in Track 3 – Malware Detection using top-level annotations with CICIDS2017 dataset, we achieved score of 0.98573 in development phase and 0.98550 in challenge so averaging that would be around 0.985615 but the score shown on website is 0.97144. Can you please clarify about the above mentioned doubts? that would be grateful.

Regards, Narayan Kulkarni

onurbarut commented 3 years ago

Hello @Narayan2407 ,

Thank you for these questions and your interest in our NetML Competition 2020.

At the beginning of this competition, we considered to award 3 prizes in total, one for each dataset. For example, Track 1 and Track 2 results would be used to determine the winner of the first prize, Track 3 and Track 4 would be for the second prize, and Track 5-7 for the third prize. After a few weeks, we increased the amount of the prizes from 3 to 7, to the winner of each of 7 tracks. After this change, I guess, there was an issue while updating the eval.ai page. This ambiguity is now resolved, thanks to your attention.

To determine the final score of each track, the overall scores obtained from 'dev' and 'challenge' tracks are to be arithmetically averaged. You are right that previously announced numbers are not the exact averages. We have fixed that, but it doesn't change the rankings.

Thanks for your contributions!

Best, Onur

Narayan2407 commented 3 years ago

Hello @onurbarut Thank you for clearing it out but I guess there was a lot of ambiguity in the statements and unclear instructions in the challenge overview which made us all confused. And one last request, can you please update the results in which we have top 3 winners in each track just for the record so that we could know our position in other track namely, NetML Malware detection using top level annotation, that would be great.

Regards, Narayan

Narayan2407 commented 3 years ago

Hello again @onurbarut .

We know that prize money will be awarded to only the winners in each of 7 tracks but do the runner-ups and second runner-ups will at least receive acknowledgment or certificate of any kind so that it will be great for us to add that in our resume which will help us in the near future. I hope you understand my point sir.

Best, Narayan Kulkarni

onurbarut commented 3 years ago

Hi @Narayan2407 ,

Thank you for your interest and contributions to our NetML Competition 2020. We will check with our Sponsor, Intel Corp. if we can provide small prizes for the runner-ups or second runner-ups.

Please stay tuned!

Narayan2407 commented 3 years ago

Hello @onurbarut. That would be great sir, we are eagerly waiting for the update from your side. Thank You, Narayan Kulkarni