search

deepinsight / insightface

State-of-the-art 2D and 3D Face Analysis Project
https://insightface.ai
22.76k stars 5.34k forks source link

[MFR21] ICCV21 masked face recognition challenge discussion #1564

Open nttstar opened 3 years ago

nttstar commented 3 years ago

How to participate: https://github.com/deepinsight/insightface/tree/master/challenges/iccv21-mfr

Submission server: http://iccv21-mfr.com/

Workshop homepage: https://ibug.doc.ic.ac.uk/resources/masked-face-recognition-challenge-workshop-iccv-21/

Any issue can be left in this thread.

yBeOne commented 3 years ago

有谁测试过baseline 模型在IJBC上的准确率吗？我测试时一直报错。

vuvko commented 3 years ago

Hello and thank you for organizing the challenge this year! In your rules, you prohibit the use of pre-trained models and any external data, but there is no mention of how you'd enforce this rule. Will you demand training pipelines and cooperation to reproduce the results from top teams?

nttstar commented 3 years ago

@vuvko Yes, top-rank players should send their solutions to organizers after submission closed.

vuvko commented 3 years ago

Thank you for your quick response! Will you check the reproducibility of the training pipeline? If so, what will you do in case of a different result model (it can be hardware-dependent)?

nttstar commented 3 years ago

@vuvko We will strictly check the reproducibility. There should not be any hardware-dependent issue.

anxiangsir commented 3 years ago

有谁测试过baseline 模型在IJBC上的准确率吗？我测试时一直报错。

Please join the QQ group or WeChat group, let’s talk in detail.

manideep2510 commented 3 years ago

Hello Uploading the model on the submission page is very very slow. I have checked my internet connection and it is good with an upload speed greater than 25 MB/s. Upload to the submission portal is happening at <50 KB/s speed. Can someone please help me with this?

vuvko commented 3 years ago

Same problem here, I cannot upload the baseline iresnet50 model due to very slow upload speed.

manideep2510 commented 3 years ago

Looks like the speed is even low now, less than 10 KB/sec is being uploaded from my system. @nttstar Can you please look into this? Thanks in advance 🙂

nttstar commented 3 years ago

@manideep2510 @vuvko The submission server is located in China and it seems to have bandwidth problems for other countries. We will try to find a solution soon.

nutsam commented 3 years ago

Is there any preprocessing like normalization before load to model ? iccv21-mfr only mentioned that the input shape should equal to 3x112x112 (RGB order)

nttstar commented 3 years ago

@nutsam Normalization setting will be auto-detected in our onnx_helper.py, please check it.

vuvko commented 3 years ago

@nttstar is there any update on the uploading problem?

nttstar commented 3 years ago

@vuvko We're trying!

nqkhai1706 commented 3 years ago
Hello @nttstar , I just submitted a model and I got: " 2021-06-24 21:39:44 ms1m ∅ ∅ ∅ FAILED ∅ -

" The error message just said "load_onnx_failed". But as I checked my model using your onnx_helper.py, there is no error message. The output is just like in bellow, on my PC. So, I would like to ask what error message appears if my model's inference time larger than 10 ms?

Thank you,

file_: ms1m_groupface_resnet101.onnx use onnx-model: onnx/ms1m_groupface_resnet101/ms1m_groupface_resnet101.onnx input-shape: ['batch_size', 3, 112, 112] 0 Shape_0 1 Constant_1 2 Gather_2 3 Conv_3 4 PRelu_4 5 MaxPool_5 6 BatchNormalization_6 7 Conv_7 max time cost exceed, given 77.5860

nttstar commented 3 years ago

@nqkhai1706 Have you checked with onnxruntime==1.6 ?

nqkhai1706 commented 3 years ago

@nqkhai1706 Have you checked with onnxruntime==1.6 ?

No, my onnxruntime version: 1.8.0. May it different on 1.6? I will check it. Thank you,

nqkhai1706 commented 3 years ago

@nttstar I have checked again with onnxruntime==1.6, the result is same with onnxruntume==1.8.0. Do you have any other idea why my model got error on your side? Thank you,

anxiangsir commented 3 years ago

@nttstar I have checked again with onnxruntime==1.6, the result is same with onnxruntume==1.8.0. Do you have any other idea why my model got error on your side? Thank you,

This is our error:

>>> import onnxruntime
>>> session = onnxruntime.InferenceSession("ms1m_groupface_resnet101.onnx",  None)                                                                                                                      Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/face/miniconda3/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 206, in __init__
    self._create_inference_session(providers, provider_options)
  File "/home/face/miniconda3/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 226, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidProtobuf: [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from /mnt/data/models/1624534624858999668/ms1m_groupface_resnet101.onnx failed:Protobuf parsing failed.

The environment of our system is as follows:

Driver Version: 450.80.02
Cuda compilation tools, release 10.2, V10.2.89
onnx==1.8.0
onnx-simplifier==0.3.5
onnxoptimizer==0.2.5
onnxruntime-gpu==1.6.0
zhanglaplace commented 3 years ago

测试的时候能否知道当前的图片是否有mask

manideep2510 commented 3 years ago

@nttstar Even though I have not submitted any model today or yesterday. The portal is still showing that 

Today, you have submitted 3 times, 0 remaining.

Can you please look into this? Thanks in advance.

nttstar commented 3 years ago

@manideep2510 What is your username?

nttstar commented 3 years ago

测试的时候能否知道当前的图片是否有mask

@zhanglaplace 目前不能

manideep2510 commented 3 years ago

@manideep2510 What is your username?

My user name is sherlock.

anxiangsir commented 3 years ago

@nttstar Even though I have not submitted any model today or yesterday. The portal is still showing that

Today, you have submitted 3 times, 0 remaining.

Can you please look into this? Thanks in advance.

It's probably a bug about jet lag, and we'll fix that in no time.

JesseEisen commented 3 years ago

@nttstar Even though I have not submitted any model today or yesterday. The portal is still showing that

Today, you have submitted 3 times, 0 remaining.

Can you please look into this? Thanks in advance.

Hi, please provide some information related to the browser to help better debug. Like which browser you use and the version of the browser.

nttstar commented 3 years ago

@manideep2510 Hi, @JesseEisen is helping us build the evaluation system, please let him know your detailed issue.

manideep2510 commented 3 years ago

I am using Mozilla Firefox Version 89.02. The portal is now showing 

Today, you have submitted 8 times, -5 remaining.

But I am able to upload and make submissions.

nttstar commented 3 years ago

@manideep2510 Can you try Chrome?

manideep2510 commented 3 years ago

Yeah, exact same thing is happening on chrome as well. Now I am not able to upload or submit again.

nttstar commented 3 years ago

@vuvko @manideep2510 You can try the link of http://124.156.136.55/ , to see if it is faster.

vuvko commented 3 years ago

@nttstar The speed is not great (around 700Kb/s), but I could at least submit a model through that link. Thank you!

champagne828 commented 3 years ago

hello, could u plz update the qr code of wechat group? it has expired. thx!

nttstar commented 3 years ago

@champagne828 Updated

muncok commented 3 years ago

Would you please add an editable column in the "Submissions" page?

It would be very helpful to remember the configuration of each submission.

nqkhai1706 commented 3 years ago

Hello everybody, Sorry if I bother you. I am participating on MS1MV3 challenge. I am very surprised to see a lot of teams passed the baseline with a very large margin: ~89 on ALL column (as my knowledge, ArcFace with partial_fc is a SOTA method).

I have tuned a lot of tricks, but the score is still low. If you are in same situation with me, can we share a bit about needed tricks for the challenge? For that, we can learn together and improve our knowledge. Thank you and you have a nice day,

powermano commented 3 years ago

test script runtime error? Something wrong with my submit model?

John1231983 commented 3 years ago

What does it mean tab "unconstrained" (beside ms1m and glint360k). Does it mean we can use external data and more larger model?

anxiangsir commented 3 years ago

What does it mean tab "unconstrained" (beside ms1m and glint360k). Does it mean we can use external data and more larger model?

yes

John1231983 commented 3 years ago

I cannot find the information in web page. Could you announce about "unconstrained" tab?

nttstar commented 3 years ago

@John1231983 It is not official track of iccv21-mfr

ABD-01 commented 3 years ago

Hi, for ms1m track we are required to train onnx model on this dataset (MS1M-RetinaFace) right?

nttstar commented 3 years ago

@ABD-01 Right

John1231983 commented 3 years ago

Does the submission still available and allow to submit after deadline?

nttstar commented 3 years ago

Yes, and we will also add LFW/CFP/AGEDB/IJBC/MegaFace benchmarks.

John1231983 commented 2 years ago

@nttstar "1 October - Stop leader-board submission (11:59 PM Pacific Time)" Does it mean the score will not update on leaderboard but the user still can see score in the own submission page?

nttstar commented 2 years ago

@John1231983 Users can select which scores to be shown on the leaderboard after submission closed for 1 or 2 days.

John1231983 commented 2 years ago

Do you have plan to upload workshop video to youtube?

nttstar commented 2 years ago

@John1231983 yes later

John1231983 commented 2 years ago

@nttstar we are still waiting video conference.