megvii-research / MSPN

Multi-Stage Pose Network
334 stars 63 forks source link

About MPII test.json, lack of instances #16

Open JJBOY opened 4 years ago

JJBOY commented 4 years ago

In your test.json, it only has 7000+ instances, but we convert the offical h5 file to get a test.json, it has 11000+ instances. So how should we use your test.json? It seems that you changed the 'scale' and 'center' in your json compared to the offical's. If ture, can you tell us how you modify the 'scale' and 'center'?

JJBOY commented 4 years ago

The ground truth is actually not available for mpii test set. But it provided the 'center' and 'scale' of all persons we need to dection the keypoints, though the annotations may be not very precise. Whatever, in the test.h5 file provided by the official, there are 11000+ persons' approximate 'center' and 'scale' which means there are 11000+ persons we need to dectect their keypoints. However, in the test.json you provided, there are only 7000+ persons' bounding box. When using your test.json, we even couldn't generate the correct file buy using the offical code because the lack of instances. Could you provided you compelete test.json that you used to generate your final results submitted to the officical? Or could please provide the code to generate the test.json file? Thanks a lot!

------------------ 原始邮件 ------------------ 发件人: "fenglinglwb"notifications@github.com; 发送时间: 2019年10月14日(星期一) 下午3:43 收件人: "megvii-detection/MSPN"MSPN@noreply.github.com; 抄送: "Hello World"1135869534@qq.com; "Author"author@noreply.github.com; 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances(#16)

Actually, for MPII test split, we use a detection model to locate instances because the ground truth is not available for multi-person pose estimation. How could you get ground truth instances?

Regards, Wenbo

发件人: "qinxin" notifications@github.com 收件人: "megvii-detection/MSPN" MSPN@noreply.github.com 抄送: "Subscribed" subscribed@noreply.github.com 发送时间: 星期二, 2019年 10 月 08日 下午 12:46:19 主题: [megvii-detection/MSPN] About MPII test.json, lack of instances (#16)

In your test.json, it only has 7000+ instances, but we convert the offical h5 file to get a test.json, it has 11000+ instances. So how should we use your test.json? It seems that you changed the 'scale' and 'center' in your json compared to the offical's. If ture, can you tell us how you modify the 'scale' and 'center'?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, [ https://github.com/megvii-detection/MSPN/issues/16?email_source=notifications&email_token=AC4GMA2A54CNKQHERF3B7E3QNQGBXA5CNFSM4I6M33NKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HQHXUFQ | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC4GMA3REDFUBWKTY5SRQ6DQNQGBXANCNFSM4I6M33NA | mute the thread ] .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

fenglinglwb commented 4 years ago

Sorry for that I made a mistake. I only tested MSPN on the single person of MPII and there is no detection at all. Following the official procedure ( [ http://human-pose.mpi-inf.mpg.de/#evaluation | http://human-pose.mpi-inf.mpg.de/#evaluation ] ), I got 7000+ persons as provided test.json. And then I submited test result and obtained 92.6 PCKh.

Are you sure that 11000+ persons are all valid? Thanks.

发件人: "qinxin" notifications@github.com 收件人: "megvii-detection/MSPN" MSPN@noreply.github.com 抄送: "fenglinglwb" fenglinglwb@sjtu.edu.cn, "Comment" comment@noreply.github.com 发送时间: 星期一, 2019年 10 月 14日 下午 5:37:15 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances (#16)

The ground truth is actually not available for mpii test set. But it provided the 'center' and 'scale' of all persons we need to dection the keypoints, though the annotations may be not very precise. Whatever, in the test.h5 file provided by the official, there are 11000+ persons' approximate 'center' and 'scale' which means there are 11000+ persons we need to dectect their keypoints. However, in the test.json you provided, there are only 7000+ persons' bounding box. When using your test.json, we even couldn't generate the correct file buy using the offical code because the lack of instances. Could you provided you compelete test.json that you used to generate your final results submitted to the officical? Or could please provide the code to generate the test.json file? Thanks a lot!

------------------ 原始邮件 ------------------ 发件人: "fenglinglwb"notifications@github.com; 发送时间: 2019年10月14日(星期一) 下午3:43 收件人: "megvii-detection/MSPN"MSPN@noreply.github.com; 抄送: "Hello World"1135869534@qq.com; "Author"author@noreply.github.com; 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances(#16)

Actually, for MPII test split, we use a detection model to locate instances because the ground truth is not available for multi-person pose estimation. How could you get ground truth instances?

Regards, Wenbo

发件人: "qinxin" notifications@github.com 收件人: "megvii-detection/MSPN" MSPN@noreply.github.com 抄送: "Subscribed" subscribed@noreply.github.com 发送时间: 星期二, 2019年 10 月 08日 下午 12:46:19 主题: [megvii-detection/MSPN] About MPII test.json, lack of instances (#16)

In your test.json, it only has 7000+ instances, but we convert the offical h5 file to get a test.json, it has 11000+ instances. So how should we use your test.json? It seems that you changed the 'scale' and 'center' in your json compared to the offical's. If ture, can you tell us how you modify the 'scale' and 'center'?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, [ https://github.com/megvii-detection/MSPN/issues/16?email_source=notifications&email_token=AC4GMA2A54CNKQHERF3B7E3QNQGBXA5CNFSM4I6M33NKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HQHXUFQ | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC4GMA3REDFUBWKTY5SRQ6DQNQGBXANCNFSM4I6M33NA | mute the thread ] .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you commented. Reply to this email directly, [ https://github.com/megvii-detection/MSPN/issues/16?email_source=notifications&email_token=AC4GMA3G6IF3K777AUYQMBLQOQ4UXA5CNFSM4I6M33NKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBD5Q7Q#issuecomment-541579390 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC4GMAYEKLASZOUMDE2N7CTQOQ4UXANCNFSM4I6M33NA | unsubscribe ] .

JJBOY commented 4 years ago

Yes, I print the number of person to make sure that there are 11000+ people in the official's file.Another evidence is that if we use the test.json you provided which contains 7247 persons, we even couldn't get the final file to submit. Because we need to change the format to matlab style using the offical tools. The tools will check the instance number. When you use the test.json we convert by ourselves, we could get the right final file to submit.

------------------ 原始邮件 ------------------ 发件人: "fenglinglwb"notifications@github.com; 发送时间: 2019年10月14日(星期一) 晚上6:46 收件人: "megvii-detection/MSPN"MSPN@noreply.github.com; 抄送: "Hello World"1135869534@qq.com; "Author"author@noreply.github.com; 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances(#16)

Sorry for that I made a mistake. I only tested MSPN on the single person of MPII and there is no detection at all. Following the official procedure ( [ http://human-pose.mpi-inf.mpg.de/#evaluation | http://human-pose.mpi-inf.mpg.de/#evaluation ] ), I got 7000+ persons as provided test.json. And then I submited test result and obtained 92.6 PCKh.

Are you sure that 11000+ persons are all valid? Thanks.

发件人: "qinxin" notifications@github.com 收件人: "megvii-detection/MSPN" MSPN@noreply.github.com 抄送: "fenglinglwb" fenglinglwb@sjtu.edu.cn, "Comment" comment@noreply.github.com 发送时间: 星期一, 2019年 10 月 14日 下午 5:37:15 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances (#16)

The ground truth is actually not available for mpii test set. But it provided the 'center' and 'scale' of all persons we need to dection the keypoints, though the annotations may be not very precise. Whatever, in the test.h5 file provided by the official, there are 11000+ persons' approximate 'center' and 'scale' which means there are 11000+ persons we need to dectect their keypoints. However, in the test.json you provided, there are only 7000+ persons' bounding box. When using your test.json, we even couldn't generate the correct file buy using the offical code because the lack of instances. Could you provided you compelete test.json that you used to generate your final results submitted to the officical? Or could please provide the code to generate the test.json file? Thanks a lot!

------------------ 原始邮件 ------------------ 发件人: "fenglinglwb"notifications@github.com; 发送时间: 2019年10月14日(星期一) 下午3:43 收件人: "megvii-detection/MSPN"MSPN@noreply.github.com; 抄送: "Hello World"1135869534@qq.com; "Author"author@noreply.github.com; 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances(#16)

Actually, for MPII test split, we use a detection model to locate instances because the ground truth is not available for multi-person pose estimation. How could you get ground truth instances?

Regards, Wenbo

发件人: "qinxin" notifications@github.com 收件人: "megvii-detection/MSPN" MSPN@noreply.github.com 抄送: "Subscribed" subscribed@noreply.github.com 发送时间: 星期二, 2019年 10 月 08日 下午 12:46:19 主题: [megvii-detection/MSPN] About MPII test.json, lack of instances (#16)

In your test.json, it only has 7000+ instances, but we convert the offical h5 file to get a test.json, it has 11000+ instances. So how should we use your test.json? It seems that you changed the 'scale' and 'center' in your json compared to the offical's. If ture, can you tell us how you modify the 'scale' and 'center'?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, [ https://github.com/megvii-detection/MSPN/issues/16?email_source=notifications&email_token=AC4GMA2A54CNKQHERF3B7E3QNQGBXA5CNFSM4I6M33NKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HQHXUFQ | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC4GMA3REDFUBWKTY5SRQ6DQNQGBXANCNFSM4I6M33NA | mute the thread ] .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you commented. Reply to this email directly, [ https://github.com/megvii-detection/MSPN/issues/16?email_source=notifications&email_token=AC4GMA3G6IF3K777AUYQMBLQOQ4UXA5CNFSM4I6M33NKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBD5Q7Q#issuecomment-541579390 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC4GMAYEKLASZOUMDE2N7CTQOQ4UXANCNFSM4I6M33NA | unsubscribe ] .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

JJBOY commented 4 years ago

There are other people meet the same problem as me. I am really confused. What's more, we used the test.json we convert from the h5 file. We couldn't get the same results as the paper report in test set but we can get the same results in validation set. So, we look up for more details in the train.json and val.json. We found the 'center' and 'scale' in your train.json and val.json were modified. The bbox provided by the official is very imprecise but yours are really precise. I guess you may change the 'center' and 'scale' using the ground truth in train and val set , and bbox detected by dection model in test set. Is that right? https://github.com/microsoft/human-pose-estimation.pytorch/issues/106

------------------ 原始邮件 ------------------ 发件人: "fenglinglwb"notifications@github.com; 发送时间: 2019年10月14日(星期一) 晚上6:46 收件人: "megvii-detection/MSPN"MSPN@noreply.github.com; 抄送: "Hello World"1135869534@qq.com; "Author"author@noreply.github.com; 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances(#16)

Sorry for that I made a mistake. I only tested MSPN on the single person of MPII and there is no detection at all. Following the official procedure ( [ http://human-pose.mpi-inf.mpg.de/#evaluation | http://human-pose.mpi-inf.mpg.de/#evaluation ] ), I got 7000+ persons as provided test.json. And then I submited test result and obtained 92.6 PCKh.

Are you sure that 11000+ persons are all valid? Thanks.

发件人: "qinxin" notifications@github.com 收件人: "megvii-detection/MSPN" MSPN@noreply.github.com 抄送: "fenglinglwb" fenglinglwb@sjtu.edu.cn, "Comment" comment@noreply.github.com 发送时间: 星期一, 2019年 10 月 14日 下午 5:37:15 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances (#16)

The ground truth is actually not available for mpii test set. But it provided the 'center' and 'scale' of all persons we need to dection the keypoints, though the annotations may be not very precise. Whatever, in the test.h5 file provided by the official, there are 11000+ persons' approximate 'center' and 'scale' which means there are 11000+ persons we need to dectect their keypoints. However, in the test.json you provided, there are only 7000+ persons' bounding box. When using your test.json, we even couldn't generate the correct file buy using the offical code because the lack of instances. Could you provided you compelete test.json that you used to generate your final results submitted to the officical? Or could please provide the code to generate the test.json file? Thanks a lot!

------------------ 原始邮件 ------------------ 发件人: "fenglinglwb"notifications@github.com; 发送时间: 2019年10月14日(星期一) 下午3:43 收件人: "megvii-detection/MSPN"MSPN@noreply.github.com; 抄送: "Hello World"1135869534@qq.com; "Author"author@noreply.github.com; 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances(#16)

Actually, for MPII test split, we use a detection model to locate instances because the ground truth is not available for multi-person pose estimation. How could you get ground truth instances?

Regards, Wenbo

发件人: "qinxin" notifications@github.com 收件人: "megvii-detection/MSPN" MSPN@noreply.github.com 抄送: "Subscribed" subscribed@noreply.github.com 发送时间: 星期二, 2019年 10 月 08日 下午 12:46:19 主题: [megvii-detection/MSPN] About MPII test.json, lack of instances (#16)

In your test.json, it only has 7000+ instances, but we convert the offical h5 file to get a test.json, it has 11000+ instances. So how should we use your test.json? It seems that you changed the 'scale' and 'center' in your json compared to the offical's. If ture, can you tell us how you modify the 'scale' and 'center'?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, [ https://github.com/megvii-detection/MSPN/issues/16?email_source=notifications&email_token=AC4GMA2A54CNKQHERF3B7E3QNQGBXA5CNFSM4I6M33NKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HQHXUFQ | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC4GMA3REDFUBWKTY5SRQ6DQNQGBXANCNFSM4I6M33NA | mute the thread ] .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you commented. Reply to this email directly, [ https://github.com/megvii-detection/MSPN/issues/16?email_source=notifications&email_token=AC4GMA3G6IF3K777AUYQMBLQOQ4UXA5CNFSM4I6M33NKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBD5Q7Q#issuecomment-541579390 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC4GMAYEKLASZOUMDE2N7CTQOQ4UXANCNFSM4I6M33NA | unsubscribe ] .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

fenglinglwb commented 4 years ago

We did not introduce detection at all. As for the center and scale, we referred HRNet to take some modification. You could find it in our codes.

发件人: "qinxin" notifications@github.com 收件人: "megvii-detection/MSPN" MSPN@noreply.github.com 抄送: "fenglinglwb" fenglinglwb@sjtu.edu.cn, "Comment" comment@noreply.github.com 发送时间: 星期一, 2019年 10 月 14日 下午 9:07:45 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances (#16)

There are other people meet the same problem as me. I am really confused. What's more, we used the test.json we convert from the h5 file. We couldn't get the same results as the paper report in test set but we can get the same results in validation set. So, we look up for more details in the train.json and val.json. We found the 'center' and 'scale' in your train.json and val.json were modified. The bbox provided by the official is very imprecise but yours are really precise. I guess you may change the 'center' and 'scale' using the ground truth in train and val set , and bbox detected by dection model in test set. Is that right? https://github.com/microsoft/human-pose-estimation.pytorch/issues/106

------------------ 原始邮件 ------------------ 发件人: "fenglinglwb"notifications@github.com; 发送时间: 2019年10月14日(星期一) 晚上6:46 收件人: "megvii-detection/MSPN"MSPN@noreply.github.com; 抄送: "Hello World"1135869534@qq.com; "Author"author@noreply.github.com; 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances(#16)

Sorry for that I made a mistake. I only tested MSPN on the single person of MPII and there is no detection at all. Following the official procedure ( [ http://human-pose.mpi-inf.mpg.de/#evaluation | http://human-pose.mpi-inf.mpg.de/#evaluation ] ), I got 7000+ persons as provided test.json. And then I submited test result and obtained 92.6 PCKh.

Are you sure that 11000+ persons are all valid? Thanks.

发件人: "qinxin" notifications@github.com 收件人: "megvii-detection/MSPN" MSPN@noreply.github.com 抄送: "fenglinglwb" fenglinglwb@sjtu.edu.cn, "Comment" comment@noreply.github.com 发送时间: 星期一, 2019年 10 月 14日 下午 5:37:15 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances (#16)

The ground truth is actually not available for mpii test set. But it provided the 'center' and 'scale' of all persons we need to dection the keypoints, though the annotations may be not very precise. Whatever, in the test.h5 file provided by the official, there are 11000+ persons' approximate 'center' and 'scale' which means there are 11000+ persons we need to dectect their keypoints. However, in the test.json you provided, there are only 7000+ persons' bounding box. When using your test.json, we even couldn't generate the correct file buy using the offical code because the lack of instances. Could you provided you compelete test.json that you used to generate your final results submitted to the officical? Or could please provide the code to generate the test.json file? Thanks a lot!

------------------ 原始邮件 ------------------ 发件人: "fenglinglwb"notifications@github.com; 发送时间: 2019年10月14日(星期一) 下午3:43 收件人: "megvii-detection/MSPN"MSPN@noreply.github.com; 抄送: "Hello World"1135869534@qq.com; "Author"author@noreply.github.com; 主题: Re: [megvii-detection/MSPN] About MPII test.json, lack of instances(#16)

Actually, for MPII test split, we use a detection model to locate instances because the ground truth is not available for multi-person pose estimation. How could you get ground truth instances?

Regards, Wenbo

发件人: "qinxin" notifications@github.com 收件人: "megvii-detection/MSPN" MSPN@noreply.github.com 抄送: "Subscribed" subscribed@noreply.github.com 发送时间: 星期二, 2019年 10 月 08日 下午 12:46:19 主题: [megvii-detection/MSPN] About MPII test.json, lack of instances (#16)

In your test.json, it only has 7000+ instances, but we convert the offical h5 file to get a test.json, it has 11000+ instances. So how should we use your test.json? It seems that you changed the 'scale' and 'center' in your json compared to the offical's. If ture, can you tell us how you modify the 'scale' and 'center'?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, [ https://github.com/megvii-detection/MSPN/issues/16?email_source=notifications&email_token=AC4GMA2A54CNKQHERF3B7E3QNQGBXA5CNFSM4I6M33NKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HQHXUFQ | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC4GMA3REDFUBWKTY5SRQ6DQNQGBXANCNFSM4I6M33NA | mute the thread ] .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you commented. Reply to this email directly, [ https://github.com/megvii-detection/MSPN/issues/16?email_source=notifications&email_token=AC4GMA3G6IF3K777AUYQMBLQOQ4UXA5CNFSM4I6M33NKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBD5Q7Q#issuecomment-541579390 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC4GMAYEKLASZOUMDE2N7CTQOQ4UXANCNFSM4I6M33NA | unsubscribe ] .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you commented. Reply to this email directly, [ https://github.com/megvii-detection/MSPN/issues/16?email_source=notifications&email_token=AC4GMAZU5U26VW32G3CXGPDQORVKDA5CNFSM4I6M33NKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBES5VA#issuecomment-541667028 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC4GMA4CFJZY5RUCETCZWFLQORVKDANCNFSM4I6M33NA | unsubscribe ] .

fenglinglwb commented 4 years ago

FYI, I used my own script rather than official tools to convert result to matlab file. You can take a try.

JJBOY commented 4 years ago

could you make the script aviaible? It will be very helpful to make the people who use your codebase get the final test results on mpii test set.

fenglinglwb commented 4 years ago

Sorry for that this script is not available since I am no longer in Megvii now. It is not difficult, so you can finish it by yourself.

JJBOY commented 4 years ago

ok, thanks for your patience to reply. I will have a try.