search

zhihou7 / HOI-CL

Series of work (ECCV2020, CVPR2021, CVPR2021, ECCV2022) about Compositional Learning for Human-Object Interaction Exploration
https://sites.google.com/view/hoi-cl
MIT License
76 stars 11 forks source link

The dimension about the Av and Ao #4

Open bnuss98 opened 3 years ago

bnuss98 commented 3 years ago

Hello, it's very kind and excellent of you to share your work! The logic of this article is very clear, but as a green-hand, I am confused about the Nv and No in Av and Ao. The Nv and No are the numbers of verbs and objects about which dataset or things else? Looking forward to your answer~ ^_^

zhihou7 commented 3 years ago

Hi, N_v and N_o are the numbers of verb categories and object categories respectively. For example, on HICO-DET, N_v is 117 and N_o is 80. A_v is the verb-HOI co-occurrence matrix and A_o is the object-HOI co-occurrence matrix. You can see this line of the code: https://github.com/zhihou7/HOI-CL/blob/7d2b55947f8e5a75d32345508f70c929c8d4e8d1/lib/networks/tools.py#L88

For example, if the i HOI category is composed by verb j and object k, we have A_v[j][i] == 1 and A_o[k][I] == 1.

Regards,

bnuss98 commented 3 years ago

Thanks! I am also confused that lo is the label of an object, is it also an one-hot label? But what is the label dimension for a new object? y = (loAo)&(lvAv), so the lo should be 1*80, but the new object is not in HOI's obj_class.

bnuss98 commented 3 years ago

    Sorry to bother you again. In "Visual Compositional Learning for Human-Object Interaction Detection", according to my own understanding, I calculated according to equation (1) and (2), but it seems that only the composited HOIs which have appeared in the list of datasets' HOIs will be valid. For example, if the composited HOI is 'play basketball', it will be valid only when there is an annotation of 'play basketball' in the list of datasets' HOIs. But if there has 'play baseball' in the datasets' annotations, and there is no 'play basketball',  the composited HOI of 'play basketball' will be invalid, right? Could you please tell me about it in detail? Thank you very much !

------------------ 原始邮件 ------------------ 发件人: "zhihou7/HOI-CL" @.>; 发送时间: 2021年5月24日(星期一) 下午3:12 @.>; @.**@.>; 主题: Re: [zhihou7/HOI-CL] The dimension about the Av and Ao (#4)

Hi, N_v and N_o are the numbers of verb categories and object categories respectively. For example, on HICO-DET, N_v is 117 and N_o is 80. A_v is the verb-HOI co-occurrence matrix and A_o is the object-HOI co-occurrence matrix. You can see this line of the code: https://github.com/zhihou7/HOI-CL/blob/7d2b55947f8e5a75d32345508f70c929c8d4e8d1/lib/networks/tools.py#L88

For example, if the i HOI category is composed by verb j and object k, we have A_v[j][i] == 1 and A_o[k][I] == 1.

Regards,

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

zhihou7 commented 3 years ago

Thanks! I am also confused that lo is the label of an object, is it also an one-hot label? But what is the label dimension for a new object? y = (loAo)&(lvAv), so the lo should be 1*80, but the new object is not in HOI's obj_class.

Currently, we only consider the objects whose categories are the same or a subset of object categories of HOI dataset. Sorry for getting you latter. I did not receive the email notification.

zhihou7 commented 3 years ago

Yes, it is right. Thanks for your nice question!

However, you can simply set the co-occurrence matrix by hand to enable the composite HOI of 'play basketball'. Then, the model can learn 'play basketball' although there are no 'play basketball' instance in HICO-DET. Well, I guess this also answer your question about composing HOIs with novel types of objects. In fact, our next work also address your problem. we'll release it once it is accepted.

regards,

bnuss98 commented 3 years ago

thank you^-^

---Original--- From: @.> Date: Wed, May 26, 2021 22:15 PM To: @.>; Cc: @.**@.>; Subject: Re: [zhihou7/HOI-CL] The dimension about the Av and Ao (#4)

Yes, it is right. Thanks for your nice question!

However, you can simply set the co-occurrence matrix by hand to enable the composite HOI of 'play basketball'. Then, the model can learn 'play basketball' although there are no 'play basketball' instance in HICO-DET.

regards,

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

rouge012 commented 1 year ago

Yes, it is right. Thanks for your nice question!

However, you can simply set the co-occurrence matrix by hand to enable the composite HOI of 'play basketball'. Then, the model can learn 'play basketball' although there are no 'play basketball' instance in HICO-DET. Well, I guess this also answer your question about composing HOIs with novel types of objects. In fact, our next work also address your problem. we'll release it once it is accepted.

regards,

Hi! How can I set the co-occurrence matrix?

zhihou7 commented 1 year ago

hi @rouge012, the co-occurrence matrix $A\in R^{N_v \times N_o} $ is a two dimension matrix, where $N_v$ indicates the length of verb categories and $N_o$ indicate the length of object categories. We can initialize $A$ as a zero matrix. For each object, there are annotated verbs. We can set the corresponding position of the matrix $A$ as 1. For each example, if the apple is combinable with "eat", "cut" in the dataset, we set corresponding position of and in $A$ as 1.

Feel free to post if you have further questions

Regards,

rouge012 commented 1 year ago

hi @rouge012, the co-occurrence matrix $A\in R^{N_v \times N_o} $ is a two dimension matrix, where Nv indicates the length of verb categories and No indicate the length of object categories. We can initialize A as a zero matrix. For each object, there are annotated verbs. We can set the corresponding position of the matrix A as 1. For each example, if the apple is combinable with "eat", "cut" in the dataset, we set corresponding position of and in A as 1.

Feel free to post if you have further questions

Regards,

Thanks for the quick and detailed clarification, and I am wondering where I can find the code for setting the co-occurrence matrix. Thank you!

zhihou7 commented 1 year ago

Hi @rouge012,

I have posted in #25. If you need to generate a matrix by yourself, you can just open the three files "https://github.com/zhihou7/HOI-CL/blob/master/misc/hico_list_hoi.txt", "https://github.com/zhihou7/HOI-CL/blob/master/misc/hico_list_obj.txt", "https://github.com/zhihou7/HOI-CL/blob/master/misc/hico_list_vb.txt", and link them together according to the object name and verb name. I think it requires just several lines of code.