Evaluation on Novel Affordances

[khanh-md]

Hello,

Thank you for your work and for making the code available. I'm trying to perform inference with novel affordance labels, following the guidance provided in a previous issue thread. However, I'm encountering a problem where the affordance map remains unchanged regardless of the affordance labels I use.

Here's what I've done so far:

1. Created a new inference script based on test.py.
2. Modified the script to accept custom affordance labels.
3. Replaced the args.class_names with custom labels during inference.
4. Used the same trained model checkpoint as provided.

Despite these changes, the output affordance map remains the same for all input labels, as long as the index passed to inference is the same as the index of the original label in AGD20k. I've verified that the new labels are being passed to the model, but it seems the model is not responding to these different inputs.

Could you please provide some additional guidance on this issue? Specifically:

1. Is there a particular part of the model or inference process that needs modification when using novel affordance labels?
2. Are there any specific modifications needed in the models/ooal.py file to enable open-vocabulary inference?
3. Could you share a sample inference code snippet that demonstrates how to correctly use novel affordance labels with the trained model?

Any insights or example code would be greatly appreciated. Thank you for your time and assistance!

[Reagan1311]

Hi, what you have done to infer novel affordances is reasonable. As mentioned in the limitations of Readme, we suggest removing text prompt learning or combing with manually designed prompts for better novel affordance recognition. The trained checkpoint is evaluated in the seen/unseen object settings, but is limited for novel affordance setting. You can try to remove the text prompt learning and train a new model.