Possible bugs in `get_state_changes`

[Ji4chenLi]

Hi @aishwaryap,

Thank you for releasing the dataset. It seems that there is a bug in the get_state_changes function: https://github.com/alexa/teach/blob/5554f02f55c22abfe5c2a749dbb24c13377726c8/src/teach/utils.py#L92

I believe it should be agent_final = final_state["agents"][idx] instead. As a result, the state differences of the agents are empty in all teach-dataset/images/$SPLIT/$REPLAYED_CODE/statediff.*.json files.

Thanks, Jiachen

[aishwaryap]

Hi @Ji4chenLi

Thanks for catching this! This is definitely a bug that we have somehow missed. I believe it should not affect any of the results in the TEACh paper as we do not actually use agents' positions or any other info in evaluation (we only use object states).

Currently I am thinking of leaving this bug as is to minimize potential disruptions to the ongoing public benchmark phase of the SimBot challenge. We will update the codebase with a fix after the challenge is over (ie sometime in May).

In the meantime, if you are interested in creating a new task whose evaluation requires agent info, you will need to fix it locally.

Thanks! Aishwarya

[Ji4chenLi]

Hi @aishwaryap,

Thank you so much for your timely reply! This plan makes sense to me.

Currently I am working on accessing the last_event of the controller at each step. Seems that the only way I could achieve this is to rerun the episode replay as the state_diff can no longer be leveraged.

Thanks, Jiachen

[aishwaryap]

Hi @Ji4chenLi

If you do need the output of last_event at each time step you will need to rerun replay. We actually chose to create the state_diff because otherwise last_event can get very large to store (although it might be reasonable if you use some binary format rather than json).

Best, Aishwarya