Depth and RGB Frame misaligned from simulator rendering.

[jaypatravali]

Hello, I noticed a problem where running the simulator or level1/level2 the RGB and depth image at frame_i dont seem to be aligned. This gives us bad results because we rely on aligned image data. This link has the input RGB and depth stored as image files. Below is frame-4 for depth and RGB that goes to our vision modules. I have attached a zip file with RGB/depth images generated from the simulator for a complete scene. Also provided is the json file used to render this scene. I have tried many other scenes downloaded from eval3 submission webpage, which also show this problem. An offset +1 on depth frame seems to fix it.

JSON rendering this scene https://oregonstate.box.com/s/qis6xyc2vmmgavavb1jkdapwdr0yanvx

This is also cross-posted on the slack: https://machinecommonsense.slack.com/archives/CTR31V12M/p1612298520083300.

[bzinberg]

@erick-u That would be great, thanks!

[erick-u]

@bzinberg were you able to test and report results from the environment I provided?

[bzinberg]

Haven't gotten to it yet -- can't download the 1GB file while on video meetings. Will download later tonight.

[erick-u]

No worries, for some reason google drive may also zip the zip sometimes so it can take a while. If needed, we can look through other means of distributing the binaries. I am currently working on cleaning up the current fixes I have as they were very "poke this see what happens" due to the speculative nature of not being able to directly reproduce the issue and not everything is valid in the context of the fix we provided. I will let you know when that is ready and am looking forward to the results of these tests.

[bzinberg]

@erick-u, I ran the repro script and have attached the results. For posterity and reproducibility, could you please commit, push, and permalink to the source code of the binary I used before making further edits? No need to clean it up.

Results The problem behavior appears in the RGB images queried from AI2Thor but not the "Unity" versions. (Btw, took me a minute to realize the Unity versions were produced in the directory that contains the binary, which for me was not the same as the working directory where I ran the repro script.)

	initial	after1action	final
screenshot
rgb
unity

Text attachment: Player.log

[erick-u]

So some things to note:

1. The Player.log from your run and my run are completely identical except for hardware-related startup and your image synthesis encoding was actually quicker than mine.

2. The rgb images generated from AI2Thor seem to contain the development text on the bottom, where the bytes generated from the images within Unity that should be sent to AI2Thor do not contain them... Does AI2Thor not actually use the bytes sent and instead take a direct screenshot of the player? @deanwetherby @ThomasSchellenbergNextCentury

[erick-u]

@bzinberg

We have another possible solution! Outside of encoding, the AgentManager.captureScreen() method was generating images at the start and at the end of the frame, then storing them in the image list. This could possibly be the issue. With this change, we are waiting for the scene to load initially and then generating the single image.

Link to new executable: https://drive.google.com/drive/folders/1qo69tNLQ4nNzjhrgsoAZxk7pJEVwlkDl This is just the .x86_64 and _data folder that should replace the existing ones in the repro environment.

To outline the changes I am testing, I have created a draft pull request here: https://github.com/NextCenturyCorporation/ai2thor/pull/116

Please follow the same reproduction steps with the updated version contained in the drive link and report back whenever you get a moment with the same images and log. Thanks.

[erick-u]

Hi @bzinberg and @jaypatravali, will you have a moment this week to test the possible fix I provided above?

[bzinberg]

Hi @erick-u, very excited about the progress and definitely want to unblock the work. Aiming to try it out late-ish tonight.

[jaypatravali]

@erick-u Sorry I have been caught with eval3.5 work. Thanks for putting in all the work, I too will get it done tonight.

[erick-u]

@bzinberg and @jaypatravali Thank you so much for following up. Looking forward to any results we may have!

[bzinberg]

	initial	after1action	final
screenshot
rgb

I'm not seeing any "Unity" PNGs being created. Looked in both the working dir from which I invoked repro.py and the directory of the --unity_app_file_path (which is where it was last time).

Text attachment: Player.log

[erick-u]

@bzinberg I see in your log, the messages for the rgb_unity image generation seem to be missing. I have tweaked a setting and uploaded a new version of that zip to the same location: https://drive.google.com/drive/folders/1qo69tNLQ4nNzjhrgsoAZxk7pJEVwlkDl

Could you try this version? We should notice the text at the bottom right missing if this works correctly.

[bzinberg]

Still not seeing any "Unity" images. @erick-u, for posterity and my understanding, what was the setting you changed?

	initial	after1action	final
screenshot
rgb

[Edit: forgot to attach my Player.log]

[erick-u]

The main change was to generate the images within the same function as the depth images are generated except without any additional passes instead of just a "screenshot". I am unsure why there are still no Unity images but it does seem it is pulling from the same function now as you can see the lighting has changed, and the text in the bottom right has disappeared.

It seems this fix may not be valid for resolving this issue. Will have to investigate more.

[bzinberg]

Thanks @erick-u.

I'm requesting that you please commit, push and link to the source code of the binary you sent in https://github.com/NextCenturyCorporation/MCS/issues/228#issuecomment-797712745. I feel it's important to utilize the issue tracker in this way, otherwise it's likely we (or whoever next investigates this or a similar issue) will end up repeating unnecessary work in the future.

[deanwetherby]

Not able to address this ai2thor issue that is exhibited only on certain hardware.