search

isaac-sim / IsaacLab

Unified framework for robot learning built on NVIDIA Isaac Sim
https://isaac-sim.github.io/IsaacLab
Other
2.15k stars 878 forks source link

[Bug Report] When headless is True and livestream is enabled, the wrong simulation experience file is loaded #381

Closed ozhanozen closed 5 months ago

ozhanozen commented 6 months ago

Description

If headless is True and livestream is enabled, the wrong simulation experience file is loaded. Normally orbit.python.headless.kit should be loaded (according to the AppLauncher descriptions), but orbit.python.kit is the one being loaded. This ultimately prevents livestream methods starting correctly, which is probably one of the reasons for the issue #268.

Steps to reproduce

Run random_ageny.py with livestream enabled (e.g., livestream=3) on any environment and observe the loaded simulation experience file.

Possible fix

I believe the problem emerges from the conditional statement in L444 of the app_launcer.py: https://github.com/NVIDIA-Omniverse/orbit/blob/7cc56c3efec11b91a7364c7e49001a48088dd389/source/extensions/omni.isaac.orbit/omni/isaac/orbit/app/app_launcher.py#L444-L447

If I am not mistaken, the condition on L444 should be something like the following: if self._headless and self._livestream >=1:

Implementing this change allows loading the correct simulation experience file, i.e., orbit.python.headless.kit, and as far as I can test using WebRTC, livestream starts. I can open a PR for this fix if desired.

However, although WebRTC-based livestream starts somehow, there is still something wrong with it. It looks like not everything is loaded correctly in the correct place. This needs further investigation.

System Info

Checklist

Acceptance Criteria

Mayankm96 commented 6 months ago

Interesting. Is there a reason why the experience file orbit.python.kit shouldn't be used for live-streaming? It enables things related to the UI (such as arranging the window layout) that should be needed. From what I recall, having a live-stream UI or local GUI should have the same experience.

However, live-streaming not working with this experience file is definitely an issue.

@hhansen-bdai, do you have some more insights here since you are using livestreaming a lot more in your workflows?

ozhanozen commented 6 months ago

Interesting. Is there a reason why the experience file orbit.python.kit shouldn't be used for live-streaming? It enables things related to the UI (such as arranging the window layout) that should be needed. From what I recall, having a live-stream UI or local GUI should have the same experience.

There are the following two lines that create a problem while using orbit.python.kit with livestream: https://github.com/NVIDIA-Omniverse/orbit/blob/7cc56c3efec11b91a7364c7e49001a48088dd389/source/apps/orbit.python.kit#L36-L37

These apparently load the native stream components before AppLauncher even checks the livestream conditions as below: https://github.com/NVIDIA-Omniverse/orbit/blob/7cc56c3efec11b91a7364c7e49001a48088dd389/source/extensions/omni.isaac.orbit/omni/isaac/orbit/app/app_launcher.py#L514-L526 In other words, components omni.kit.livestream.native and omni.services.streaming.manager are loaded regardless of which livestream is selected. This is a problem if any livestream mode other native one is selected, as they create conflict and secondary livestream methods (e.g., webrtc) do not work anymore.

I have tried commenting out L36-L37 inside orbit.python.kit, and it actually worked better than orbit.python.headless.kit; everything is in the correct place as desired :). Would commenting out these lines (and maybe getting rid of orbit.python.headless.kit if not needed) be an appropriate fix? I cannot test the native livestreaming currently, so it is hard to say if I am breaking something else.

Btw, this issue probably emerges from a bug in the Isaac Sim-related script, as reported by a user in Nvidia forums. It seems like python.sh of Isaac Sim might be trying to run the omni.isaac.sim.python.kit (rather than the headless version), creating livestream fails due to the same two lines that enable native streaming. Does the bug on the Isaac Sim side needs to be fixed by Isaac Sim team or is there something we can do about it? (i.e., relevant for running non-orbit scripts such as livestream.py within the orbit container)

Mayankm96 commented 6 months ago

Hi @ozhanozen

I have tried commenting out L36-L37 inside orbit.python.kit, and it actually worked better than orbit.python.headless.kit; everything is in the correct place as desired :). Would commenting out these lines (and maybe getting rid of orbit.python.headless.kit if not needed) be an appropriate fix? I cannot test the native livestreaming currently, so it is hard to say if I am breaking something else.

I think you are right here. Makes sense to remove it to simplify the file as well :)

Btw, this issue probably emerges from a bug in the Isaac Sim-related script, as reported by a user in Nvidia forums. It seems like python.sh of Isaac Sim might be trying to run the omni.isaac.sim.python.kit

Yes, by default, the SimulationApp in Isaac Sim always loads the experience file omni.isaac.python.kit. I think before, we used to check if headless flag is enabled in all our standalone scripts and change the experience file parameter to be the headless one.

At least in Orbit, we moved all these checks inside the AppLauncher to avoid users to bother with it. At the same time, you can still choose which experience file to load from command-line by mentioning the file name or its path.

For instance:

./orbit.sh -p source/standalone/demos/quadrupeds.py --experience omni.isaac.sim.python.kit
ozhanozen commented 6 months ago

Hi @Mayankm96

It seems like orbit.python.headless.kit is still needed when running headless without livestream, e.g., when using training scripts. Therefore, my suggestion would be to:

  1. Remove the following lines from orbit.python.kit: https://github.com/NVIDIA-Omniverse/orbit/blob/7cc56c3efec11b91a7364c7e49001a48088dd389/source/apps/orbit.python.kit#L35-L37
  2. Update the following lines in the documentation: https://github.com/NVIDIA-Omniverse/orbit/blob/7cc56c3efec11b91a7364c7e49001a48088dd389/source/extensions/omni.isaac.orbit/omni/isaac/orbit/app/app_launcher.py#L162-L165 such as the following:

    If provided as an empty string, the experience file is determined based on the headless flag and livestream variable:

    • If headless is True and livestream is 0, the experience file is set to orbit.python.headless.kit.
    • If headless is False or livestream > 0, the experience file is set to orbit.python.kit.

Would you agree with these? I can open a PR if you like.

Dhoeller19 commented 5 months ago

Hi Özhan, This should be fixed now.

ozhanozen commented 5 months ago

Thanks @Dhoeller19, i can confirm it works!