search

facebookresearch / habitat-sim

A flexible, high-performance 3D simulator for Embodied AI research.
https://aihabitat.org/
MIT License
2.54k stars 414 forks source link

A100 Render Problem: Black Squares #2310

Closed maytusp closed 5 months ago

maytusp commented 7 months ago

Habitat-Sim version

v0.3.0

Habitat is under active development, and we advise users to restrict themselves to stable releases. Are you using the latest release version of Habitat-Sim? Your question may already be addressed in the latest version. We may also not be able to help with problems in earlier versions because they sometimes lack the more verbose logging needed for debugging.

Main branch contains 'bleeding edge' code and should be used at your own risk.

Docs and Tutorials

Did you read the docs? https://aihabitat.org/docs/habitat-sim/ Yes

Did you check out the tutorials? https://aihabitat.org/tutorial/2020/ Yes

Perhaps your question is answered there. If not, carry on! Yes

❓ Questions and Help

I’ve found an issue on all A100 GPUs. It has black squares on every frame that the simulation render.

This problem never occurred with V100 GPUs, so I guess it’s the A100 specific problem. I also found that the OpenGL software versions are the same for both V100 and A100. I’m not sure how to fix this issue. It seems I must change rendering setting.

image

image

aclegg3 commented 6 months ago

hey @maytusp,

While I have not observed this issue personally, it seems to me likely that driver support is culprit.

I don't really know how to debug this further except to try re-installing/updating the gl and cuda drivers on the system.

@0mdc have we seen this anywhere? I think we've used A100s without issue. :thinking:

0mdc commented 6 months ago

@aclegg3 I don't believe that we've seen this issue.

This may an issue that's specific to A100s. I searched a bit around, and the GPU/driver seem to present this artifact in other software. E.g.:

I would also suggest reinstalling the latest drivers.

maytusp commented 5 months ago

Hi, Thank you for helping. The problem is solved by updating CUDA to be version 12.2.

0mdc commented 5 months ago

This is really helpful! Thanks for letting us know.

cuijiaxun commented 4 months ago

@maytusp Have you tried any 11.{7, 8} cuda version?