[BUG] Running into Failed to import torchrl C++ binaries for local install

Describe the bug

I tried installing both tensordict and torchrl from latest master and v0.5.0 and I am running into Failed to import torchrl C++ binaries

To Reproduce

I am installing TorchRL with the following script:

#!/usr/bin/env bash
set -Eeuo pipefail

eval "$(conda "shell.$(basename "${SHELL}")" hook)"

# cd to the script location
cd "$(dirname "${BASH_SOURCE[0]}")"

conda create -y --name quantized-offline-rl python=3.10
conda activate quantized-offline-rl

# pytorch
conda install pytorch==2.4.1 torchvision==0.19.1 pytorch-cuda=12.4 -c pytorch -c nvidia -y

# torchrl
pip install -U ninja
pip install cmake
cd submodules/tensordict
python setup.py develop
cd ../rl
python setup.py develop
cd ../..

I have tried with torchrl and tensordict checked out to v0.5.0 and latest master. In both cases if I run:

python -c 'from torchrl.envs.libs.gym import GymEnv'

I get:

/home/jorbik/Projects/project/submodules/rl/torchrl/data/replay_buffers/samplers.py:37: UserWarning: Failed to import torchrl C++ binaries. Some modules (eg, prioritized replay buffers) may not work with your installation. If you installed TorchRL from PyPI, please report the bug on TorchRL github. If you installed TorchRL locally and/or in development mode, check that you have all the required compiling packages.

If I replace the local installation steps with

pip install tensordict==0.5.0
pip install torchrl==0.5.0

the import works without any errors

Expected behavior

No C++ extension errors on import

System info

Describe the characteristic of your environment:

Describe how the library was installed (pip, source, ...) - in the bug description
Python version - 3.10
Versions of any other relevant libraries - pytorch 2.4.1, torchvision 0.19.1

import torchrl, numpy, sys
print(torchrl.__version__, numpy.__version__, sys.version, sys.platform)

/home/jorbik/Projects/project/submodules/rl/torchrl/data/replay_buffers/samplers.py:37: UserWarning: Failed to import torchrl C++ binaries. Some modules (eg, prioritized replay buffers) may not work with your installation. If you installed TorchRL from PyPI, please report the bug on TorchRL github. If you installed TorchRL locally and/or in development mode, check that you have all the required compiling packages.
  warnings.warn(EXTENSION_WARNING)
>>> print(torchrl.__version__, numpy.__version__, sys.version, sys.platform)
0.5.0+9565398 2.0.1 3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0] linux

Additional context

I have tried running the code outside of any git repositories as advised in the README

Checklist

[x] I have checked that there is no similar issue in the repo (required)
[x] I have read the documentation (required)
[x] I have provided a minimal working example to reproduce the bug (required)

pytorch / rl