AttributeError: '_OpNamespace' 'fbgemm' object has no attribute 'jagged_2d_to_dense'

[zhuhaozhe]

We are working on CPU-only build env. When building latest Pytorch from src and install fbgemm_nightly via pip

pip install fbgemm-gpu-nightly-cpu
# or pip install fbgemm-gpu --index-url https://download.pytorch.org/whl/nightly/cpu

We will met

/home/haozhe/miniconda3/envs/ipex/lib/python3.9/site-packages/fbgemm_gpu/fbgemm_gpu_py.so: undefined symbol: _ZNK5torch8autograd4Node4nameEv
Traceback (most recent call last):
  File "/home/haozhe/ipex/test_import.py", line 2, in <module>
    import fbgemm_gpu
  File "/home/haozhe/miniconda3/envs/ipex/lib/python3.9/site-packages/fbgemm_gpu/__init__.py", line 23, in <module>
    from . import _fbgemm_gpu_docs, sparse_ops  # noqa: F401, E402  # noqa: F401, E402
  File "/home/haozhe/miniconda3/envs/ipex/lib/python3.9/site-packages/fbgemm_gpu/_fbgemm_gpu_docs.py", line 19, in <module>
    torch.ops.fbgemm.jagged_2d_to_dense,
  File "/home/haozhe/ipex/pytorch/torch/_ops.py", line 927, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' 'fbgemm' object has no attribute 'jagged_2d_to_dense'

Is there a way to build latest Pytorch from src while install fbgemm_gpu via pip can work?

Install pytorch nightly and fbgemm_gpu nightly via pip can work.

pip install torch fbgemm_gpu --index-url https://download.pytorch.org/whl/nightly/cpu

[q10]

Hi @zhuhaozhe could you let us know what is the PyTorch version you are using? It is likely that you have a non-nightly version of PyTorch installed. We will make this clearer in the installation instructions, but nightly fbgemm_gpu is intended to run against nightly PyTorch.

[zhuhaozhe]

Hi, @q10. I am using 2024-03-13 nightly release. I build it from src with

conda install cmake ninja
pip install -r requirements.txt
git submodule sync
git submodule update --init --recursive
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py develop

nightly fbgemm_gpu is intended to run against nightly PyTorch

• Sometimes we need to use nightly Pytorch + some not merged PR, so we need to build Pytorch from src and we will encountered with this issue.

[gamdwk]

你好@zhuhaozhe您能告诉我们您正在使用的 PyTorch 版本是什么吗？您可能安装了非夜间版本的 PyTorch。我们将在安装说明中更清楚地说明这一点，但 nightly fbgemm_gpu 旨在针对 nightly PyTorch 运行。

I have same question.

 python -c "import torch; import fbgemm_gpu; print(torch.ops.fbgemm.merge_pooled_embeddings)"
/opt/conda/envs/pytorch-ci/lib/python3.8/site-packages/fbgemm_gpu/fbgemm_gpu_py.so: undefined symbol: _ZN3c109TupleTypeC1ESt6vectorINS_4Type24SingletonOrSharedTypePtrIS2_EESaIS4_EESt8optionalINS_13QualifiedNameEESt10shared_ptrINS_14FunctionSchemaEE
Traceback (most recent call last):
  File "/opt/conda/envs/pytorch-ci/lib/python3.8/site-packages/torch/_ops.py", line 565, in __getattr__
    op, overload_names = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator fbgemm::jagged_2d_to_dense

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/pytorch-ci/lib/python3.8/site-packages/fbgemm_gpu/__init__.py", line 23, in <module>
    from . import _fbgemm_gpu_docs, sparse_ops  # noqa: F401, E402  # noqa: F401, E402
  File "/opt/conda/envs/pytorch-ci/lib/python3.8/site-packages/fbgemm_gpu/_fbgemm_gpu_docs.py", line 19, in <module>
    torch.ops.fbgemm.jagged_2d_to_dense,
  File "/opt/conda/envs/pytorch-ci/lib/python3.8/site-packages/torch/_ops.py", line 569, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' 'fbgemm' object has no attribute 'jagged_2d_to_dense'

but I just install fbgemm-gpu by pip install fbgemm-gpu --index-url https://download.pytorch.org/whl/cu118/ my pytorch is installed by conda before. and the pytorch version is 2.0.0+cu118

[q10]

你好@zhuhaozhe您能告诉我们您正在使用的 PyTorch 版本是什么吗？您可能安装了非夜间版本的 PyTorch。我们将在安装说明中更清楚地说明这一点，但 nightly fbgemm_gpu 旨在针对 nightly PyTorch 运行。

I have same question.

 python -c "import torch; import fbgemm_gpu; print(torch.ops.fbgemm.merge_pooled_embeddings)"
/opt/conda/envs/pytorch-ci/lib/python3.8/site-packages/fbgemm_gpu/fbgemm_gpu_py.so: undefined symbol: _ZN3c109TupleTypeC1ESt6vectorINS_4Type24SingletonOrSharedTypePtrIS2_EESaIS4_EESt8optionalINS_13QualifiedNameEESt10shared_ptrINS_14FunctionSchemaEE
Traceback (most recent call last):
  File "/opt/conda/envs/pytorch-ci/lib/python3.8/site-packages/torch/_ops.py", line 565, in __getattr__
    op, overload_names = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator fbgemm::jagged_2d_to_dense

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/pytorch-ci/lib/python3.8/site-packages/fbgemm_gpu/__init__.py", line 23, in <module>
    from . import _fbgemm_gpu_docs, sparse_ops  # noqa: F401, E402  # noqa: F401, E402
  File "/opt/conda/envs/pytorch-ci/lib/python3.8/site-packages/fbgemm_gpu/_fbgemm_gpu_docs.py", line 19, in <module>
    torch.ops.fbgemm.jagged_2d_to_dense,
  File "/opt/conda/envs/pytorch-ci/lib/python3.8/site-packages/torch/_ops.py", line 569, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' 'fbgemm' object has no attribute 'jagged_2d_to_dense'

but I just install fbgemm-gpu by pip install fbgemm-gpu --index-url https://download.pytorch.org/whl/cu118/ my pytorch is installed by conda before. and the pytorch version is 2.0.0+cu118

Hi @gamdwk you should be installing the latest pytorch nightly, since fbgemm-gpu 0.6.0 is only compatible with pytorch 2.2+. The full installation instructions can be found in https://pytorch.org/FBGEMM/fbgemm_gpu-development/BuildInstructions.html.

[q10]

Hi, @q10. I am using 2024-03-13 nightly release. I build it from src with

conda install cmake ninja
pip install -r requirements.txt
git submodule sync
git submodule update --init --recursive
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py develop

nightly fbgemm_gpu is intended to run against nightly PyTorch

• Sometimes we need to use nightly Pytorch + some not merged PR, so we need to build Pytorch from src and we will encountered with this issue.

@zhuhaozhe Unfortunately, we don't have the bandwidth to offer support for the situation you have described.

In general, our CI builds and tests the fbgemm-gpu package against published pytorch nightly releases, and the error signature you observe generally points to a mismatch between the pytorch version that the fbgemm-gpu pacakge was built against, vs the pytorch version installed in the system.

If you have not already, please check the full installation instructions for fbgemm-gpu in https://pytorch.org/FBGEMM/fbgemm_gpu-development/InstallationInstructions.html, and let us know if you continue to see the issue.

[isuruf]

When building latest Pytorch from src and install fbgemm_nightly via pip

This is not expected to work as Pytorch will be compiled with C++11 ABI when you build from source whereas the Pytorch binary wheel be compiled with pre-C++11 ABI. FBGEMM nightly wheel via pip expects to be used with a pytorch compiled with pre-C++11 ABI.

So, you have to either build both pytorch and fbgemm from source (with C++11 ABI) or use pip wheels for both.