PyTorch-Direct code on top of PyTorch-1.8.0nightly (e152ca5) for Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture (accepted by PVLDB)
I'm trying to build Pytorch from source, but during installation facing the below issue:
[1/336] Building CXX object caffe2/CMakeFiles/caffe2_pybind11_state.dir/python/pybind_state_registry.cc.o
FAILED: caffe2/CMakeFiles/caffe2_pybind11_state.dir/python/pybind_state_registry.cc.o
/usr/bin/c++ -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMAGMA_V2 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAM
ESPACE=onnx_torch -DTH_BLAS_MKL -DUSE_EXTERNAL_MZCRC -D_FILE_OFFSET_BITS=64 -Dcaffe2_pybind11_state_EXPORTS -I/mnt/utkrisht/pytorch-direct/build/aten/src -I/mnt/utkrisht/pytorch-direct/aten/src -I/mnt/utkr
isht/pytorch-direct/build -I/mnt/utkrisht/pytorch-direct -I/mnt/utkrisht/pytorch-direct/cmake/../third_party/benchmark/include -I/mnt/utkrisht/pytorch-direct/build/caffe2/contrib/aten -I/mnt/utkrisht/pytor
ch-direct/third_party/onnx -I/mnt/utkrisht/pytorch-direct/build/third_party/onnx -I/mnt/utkrisht/pytorch-direct/third_party/foxi -I/mnt/utkrisht/pytorch-direct/build/third_party/foxi -I/mnt/utkrisht/pytorc
h-direct/build/caffe2/aten/src/TH -I/mnt/utkrisht/pytorch-direct/aten/src/TH -I/mnt/utkrisht/pytorch-direct/build/caffe2/aten/src -I/mnt/utkrisht/pytorch-direct/aten/../third_party/catch/single_include -I/
mnt/utkrisht/pytorch-direct/aten/src/ATen/.. -I/mnt/utkrisht/pytorch-direct/build/caffe2/aten/src/ATen -I/mnt/utkrisht/pytorch-direct/third_party/miniz-2.0.8 -I/mnt/utkrisht/pytorch-direct/caffe2/core/nomn
igraph/include -I/mnt/utkrisht/pytorch-direct/torch/csrc/api -I/mnt/utkrisht/pytorch-direct/torch/csrc/api/include -I/mnt/utkrisht/pytorch-direct/c10/.. -I/mnt/utkrisht/pytorch-direct/build/third_party/ide
ep/mkl-dnn/include -I/mnt/utkrisht/pytorch-direct/third_party/ideep/mkl-dnn/src/../include -I/mnt/utkrisht/pytorch-direct/c10/cuda/../.. -isystem /mnt/utkrisht/pytorch-direct/build/third_party/gloo -isyste
m /mnt/utkrisht/pytorch-direct/cmake/../third_party/gloo -isystem /mnt/utkrisht/pytorch-direct/cmake/../third_party/googletest/googlemock/include -isystem /mnt/utkrisht/pytorch-direct/cmake/../third_party/
googletest/googletest/include -isystem /mnt/utkrisht/pytorch-direct/third_party/protobuf/src -isystem /mnt/utkrisht/miniconda3/envs/pytorch-direct/include -isystem /mnt/utkrisht/pytorch-direct/third_party/
gemmlowp -isystem /mnt/utkrisht/pytorch-direct/third_party/neon2sse -isystem /mnt/utkrisht/pytorch-direct/third_party/XNNPACK/include -isystem /mnt/utkrisht/pytorch-direct/third_party -isystem /mnt/utkrish
t/pytorch-direct/cmake/../third_party/eigen -isystem /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11 -isystem /mnt/utkrisht/miniconda3/envs/pytorch-direct/lib/python3.11/site-packages/numpy
/core/include -isystem /mnt/utkrisht/pytorch-direct/cmake/../third_party/pybind11/include -isystem /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -isystem /usr/lib/x86_64-linux-gnu/openmpi/include -isys
tem /usr/local/cuda-11.7/include -isystem /mnt/utkrisht/pytorch-direct/third_party/ideep/mkl-dnn/include -isystem /mnt/utkrisht/pytorch-direct/third_party/ideep/include -Wno-deprecated -fvisibility-inlines
-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initial
izers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-ove
rflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned
-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFI
NITION -O3 -DNDEBUG -DNDEBUG -fPIC -fvisibility=hidden -DCAFFE2_USE_GLOO -DCUDA_HAS_FP16=1 -DHAVE_GCC_GET_CPUID -DUSE_AVX -DUSE_AVX2 -DTH_HAVE_THREAD -pthread -MD -MT caffe2/CMakeFiles/caffe2_pybind11_stat
e.dir/python/pybind_state_registry.cc.o -MF caffe2/CMakeFiles/caffe2_pybind11_state.dir/python/pybind_state_registry.cc.o.d -o caffe2/CMakeFiles/caffe2_pybind11_state.dir/python/pybind_state_registry.cc.o
-c /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc
In file included from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h: In function ‘std::string pybind11::detail::error_string()’:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:446:36: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
446 | " " + handle(frame->f_code->co_filename).cast<std::string>() +
| ^~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
In file included from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:446:75: error: expected primary-expression before ‘>’ token
446 | " " + handle(frame->f_code->co_filename).cast<std::string>() +
| ^
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:446:77: error: expected primary-expression before ‘)’ token
446 | " " + handle(frame->f_code->co_filename).cast<std::string>() +
| ^
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:448:29: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
448 | handle(frame->f_code->co_name).cast<std::string>() + "\n";
| ^~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
In file included from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13, [78/1188]
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:448:64: error: expected primary-expression before ‘>’ token
448 | handle(frame->f_code->co_name).cast<std::string>() + "\n";
| ^
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:448:66: error: expected primary-expression before ‘)’ token
448 | handle(frame->f_code->co_name).cast<std::string>() + "\n";
| ^
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:449:26: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
449 | frame = frame->f_back;
| ^~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
In file included from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h: In function ‘pybind11::function pybind11::detail::get_type_override(const void*, const pybind11::detail::type_info*, const cha
r*)’:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:2232:49: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
2232 | PyFrameObject *frame = PyThreadState_Get()->frame;
| ^~~~~
| cframe
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:2233:41: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
2233 | if (frame && (std::string) str(frame->f_code->co_name) == name &&
| ^~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
In file included from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:2234:14: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
2234 | frame->f_code->co_argcount > 0) {
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
In file included from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:2237:18: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
2237 | frame->f_locals, PyTuple_GET_ITEM(frame->f_code->co_varnames, 0));
| ^~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:38,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:2237:30: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
2237 | frame->f_locals, PyTuple_GET_ITEM(frame->f_code->co_varnames, 0));
| ^~~~~~~~~~~~~~~~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
cython is latest version within my conda environment
$cython -V
Cython version 0.29.35
$ which cython
/mnt/utkrisht/miniconda3/envs/pytorch-direct/bin/cython
pybind11 is installed
pybind11 2.10.4
Have tried installing with Python 3.8 but still facing the same issue.
(Im trying to install a specific commit as I am setting up PyTorch-direct a functionality available only in that specific commit of PyTorch)
Can someone please suggest what can be done to circumvent this issue?
I'm trying to build Pytorch from source, but during installation facing the below issue:
(Im trying to install a specific commit as I am setting up PyTorch-direct a functionality available only in that specific commit of PyTorch) Can someone please suggest what can be done to circumvent this issue?