Code for Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture (accepted by PVLDB).The outdated write-up (https://arxiv.org/abs/2101.07956) explains engineering details, but only a portion of the functionality is migrated to this newer PyTorch version 1.8.0nightly (e152ca5).
I'm trying to build Pytorch from source, but during installation facing the below issue:
[1/336] Building CXX object caffe2/CMakeFiles/caffe2_pybind11_state.dir/python/pybind_state_registry.cc.o
FAILED: caffe2/CMakeFiles/caffe2_pybind11_state.dir/python/pybind_state_registry.cc.o
/usr/bin/c++ -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMAGMA_V2 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAM
ESPACE=onnx_torch -DTH_BLAS_MKL -DUSE_EXTERNAL_MZCRC -D_FILE_OFFSET_BITS=64 -Dcaffe2_pybind11_state_EXPORTS -I/mnt/utkrisht/pytorch-direct/build/aten/src -I/mnt/utkrisht/pytorch-direct/aten/src -I/mnt/utkr
isht/pytorch-direct/build -I/mnt/utkrisht/pytorch-direct -I/mnt/utkrisht/pytorch-direct/cmake/../third_party/benchmark/include -I/mnt/utkrisht/pytorch-direct/build/caffe2/contrib/aten -I/mnt/utkrisht/pytor
ch-direct/third_party/onnx -I/mnt/utkrisht/pytorch-direct/build/third_party/onnx -I/mnt/utkrisht/pytorch-direct/third_party/foxi -I/mnt/utkrisht/pytorch-direct/build/third_party/foxi -I/mnt/utkrisht/pytorc
h-direct/build/caffe2/aten/src/TH -I/mnt/utkrisht/pytorch-direct/aten/src/TH -I/mnt/utkrisht/pytorch-direct/build/caffe2/aten/src -I/mnt/utkrisht/pytorch-direct/aten/../third_party/catch/single_include -I/
mnt/utkrisht/pytorch-direct/aten/src/ATen/.. -I/mnt/utkrisht/pytorch-direct/build/caffe2/aten/src/ATen -I/mnt/utkrisht/pytorch-direct/third_party/miniz-2.0.8 -I/mnt/utkrisht/pytorch-direct/caffe2/core/nomn
igraph/include -I/mnt/utkrisht/pytorch-direct/torch/csrc/api -I/mnt/utkrisht/pytorch-direct/torch/csrc/api/include -I/mnt/utkrisht/pytorch-direct/c10/.. -I/mnt/utkrisht/pytorch-direct/build/third_party/ide
ep/mkl-dnn/include -I/mnt/utkrisht/pytorch-direct/third_party/ideep/mkl-dnn/src/../include -I/mnt/utkrisht/pytorch-direct/c10/cuda/../.. -isystem /mnt/utkrisht/pytorch-direct/build/third_party/gloo -isyste
m /mnt/utkrisht/pytorch-direct/cmake/../third_party/gloo -isystem /mnt/utkrisht/pytorch-direct/cmake/../third_party/googletest/googlemock/include -isystem /mnt/utkrisht/pytorch-direct/cmake/../third_party/
googletest/googletest/include -isystem /mnt/utkrisht/pytorch-direct/third_party/protobuf/src -isystem /mnt/utkrisht/miniconda3/envs/pytorch-direct/include -isystem /mnt/utkrisht/pytorch-direct/third_party/
gemmlowp -isystem /mnt/utkrisht/pytorch-direct/third_party/neon2sse -isystem /mnt/utkrisht/pytorch-direct/third_party/XNNPACK/include -isystem /mnt/utkrisht/pytorch-direct/third_party -isystem /mnt/utkrish
t/pytorch-direct/cmake/../third_party/eigen -isystem /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11 -isystem /mnt/utkrisht/miniconda3/envs/pytorch-direct/lib/python3.11/site-packages/numpy
/core/include -isystem /mnt/utkrisht/pytorch-direct/cmake/../third_party/pybind11/include -isystem /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -isystem /usr/lib/x86_64-linux-gnu/openmpi/include -isys
tem /usr/local/cuda-11.7/include -isystem /mnt/utkrisht/pytorch-direct/third_party/ideep/mkl-dnn/include -isystem /mnt/utkrisht/pytorch-direct/third_party/ideep/include -Wno-deprecated -fvisibility-inlines
-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initial
izers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-ove
rflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned
-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFI
NITION -O3 -DNDEBUG -DNDEBUG -fPIC -fvisibility=hidden -DCAFFE2_USE_GLOO -DCUDA_HAS_FP16=1 -DHAVE_GCC_GET_CPUID -DUSE_AVX -DUSE_AVX2 -DTH_HAVE_THREAD -pthread -MD -MT caffe2/CMakeFiles/caffe2_pybind11_stat
e.dir/python/pybind_state_registry.cc.o -MF caffe2/CMakeFiles/caffe2_pybind11_state.dir/python/pybind_state_registry.cc.o.d -o caffe2/CMakeFiles/caffe2_pybind11_state.dir/python/pybind_state_registry.cc.o
-c /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc
In file included from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h: In function ‘std::string pybind11::detail::error_string()’:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:446:36: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
446 | " " + handle(frame->f_code->co_filename).cast<std::string>() +
| ^~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
In file included from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:446:75: error: expected primary-expression before ‘>’ token
446 | " " + handle(frame->f_code->co_filename).cast<std::string>() +
| ^
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:446:77: error: expected primary-expression before ‘)’ token
446 | " " + handle(frame->f_code->co_filename).cast<std::string>() +
| ^
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:448:29: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
448 | handle(frame->f_code->co_name).cast<std::string>() + "\n";
| ^~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
In file included from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13, [78/1188]
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:448:64: error: expected primary-expression before ‘>’ token
448 | handle(frame->f_code->co_name).cast<std::string>() + "\n";
| ^
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:448:66: error: expected primary-expression before ‘)’ token
448 | handle(frame->f_code->co_name).cast<std::string>() + "\n";
| ^
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:449:26: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
449 | frame = frame->f_back;
| ^~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
In file included from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h: In function ‘pybind11::function pybind11::detail::get_type_override(const void*, const pybind11::detail::type_info*, const cha
r*)’:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:2232:49: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
2232 | PyFrameObject *frame = PyThreadState_Get()->frame;
| ^~~~~
| cframe
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:2233:41: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
2233 | if (frame && (std::string) str(frame->f_code->co_name) == name &&
| ^~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
In file included from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:2234:14: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
2234 | frame->f_code->co_argcount > 0) {
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
In file included from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:2237:18: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
2237 | frame->f_locals, PyTuple_GET_ITEM(frame->f_code->co_varnames, 0));
| ^~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:38,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:2237:30: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
2237 | frame->f_locals, PyTuple_GET_ITEM(frame->f_code->co_varnames, 0));
| ^~~~~~~~~~~~~~~~
In file included from /mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/Python.h:42,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/detail/common.h:122,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pytypes.h:12,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/cast.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/attr.h:13,
from /mnt/utkrisht/pytorch-direct/third_party/pybind11/include/pybind11/pybind11.h:45,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.h:3,
from /mnt/utkrisht/pytorch-direct/caffe2/python/pybind_state_registry.cc:1:
/mnt/utkrisht/miniconda3/envs/pytorch-direct/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
22 | typedef struct _frame PyFrameObject;
| ^~~~~~
cython is latest version within my conda environment
$cython -V
Cython version 0.29.35
$ which cython
/mnt/utkrisht/miniconda3/envs/pytorch-direct/bin/cython
pybind11 is installed
pybind11 2.10.4
Have tried installing with Python 3.8 but still facing the same issue.
(Im trying to install a specific commit as I am setting up PyTorch-direct a functionality available only in that specific commit of PyTorch)
Can someone please suggest what can be done to circumvent this issue?
Please note that this issue tracker is not a help form and this issue will be closed.
❓ Questions and Help
I'm trying to build Pytorch from source, but during installation facing the below issue:
(Im trying to install a specific commit as I am setting up PyTorch-direct a functionality available only in that specific commit of PyTorch) Can someone please suggest what can be done to circumvent this issue?
Please note that this issue tracker is not a help form and this issue will be closed.
We have a set of listed resources available on the website. Our primary means of support is our discussion forum: