ezyang
commented
11 months ago Oh, it's Iconv::Iconv, why is this not working 🤔
Open ezyang opened 11 months ago
ezyang
commented
11 months ago Oh, it's Iconv::Iconv, why is this not working 🤔
jc-audet
commented
10 months ago I'm having a similar issue where I cannot compile the native lib because of libraries.
Context: I'm using conda and not venv, which might be part of the problem. Details below:
Environment creation:
conda create -n fairseq2_source python=3.8
conda activate fairseq2_source
conda install -c conda-forge libsndfile==1.0.31
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
conda install -c conda-forge compilers
git clone --recurse-submodules https://github.com/facebookresearch/fairseq2.git
Cd fairseq2
pip install -r native/python/requirements-build.txt
cd native/
cmake -GNinja -B build
cmake --build buildConda env:
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
binutils 2.38 h1680402_1
binutils_impl_linux-64 2.38 h2a08ee3_1
binutils_linux-64 2.38.0 hc2dff05_0
blas 1.0 mkl
brotli-python 1.0.9 py38h6a678d5_7
bzip2 1.0.8 h7b6447c_0
c-compiler 1.5.2 h0b41bf4_0 conda-forge
ca-certificates 2023.11.17 hbcca054_0 conda-forge
certifi 2023.11.17 pyhd8ed1ab_0 conda-forge
cffi 1.16.0 py38h5eee18b_0
charset-normalizer 2.0.4 pyhd3eb1b0_0
cmake 3.28.0 pypi_0 pypi
compilers 1.5.2 ha770c72_0 conda-forge
cryptography 41.0.7 py38hdda0065_0
cuda-cudart 12.1.105 0 nvidia
cuda-cupti 12.1.105 0 nvidia
cuda-libraries 12.1.0 0 nvidia
cuda-nvrtc 12.1.105 0 nvidia
cuda-nvtx 12.1.105 0 nvidia
cuda-opencl 12.3.101 0 nvidia
cuda-runtime 12.1.0 0 nvidia
cxx-compiler 1.5.2 hf52228f_0 conda-forge
ffmpeg 4.3 hf484d3e_0 pytorch
filelock 3.13.1 py38h06a4308_0
fortran-compiler 1.5.2 hdb1a99f_0 conda-forge
freetype 2.12.1 h4a9f257_0
gcc 11.2.0 h702ea55_10 conda-forge
gcc_impl_linux-64 11.2.0 h1234567_1
gcc_linux-64 11.2.0 h5c386dc_0
gettext 0.21.0 h39681ba_1
gfortran 11.2.0 h8811e0c_10 conda-forge
gfortran_impl_linux-64 11.2.0 h7a446d4_16 conda-forge
gfortran_linux-64 11.2.0 hc2dff05_0
giflib 5.2.1 h5eee18b_3
gmp 6.2.1 h295c915_3
gmpy2 2.1.2 py38heeb90bb_0
gnutls 3.6.15 he1e5248_0
gxx 11.2.0 h702ea55_10 conda-forge
gxx_impl_linux-64 11.2.0 h1234567_1
gxx_linux-64 11.2.0 hc2dff05_0
icu 58.2 hf484d3e_1000 conda-forge
idna 3.4 py38h06a4308_0
intel-openmp 2023.1.0 hdb19cb5_46306
jinja2 3.1.2 py38h06a4308_0
jpeg 9e h5eee18b_1
kernel-headers_linux-64 2.6.32 he073ed8_16 conda-forge
lame 3.100 h7b6447c_0
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libcublas 12.1.0.26 0 nvidia
libcufft 11.0.2.4 0 nvidia
libcufile 1.8.1.2 0 nvidia
libcurand 10.3.4.101 0 nvidia
libcusolver 11.4.4.55 0 nvidia
libcusparse 12.0.2.55 0 nvidia
libdeflate 1.17 h5eee18b_1
libffi 3.4.4 h6a678d5_0
libflac 1.3.4 h27087fc_0 conda-forge
libgcc-devel_linux-64 11.2.0 h1234567_1
libgcc-ng 11.2.0 h1234567_1
libgfortran5 13.2.0 ha4646dd_0 conda-forge
libgomp 11.2.0 h1234567_1
libiconv 1.16 h7f8727e_2
libidn2 2.3.4 h5eee18b_0
libjpeg-turbo 2.0.0 h9bf148f_0 pytorch
libnpp 12.0.2.50 0 nvidia
libnvjitlink 12.1.105 0 nvidia
libnvjpeg 12.1.1.14 0 nvidia
libogg 1.3.4 h7f98852_1 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpng 1.6.39 h5eee18b_0
libsndfile 1.0.31 h9c3ff4c_1 conda-forge
libstdcxx-devel_linux-64 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libtasn1 4.19.0 h5eee18b_0
libtiff 4.5.1 h6a678d5_0
libunistring 0.9.10 h27cfd23_0
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp 1.3.2 h11a3e52_0
libwebp-base 1.3.2 h5eee18b_0
libxml2 2.10.4 hcbfbd50_0
llvm-openmp 14.0.6 h9e868ea_0
lz4-c 1.9.4 h6a678d5_0
markupsafe 2.1.1 py38h7f8727e_0
mkl 2023.1.0 h213fc3f_46344
mkl-service 2.4.0 py38h5eee18b_1
mkl_fft 1.3.8 py38h5eee18b_0
mkl_random 1.2.4 py38hdb19cb5_0
mpc 1.1.0 h10f8cd9_1
mpfr 4.0.2 hb69a4c5_1
mpmath 1.3.0 py38h06a4308_0
ncurses 6.4 h6a678d5_0
nettle 3.7.3 hbbd107a_1
networkx 3.1 py38h06a4308_0
ninja 1.11.1.1 pypi_0 pypi
numpy 1.24.3 py38hf6e8229_1
numpy-base 1.24.3 py38h060ed82_1
openh264 2.1.1 h4ff587b_0
openjpeg 2.4.0 h3ad879b_0
openssl 3.0.12 h7f8727e_0
packaging 23.2 pypi_0 pypi
pillow 10.0.1 py38ha6cbd5a_0
pip 23.3.1 py38h06a4308_0
pycparser 2.21 pyhd3eb1b0_0
pyopenssl 23.2.0 py38h06a4308_0
pysocks 1.7.1 py38h06a4308_0
python 3.8.18 h955ad1f_0
pytorch 2.1.1 py3.8_cuda12.1_cudnn8.9.2_0 pytorch
pytorch-cuda 12.1 ha16c6d3_5 pytorch
pytorch-mutex 1.0 cuda pytorch
pyyaml 6.0.1 py38h5eee18b_0
readline 8.2 h5eee18b_0
requests 2.31.0 py38h06a4308_0
setuptools 67.8.0 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0
sympy 1.12 py38h06a4308_0
sysroot_linux-64 2.12 he073ed8_16 conda-forge
tbb 2021.8.0 pypi_0 pypi
tbb-devel 2021.8.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
torchaudio 2.1.1 py38_cu121 pytorch
torchtriton 2.1.0 py38 pytorch
torchvision 0.16.1 py38_cu121 pytorch
typing_extensions 4.7.1 py38h06a4308_0
urllib3 1.26.18 py38h06a4308_0
wheel 0.41.2 py38h06a4308_0
xz 5.4.5 h5eee18b_0
yaml 0.2.5 h7b6447c_0
zlib 1.2.13 h5eee18b_0
zstd 1.5.5 hc292b87_0Output from cmake -GNinja -B build
-- Module support is disabled.
-- Version: 9.1.0
-- Build type: RelWithDebInfo
-- CXX_STANDARD: 14
-- Required features: cxx_variadic_templates
CMake Deprecation Warning at third-party/kaldi-native-fbank/CMakeLists.txt:24 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
-- CMAKE_BUILD_TYPE: RelWithDebInfo
-- CMAKE_EXPORT_COMPILE_COMMANDS:
-- BUILD_SHARED_LIBS: FALSE
-- KALDI_NATIVE_FBANK_BUILD_TESTS: OFF
-- KALDI_NATIVE_FBANK_BUILD_PYTHON: OFF
-- KALDI_NATIVE_FBANK_ENABLE_CHECK: OFF
-- KALDI_NATIVE_FBANK_ENABLE_CHECK: OFF
-- CMAKE_CXX_FLAGS: -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /private/home/jcaudet/.conda/envs/fairseq2_source/include
-- CMAKE_INSTALL_PREFIX: /usr/local
-- Disable building Python
CMake Deprecation Warning at third-party/sentencepiece/CMakeLists.txt:15 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
-- VERSION: 0.1.97
-- Not Found TCMalloc: TCMALLOC_LIB-NOTFOUND
CMake Deprecation Warning at third-party/libpng/CMakeLists.txt:14 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
CMake Deprecation Warning at third-party/libpng/CMakeLists.txt:15 (cmake_policy):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
-- Symbol prefix:
-- pybind11 v2.11.1
--
-- ******************** Summary ********************
-- CMake Version : 3.28.0
-- CMake Command : /private/home/jcaudet/.conda/envs/fairseq2_source/lib/python3.8/site-packages/cmake/data/bin/cmake
-- System : Linux
-- C++ Compiler : /private/home/jcaudet/.conda/envs/fairseq2_source/bin/x86_64-conda-linux-gnu-c++
-- C++ Compiler Version : 11.2.0
-- Python Version : 3.8.18
-- Python Interpreter : /private/home/jcaudet/.conda/envs/fairseq2_source/bin/python3.8
-- Torch Version : 2.1.1
-- Torch Library : /private/home/jcaudet/.conda/envs/fairseq2_source/lib/python3.8/site-packages/torch/lib/libtorch.so
-- Build Type : RelWithDebInfo
--
-- PROJECT_VERSION : 0.3.0
-- FAIRSEQ2N_BUILD_FOR_NATIVE : OFF
-- FAIRSEQ2N_INSTALL_STANDALONE : ON
-- FAIRSEQ2N_PERFORM_LTO : OFF
-- FAIRSEQ2N_RUN_CLANG_TIDY : OFF
-- FAIRSEQ2N_SANITIZERS : -
-- FAIRSEQ2N_TREAT_WARNINGS_AS_ERRORS : OFF
-- FAIRSEQ2N_SUPPORT_IMAGE : ON
-- FAIRSEQ2N_USE_LIBTORCH : OFF
-- FAIRSEQ2N_USE_CUDA : OFF
-- FAIRSEQ2N_BUILD_PYTHON_BINDINGS : ON
-- FAIRSEQ2N_PYTHON_DEVEL : ON
--
-- System Dependencies
-- Intel oneTBB : 2021.8.0
-- libsndfile : 1.0.31
--
-- Configuring done (12.4s)
-- Generating done (0.5s)
-- Build files have been written to: /private/home/jcaudet/repos/fairseq2/native/buildOutput of cmake --build build.
[28/122] Building CXX object third-party/sentencep...tencepiece-static.dir/sentencepiece_processor.cc.o
FAILED: third-party/sentencepiece/src/CMakeFiles/sentencepiece-static.dir/sentencepiece_processor.cc.o
/private/home/jcaudet/.conda/envs/fairseq2_source/bin/x86_64-conda-linux-gnu-c++ -D_GLIBCXX_USE_CXX11_ABI=0 -D_USE_INTERNAL_STRING_VIEW -I/private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece -I/private/home/jcaudet/repos/fairseq2/native/build/third-party/sentencepiece -I/private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/src/../third_party/protobuf-lite -I/private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/src/builtin_pb -I/private/home/jcaudet/repos/fairseq2/native/build/third-party/sentencepiece/src -I/private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/src/../third_party -isystem /private/home/jcaudet/repos/fairseq2/native/third-party -isystem /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/third_party/protobuf-lite -O3 -Wall -fPIC -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /private/home/jcaudet/.conda/envs/fairseq2_source/include -fmacro-prefix-map=/private/home/jcaudet/repos/fairseq2/native/='' -O2 -g -DNDEBUG -std=gnu++17 -fPIC -fvisibility=hidden -pthread -DHAVE_PTHREAD=1 -Wno-sign-compare -Wno-stringop-overflow -MD -MT third-party/sentencepiece/src/CMakeFiles/sentencepiece-static.dir/sentencepiece_processor.cc.o -MF third-party/sentencepiece/src/CMakeFiles/sentencepiece-static.dir/sentencepiece_processor.cc.o.d -o third-party/sentencepiece/src/CMakeFiles/sentencepiece-static.dir/sentencepiece_processor.cc.o -c /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/src/sentencepiece_processor.cc
In file included from /private/home/jcaudet/.conda/envs/fairseq2_source/x86_64-conda-linux-gnu/include/c++/11.2.0/chrono:42,
from /private/home/jcaudet/.conda/envs/fairseq2_source/x86_64-conda-linux-gnu/include/c++/11.2.0/mutex:39,
from /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/third_party/protobuf-lite/google/protobuf/stubs/once.h:34,
from /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/third_party/protobuf-lite/google/protobuf/message_lite.h:50,
from /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/third_party/protobuf-lite/google/protobuf/generated_enum_util.h:36,
from /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/third_party/protobuf-lite/google/protobuf/map.h:55,
from /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/third_party/protobuf-lite/google/protobuf/generated_message_table_driven.h:34,
from /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/src/builtin_pb/sentencepiece_model.pb.h:26,
from /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/src/normalizer.h:25,
from /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/src/model_interface.h:25,
from /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/src/model_factory.h:20,
from /private/home/jcaudet/repos/fairseq2/native/third-party/sentencepiece/src/sentencepiece_processor.cc:23:
/private/home/jcaudet/.conda/envs/fairseq2_source/x86_64-conda-linux-gnu/include/c++/11.2.0/ctime:80:11: error: 'timespec_get' has not been declared in '::'
80 | using ::timespec_get;
| ^~~~~~~~~~~~Yeah, if you do the venv it will work, and I have no idea why lol.
I don't see any target_link_libraries to iconv so I'm not sure why this is expected to work in the first place