search

neulab / guided_summarization

GSum: A General Framework for Guided Neural Abstractive Summarization
MIT License
113 stars 27 forks source link

ImportError: Please build Cython components with: `pip install --editable .` or `python setup.py build_ext --inplace` #13

Closed SONGPENG closed 3 years ago

SONGPENG commented 3 years ago

2021-06-08 23:33:42 | INFO | fairseq_cli.train | model guided_bart_large, criterion LabelSmoothedCrossEntropyCriterion 2021-06-08 23:33:42 | INFO | fairseq_cli.train | num. model params: 469292032 (num. trained: 469292032) 2021-06-08 23:33:45 | INFO | fairseq_cli.train | training on 1 GPUs 2021-06-08 23:33:45 | INFO | fairseq_cli.train | max tokens per GPU = 2048 and max sentences per GPU = None 2021-06-08 23:33:45 | INFO | fairseq.trainer | no existing checkpoint found /projects/tir5/users/pliu3/zdou/fairseq/bart.large/model.pt 2021-06-08 23:33:45 | INFO | fairseq.trainer | loading train data for epoch 0 2021-06-08 23:33:46 | INFO | fairseq.data.data_utils | loaded 287227 examples from: /data/songpeng/project_my/guided_summarization/bart/data-bin/cnn_dm-bin/train.source-target.source 2021-06-08 23:33:47 | INFO | fairseq.data.data_utils | loaded 287227 examples from: /data/songpeng/project_my/guided_summarization/bart/data-bin/cnn_dm-bin/train.source-target.target 2021-06-08 23:33:47 | INFO | fairseq.data.data_utils | loaded 287227 examples from: /data/songpeng/project_my/guided_summarization/bart/data-bin/cnn_dm-bin/train.source-target.z 2021-06-08 23:33:47 | INFO | fairseq.tasks.guided_translation | /data/songpeng/project_my/guided_summarization/bart/data-bin/cnn_dm-bin/ train source-target 287227 examples 2021-06-08 23:33:49 | WARNING | fairseq.data.data_utils | 4 samples have invalid sizes and will be skipped, max_positions=(1024, 1024, 1024), first few sample ids=[189447, 112053, 286032, 172051] Traceback (most recent call last): File "/data/songpeng/project_my/guided_summarization/bart/fairseq/data/data_utils.py", line 221, in batch_by_size from fairseq.data.data_utils_fast import batch_by_size_fast ModuleNotFoundError: No module named 'fairseq.data.data_utils_fast'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train.py", line 11, in cli_main() File "/data/songpeng/project_my/guided_summarization/bart/fairseq_cli/train.py", line 318, in cli_main main(args) File "/data/songpeng/project_my/guided_summarization/bart/fairseq_cli/train.py", line 81, in main extra_state, epoch_itr = checkpoint_utils.load_checkpoint(args, trainer) File "/data/songpeng/project_my/guided_summarization/bart/fairseq/checkpoint_utils.py", line 152, in load_checkpoint epoch_itr = trainer.get_train_iterator( File "/data/songpeng/project_my/guided_summarization/bart/fairseq/trainer.py", line 275, in get_train_iterator return self.task.get_batch_iterator( File "/data/songpeng/project_my/guided_summarization/bart/fairseq/tasks/fairseq_task.py", line 176, in get_batch_iterator batch_sampler = data_utils.batch_by_size( File "/data/songpeng/project_my/guided_summarization/bart/fairseq/data/data_utils.py", line 223, in batch_by_size raise ImportError( ImportError: Please build Cython components with: pip install --editable . or python setup.py build_ext --inplace srun: error: pgpu10: task 0: Exited with exit code 1

Are there some errors in my fairseq?

SONGPENG commented 3 years ago

And then I use pip install --editable . , I got

Obtaining file:///data/songpeng/project_my/guided_summarization/bart Requirement already satisfied: cffi in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (1.14.5) Requirement already satisfied: cython in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (0.29.23) Requirement already satisfied: numpy in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (1.20.2) Requirement already satisfied: regex in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (2021.4.4) Requirement already satisfied: sacrebleu in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (1.5.1) Requirement already satisfied: torch in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (1.8.1) Requirement already satisfied: tqdm in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (4.61.0) Requirement already satisfied: pycparser in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from cffi->fairseq==0.9.0) (2.20) Requirement already satisfied: portalocker==2.0.0 in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from sacrebleu->fairseq==0.9.0) (2.0.0) Requirement already satisfied: typing_extensions in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from torch->fairseq==0.9.0) (3.7.4.3) Installing collected packages: fairseq Running setup.py develop for fairseq ERROR: Command errored out with exit status 1: command: /data/songpeng/.conda/envs/feqsem/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/data/songpeng/project_my/guided_summarization/bart/setup.py'"'"'; file='"'"'/data/songpeng/project_my/guided_summarization/bart/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps cwd: /data/songpeng/project_my/guided_summarization/bart/ Complete output (106 lines): No CUDA runtime is found, using CUDA_HOME='/cm/shared/apps/cuda11.1/toolkit/11.1.0' running develop running egg_info writing fairseq.egg-info/PKG-INFO writing dependency_links to fairseq.egg-info/dependency_links.txt writing entry points to fairseq.egg-info/entry_points.txt writing requirements to fairseq.egg-info/requires.txt writing top-level names to fairseq.egg-info/top_level.txt reading manifest file 'fairseq.egg-info/SOURCES.txt' writing manifest file 'fairseq.egg-info/SOURCES.txt' running build_ext cythoning fairseq/data/data_utils_fast.pyx to fairseq/data/data_utils_fast.cpp cythoning fairseq/data/token_block_utils_fast.pyx to fairseq/data/token_block_utils_fast.cpp building 'fairseq.libbleu' extension Emitting ninja build file /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. g++ -pthread -shared -B /data/songpeng/.conda/envs/feqsem/compiler_compat -L/data/songpeng/.conda/envs/feqsem/lib -Wl,-rpath=/data/songpeng/.conda/envs/feqsem/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu/libbleu.o /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu/module.o -o build/lib.linux-x86_64-3.8/fairseq/libbleu.cpython-38-x86_64-linux-gnu.so building 'fairseq.data.data_utils_fast' extension Emitting ninja build file /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/1] c++ -MMD -MF /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/data_utils_fast.o.d -pthread -B /data/songpeng/.conda/envs/feqsem/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/feqsem/include/python3.8 -c -c /data/songpeng/project_my/guided_summarization/bart/fairseq/data/data_utils_fast.cpp -o /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/data_utils_fast.o -std=c++11 -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=data_utils_fast -D_GLIBCXX_USE_CXX11_ABI=0 cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ In file included from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1944, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /data/songpeng/project_my/guided_summarization/bart/fairseq/data/data_utils_fast.cpp:626: /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~ g++ -pthread -shared -B /data/songpeng/.conda/envs/feqsem/compiler_compat -L/data/songpeng/.conda/envs/feqsem/lib -Wl,-rpath=/data/songpeng/.conda/envs/feqsem/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/data_utils_fast.o -o build/lib.linux-x86_64-3.8/fairseq/data/data_utils_fast.cpython-38-x86_64-linux-gnu.so building 'fairseq.data.token_block_utils_fast' extension Emitting ninja build file /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/1] c++ -MMD -MF /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.o.d -pthread -B /data/songpeng/.conda/envs/feqsem/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/feqsem/include/python3.8 -c -c /data/songpeng/project_my/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp -o /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.o -std=c++11 -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=token_block_utils_fast -D_GLIBCXX_USE_CXX11_ABI=0 cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ In file included from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1944, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /data/songpeng/project_my/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp:627: /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~ /data/songpeng/project_my/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp: In function ‘PyArrayObject __pyx_f_7fairseq_4data_22token_block_utils_fast__get_slice_indices_fast(PyArrayObject, PyObject*, int, int, int)’: /data/songpeng/project_my/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp:3306:36: warning: comparison of integer expressions of different signedness: ‘pyx_t_7fairseq_4data_22token_block_utils_fast_DTYPE_t’ {aka ‘long int’} and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare] 3306 | pyx_t_4 = ((pyx_v_sz_idx < __pyx_t_10) != 0); | ~~~^~~~ /data/songpeng/project_my/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp:3501:36: warning: comparison of integer expressions of different signedness: ‘pyx_t_7fairseq_4data_22token_block_utils_fast_DTYPE_t’ {aka ‘long int’} and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare] 3501 | pyx_t_3 = ((pyx_v_sz_idx < pyx_t_10) != 0); | ~~~^~~~ g++ -pthread -shared -B /data/songpeng/.conda/envs/feqsem/compiler_compat -L/data/songpeng/.conda/envs/feqsem/lib -Wl,-rpath=/data/songpeng/.conda/envs/feqsem/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.o -o build/lib.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.cpython-38-x86_64-linux-gnu.so building 'fairseq.libnat' extension Emitting ninja build file /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. g++ -pthread -shared -B /data/songpeng/.conda/envs/feqsem/compiler_compat -L/data/songpeng/.conda/envs/feqsem/lib -Wl,-rpath=/data/songpeng/.conda/envs/feqsem/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project_my/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libnat/edit_dist.o -L/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-3.8/fairseq/libnat.cpython-38-x86_64-linux-gnu.so building 'fairseq.libnat_cuda' extension Traceback (most recent call last): File "", line 1, in File "/data/songpeng/project_my/guided_summarization/bart/setup.py", line 118, in setup( File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/setuptools/init__.py", line 153, in setup return distutils.core.setup(**attrs) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/core.py", line 148, in setup dist.run_commands() File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run self.install_for_development() File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/setuptools/command/develop.py", line 136, in install_for_development self.run_command('build_ext') File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run _build_ext.run(self) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 708, in build_extensions build_ext.build_extensions(self) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions _build_ext.build_ext.build_extensions(self) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions self._build_extensions_serial() File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial self.build_extension(ext) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 196, in build_extension _build_ext.build_extension(self, ext) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension objects = self.compiler.compile(sources, File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 524, in unix_wrap_ninja_compile cuda_post_cflags = unix_cuda_flags(cuda_post_cflags) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 423, in unix_cuda_flags cflags + _get_cuda_arch_flags(cflags)) File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1561, in _get_cuda_arch_flags arch_list[-1] += '+PTX' IndexError: list index out of range

ERROR: Command errored out with exit status 1: /data/songpeng/.conda/envs/feqsem/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/data/songpeng/project_my/guided_summarization/bart/setup.py'"'"'; file='"'"'/data/songpeng/project_my/guided_summarization/bart/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output. (feqsem) [songpeng@pbcmlg01 bart]$ cd /data/songpeng/project_my/guided_summarization/bart/ (feqsem) [songpeng@pbcmlg01 bart]$ cp -r data-bin /data/songpeng/project_formal/guided_summarization/bart/ (feqsem) [songpeng@pbcmlg01 bart]$ cd /data/songpeng/project_formal/guided_summarization/bart/ (feqsem) [songpeng@pbcmlg01 bart]$ pip install --editable . Obtaining file:///data/songpeng/project_formal/guided_summarization/bart Requirement already satisfied: cffi in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (1.14.5) Requirement already satisfied: cython in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (0.29.23) Requirement already satisfied: numpy in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (1.20.2) Requirement already satisfied: regex in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (2021.4.4) Requirement already satisfied: sacrebleu in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (1.5.1) Requirement already satisfied: torch in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (1.8.1) Requirement already satisfied: tqdm in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from fairseq==0.9.0) (4.61.0) Requirement already satisfied: pycparser in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from cffi->fairseq==0.9.0) (2.20) Requirement already satisfied: portalocker==2.0.0 in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from sacrebleu->fairseq==0.9.0) (2.0.0) Requirement already satisfied: typing_extensions in /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages (from torch->fairseq==0.9.0) (3.7.4.3) Installing collected packages: fairseq Running setup.py develop for fairseq ERROR: Command errored out with exit status 1: command: /data/songpeng/.conda/envs/feqsem/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/data/songpeng/project_formal/guided_summarization/bart/setup.py'"'"'; file='"'"'/data/songpeng/project_formal/guided_summarization/bart/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps cwd: /data/songpeng/project_formal/guided_summarization/bart/ Complete output (133 lines): No CUDA runtime is found, using CUDA_HOME='/cm/shared/apps/cuda11.1/toolkit/11.1.0' running develop running egg_info creating fairseq.egg-info writing fairseq.egg-info/PKG-INFO writing dependency_links to fairseq.egg-info/dependency_links.txt writing entry points to fairseq.egg-info/entry_points.txt writing requirements to fairseq.egg-info/requires.txt writing top-level names to fairseq.egg-info/top_level.txt writing manifest file 'fairseq.egg-info/SOURCES.txt' reading manifest file 'fairseq.egg-info/SOURCES.txt' writing manifest file 'fairseq.egg-info/SOURCES.txt' running build_ext cythoning fairseq/data/data_utils_fast.pyx to fairseq/data/data_utils_fast.cpp cythoning fairseq/data/token_block_utils_fast.pyx to fairseq/data/token_block_utils_fast.cpp building 'fairseq.libbleu' extension creating /data/songpeng/project_formal/guided_summarization/bart/build creating /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8 creating /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq creating /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib creating /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu Emitting ninja build file /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/2] c++ -MMD -MF /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu/module.o.d -pthread -B /data/songpeng/.conda/envs/feqsem/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data/songpeng/.conda/envs/feqsem/include/python3.8 -c -c /data/songpeng/project_formal/guided_summarization/bart/fairseq/clib/libbleu/module.cpp -o /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu/module.o -std=c++11 -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=libbleu -D_GLIBCXX_USE_CXX11_ABI=0 cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [2/2] c++ -MMD -MF /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu/libbleu.o.d -pthread -B /data/songpeng/.conda/envs/feqsem/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data/songpeng/.conda/envs/feqsem/include/python3.8 -c -c /data/songpeng/project_formal/guided_summarization/bart/fairseq/clib/libbleu/libbleu.cpp -o /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu/libbleu.o -std=c++11 -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=libbleu -D_GLIBCXX_USE_CXX11_ABI=0 cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/fairseq g++ -pthread -shared -B /data/songpeng/.conda/envs/feqsem/compiler_compat -L/data/songpeng/.conda/envs/feqsem/lib -Wl,-rpath=/data/songpeng/.conda/envs/feqsem/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu/libbleu.o /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu/module.o -o build/lib.linux-x86_64-3.8/fairseq/libbleu.cpython-38-x86_64-linux-gnu.so building 'fairseq.data.data_utils_fast' extension creating /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data Emitting ninja build file /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/1] c++ -MMD -MF /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/data_utils_fast.o.d -pthread -B /data/songpeng/.conda/envs/feqsem/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/feqsem/include/python3.8 -c -c /data/songpeng/project_formal/guided_summarization/bart/fairseq/data/data_utils_fast.cpp -o /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/data_utils_fast.o -std=c++11 -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=data_utils_fast -D_GLIBCXX_USE_CXX11_ABI=0 cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ In file included from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1944, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /data/songpeng/project_formal/guided_summarization/bart/fairseq/data/data_utils_fast.cpp:626: /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 #warning "Using deprecated NumPy API, disable it with " \ ^~~ creating build/lib.linux-x86_64-3.8/fairseq/data g++ -pthread -shared -B /data/songpeng/.conda/envs/feqsem/compiler_compat -L/data/songpeng/.conda/envs/feqsem/lib -Wl,-rpath=/data/songpeng/.conda/envs/feqsem/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/data_utils_fast.o -o build/lib.linux-x86_64-3.8/fairseq/data/data_utils_fast.cpython-38-x86_64-linux-gnu.so building 'fairseq.data.token_block_utils_fast' extension Emitting ninja build file /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/1] c++ -MMD -MF /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.o.d -pthread -B /data/songpeng/.conda/envs/feqsem/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/feqsem/include/python3.8 -c -c /data/songpeng/project_formal/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp -o /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.o -std=c++11 -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=token_block_utils_fast -D_GLIBCXX_USE_CXX11_ABI=0 cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ In file included from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1944, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /data/songpeng/project_formal/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp:627: /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 #warning "Using deprecated NumPy API, disable it with " \ ^~~ /data/songpeng/project_formal/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp: In function ‘PyArrayObject __pyx_f_7fairseq_4data_22token_block_utils_fast__get_slice_indices_fast(PyArrayObject, PyObject*, int, int, int)’: /data/songpeng/project_formal/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp:3306:36: warning: comparison of integer expressions of different signedness: ‘__pyx_t_7fairseq_4data_22token_block_utils_fast_DTYPE_t’ {aka ‘long int’} and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare] 3306 pyx_t_4 = ((pyx_v_sz_idx < __pyx_t_10) != 0); ~~~^~~~ /data/songpeng/project_formal/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp:3501:36: warning: comparison of integer expressions of different signedness: ‘__pyx_t_7fairseq_4data_22token_block_utils_fast_DTYPE_t’ {aka ‘long int’} and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare] 3501 pyx_t_3 = ((pyx_v_sz_idx < __pyx_t_10) != 0); ~~~^~~~ g++ -pthread -shared -B /data/songpeng/.conda/envs/feqsem/compiler_compat -L/data/songpeng/.conda/envs/feqsem/lib -Wl,-rpath=/data/songpeng/.conda/envs/feqsem/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.o -o build/lib.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.cpython-38-x86_64-linux-gnu.so building 'fairseq.libnat' extension creating /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libnat Emitting ninja build file /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/1] c++ -MMD -MF /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libnat/edit_dist.o.d -pthread -B /data/songpeng/.conda/envs/feqsem/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include/TH -I/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include/THC -I/data/songpeng/.conda/envs/feqsem/include/python3.8 -c -c /data/songpeng/project_formal/guided_summarization/bart/fairseq/clib/libnat/edit_dist.cpp -o /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libnat/edit_dist.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=libnat -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14 cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ In file included from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:140, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:13, from /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3, from /data/songpeng/project_formal/guided_summarization/bart/fairseq/clib/libnat/edit_dist.cpp:9: /data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:83: warning: ignoring #pragma omp parallel [-Wunknown-pragmas] 83 #pragma omp parallel for if ((end - begin) >= grain_size) 
g++ -pthread -shared -B /data/songpeng/.conda/envs/feqsem/compiler_compat -L/data/songpeng/.conda/envs/feqsem/lib -Wl,-rpath=/data/songpeng/.conda/envs/feqsem/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libnat/edit_dist.o -L/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-3.8/fairseq/libnat.cpython-38-x86_64-linux-gnu.so
building 'fairseq.libnat_cuda' extension
creating /data/songpeng/project_formal/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libnat_cuda
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/songpeng/project_formal/guided_summarization/bart/setup.py", line 118, in <module>
    setup(
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run
    self.install_for_development()
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/setuptools/command/develop.py", line 136, in install_for_development
    self.run_command('build_ext')
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 708, in build_extensions
    build_ext.build_extensions(self)
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
    _build_ext.build_extension(self, ext)
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 524, in unix_wrap_ninja_compile
    cuda_post_cflags = unix_cuda_flags(cuda_post_cflags)
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 423, in unix_cuda_flags
    cflags + _get_cuda_arch_flags(cflags))
  File "/data/songpeng/.conda/envs/feqsem/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1561, in _get_cuda_arch_flags
    arch_list[-1] += '+PTX'
IndexError: list index out of range
----------------------------------------

ERROR: Command errored out with exit status 1: /data/songpeng/.conda/envs/feqsem/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/data/songpeng/project_formal/guided_summarization/bart/setup.py'"'"'; file='"'"'/data/songpeng/project_formal/guided_summarization/bart/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output. (feqsem) [songpeng@pbcmlg01 bart]$ tmux a -t gpu0 self.run_command('build_ext') File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run _build_ext.run(self) File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 708, in build_extensions build_ext.build_extensions(self) File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions _build_ext.build_ext.build_extensions(self) File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions self._build_extensions_serial() File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial self.build_extension(ext) File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 196, in build_extension _build_ext.build_extension(self, ext) File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension objects = self.compiler.compile(sources, File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 524, in unix_wrap_ninja_compile cuda_post_cflags = unix_cuda_flags(cuda_post_cflags) File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 423, in unix_cuda_flags cflags + _get_cuda_arch_flags(cflags)) File "/home/songpeng/.conda/envs/fairseq/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1561, in _get_cuda_arch_flags arch_list[-1] += '+PTX' IndexError: list index out of range

Rolling back uninstall of fairseq Moving to /home/songpeng/.conda/envs/fairseq/bin/fairseq-eval-lm from /tmp/pip-uninstall-xk2ohr_a/fairseq-eval-lm Moving to /home/songpeng/.conda/envs/fairseq/bin/fairseq-generate from /tmp/pip-uninstall-xk2ohr_a/fairseq-generate Moving to /home/songpeng/.conda/envs/fairseq/bin/fairseq-hydra-train from /tmp/pip-uninstall-xk2ohr_a/fairseq-hydra-train Moving to /home/songpeng/.conda/envs/fairseq/bin/fairseq-interactive from /tmp/pip-uninstall-xk2ohr_a/fairseq-interactive Moving to /home/songpeng/.conda/envs/fairseq/bin/fairseq-preprocess from /tmp/pip-uninstall-xk2ohr_a/fairseq-preprocess Moving to /home/songpeng/.conda/envs/fairseq/bin/fairseq-score from /tmp/pip-uninstall-xk2ohr_a/fairseq-score Moving to /home/songpeng/.conda/envs/fairseq/bin/fairseq-train from /tmp/pip-uninstall-xk2ohr_a/fairseq-train Moving to /home/songpeng/.conda/envs/fairseq/bin/fairseq-validate from /tmp/pip-uninstall-xk2ohr_a/fairseq-validate Moving to /home/songpeng/.conda/envs/fairseq/lib/python3.8/site-packages/fairseq.egg-link from /tmp/pip-uninstall-io6x89th/fairseq.egg-link ERROR: Command errored out with exit status 1: /home/songpeng/.conda/envs/fairseq/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/data/songpeng/project_my/guided_summarization/bart/setup.py'"'"'; file='"'"'/data/songpeng/project_my/guided_summarization/bart/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

zdou0830 commented 3 years ago

Hi, this seems like an issue in fairseq and maybe this thread would be helpful?

SONGPENG commented 3 years ago

It does not work for me.... I can run the fairseq sample you provided (https://github.com/pytorch/fairseq/blob/master/examples/bart/README.summarization.md) Does it suggest my fairseq is correct?

Can you give more information? 1) After git clone the project, should I run pip install --editable . for the project ?

zdou0830 commented 3 years ago

I used this version of fairseq. If you can install this version, it means that your environment can support our code.

Yes. Also, you shouldn't install the original fairseq, just installing our repo is enough.

SONGPENG commented 3 years ago

Thank you for your patience, but I still have a problem....

I prepared two same enviroments conda create A --clone B. One can run pip install --editable ./ successfully for fairseq . But for this repo, I still receive the error.

The steps are 1) PYTHON: conda create -n A python=3.8 2) PYTORCH: conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia 3) APEX: follow this 4) REPO: git clone this repo 5) cd guided_summarization/bart/ and pip install --editable ./

The error is:

(pytorch) [songpeng@pbcmlg01 bart]$ pip install --editable ./ Obtaining file:///data/songpeng/project/guided_summarization/bart Requirement already satisfied: cffi in /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages (from fairseq==0.9.0) (1.14.5) Requirement already satisfied: cython in /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages (from fairseq==0.9.0) (0.29.23) Requirement already satisfied: numpy in /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages (from fairseq==0.9.0) (1.20.2) Requirement already satisfied: regex in /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages (from fairseq==0.9.0) (2021.4.4) Requirement already satisfied: sacrebleu in /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages (from fairseq==0.9.0) (1.5.1) Requirement already satisfied: torch in /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages (from fairseq==0.9.0) (1.8.1) Requirement already satisfied: tqdm in /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages (from fairseq==0.9.0) (4.61.0) Requirement already satisfied: pycparser in /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages (from cffi->fairseq==0.9.0) (2.20) Requirement already satisfied: portalocker==2.0.0 in /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages (from sacrebleu->fairseq==0.9.0) (2.0.0) Requirement already satisfied: typing_extensions in /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages (from torch->fairseq==0.9.0) (3.7.4.3) Installing collected packages: fairseq Attempting uninstall: fairseq Found existing installation: fairseq 1.0.0a0+2fd9d8a Uninstalling fairseq-1.0.0a0+2fd9d8a: Successfully uninstalled fairseq-1.0.0a0+2fd9d8a Running setup.py develop for fairseq ERROR: Command errored out with exit status 1: command: /data/songpeng/.conda/envs/pytorch/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/data/songpeng/project/guided_summarization/bart/setup.py'"'"'; file='"'"'/data/songpeng/project/guided_summarization/bart/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps cwd: /data/songpeng/project/guided_summarization/bart/ Complete output (106 lines): No CUDA runtime is found, using CUDA_HOME='/cm/shared/apps/cuda11.1/toolkit/11.1.0' running develop running egg_info writing fairseq.egg-info/PKG-INFO writing dependency_links to fairseq.egg-info/dependency_links.txt writing entry points to fairseq.egg-info/entry_points.txt writing requirements to fairseq.egg-info/requires.txt writing top-level names to fairseq.egg-info/top_level.txt reading manifest file 'fairseq.egg-info/SOURCES.txt' writing manifest file 'fairseq.egg-info/SOURCES.txt' running build_ext cythoning fairseq/data/data_utils_fast.pyx to fairseq/data/data_utils_fast.cpp cythoning fairseq/data/token_block_utils_fast.pyx to fairseq/data/token_block_utils_fast.cpp building 'fairseq.libbleu' extension Emitting ninja build file /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. g++ -pthread -shared -B /data/songpeng/.conda/envs/pytorch/compiler_compat -L/data/songpeng/.conda/envs/pytorch/lib -Wl,-rpath=/data/songpeng/.conda/envs/pytorch/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu/libbleu.o /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libbleu/module.o -o build/lib.linux-x86_64-3.8/fairseq/libbleu.cpython-38-x86_64-linux-gnu.so building 'fairseq.data.data_utils_fast' extension Emitting ninja build file /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/1] c++ -MMD -MF /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/data_utils_fast.o.d -pthread -B /data/songpeng/.conda/envs/pytorch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/pytorch/include/python3.8 -c -c /data/songpeng/project/guided_summarization/bart/fairseq/data/data_utils_fast.cpp -o /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/data_utils_fast.o -std=c++11 -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=data_utils_fast -D_GLIBCXX_USE_CXX11_ABI=0 cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ In file included from /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1944, from /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /data/songpeng/project/guided_summarization/bart/fairseq/data/data_utils_fast.cpp:626: /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~ g++ -pthread -shared -B /data/songpeng/.conda/envs/pytorch/compiler_compat -L/data/songpeng/.conda/envs/pytorch/lib -Wl,-rpath=/data/songpeng/.conda/envs/pytorch/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/data_utils_fast.o -o build/lib.linux-x86_64-3.8/fairseq/data/data_utils_fast.cpython-38-x86_64-linux-gnu.so building 'fairseq.data.token_block_utils_fast' extension Emitting ninja build file /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/1] c++ -MMD -MF /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.o.d -pthread -B /data/songpeng/.conda/envs/pytorch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include -I/data/songpeng/.conda/envs/pytorch/include/python3.8 -c -c /data/songpeng/project/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp -o /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.o -std=c++11 -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=token_block_utils_fast -D_GLIBCXX_USE_CXX11_ABI=0 cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ In file included from /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1944, from /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /data/songpeng/project/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp:627: /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~ /data/songpeng/project/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp: In function ‘PyArrayObject __pyx_f_7fairseq_4data_22token_block_utils_fast__get_slice_indices_fast(PyArrayObject, PyObject*, int, int, int)’: /data/songpeng/project/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp:3306:36: warning: comparison of integer expressions of different signedness: ‘pyx_t_7fairseq_4data_22token_block_utils_fast_DTYPE_t’ {aka ‘long int’} and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare] 3306 | pyx_t_4 = ((pyx_v_sz_idx < pyx_t_10) != 0); | ~~~^~~~ /data/songpeng/project/guided_summarization/bart/fairseq/data/token_block_utils_fast.cpp:3501:36: warning: comparison of integer expressions of different signedness: ‘pyx_t_7fairseq_4data_22token_block_utils_fast_DTYPE_t’ {aka ‘long int’} and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare] 3501 | pyx_t_3 = ((pyx_v_sz_idx < pyx_t_10) != 0); | ~~~^~~~ g++ -pthread -shared -B /data/songpeng/.conda/envs/pytorch/compiler_compat -L/data/songpeng/.conda/envs/pytorch/lib -Wl,-rpath=/data/songpeng/.conda/envs/pytorch/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.o -o build/lib.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.cpython-38-x86_64-linux-gnu.so building 'fairseq.libnat' extension Emitting ninja build file /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. g++ -pthread -shared -B /data/songpeng/.conda/envs/pytorch/compiler_compat -L/data/songpeng/.conda/envs/pytorch/lib -Wl,-rpath=/data/songpeng/.conda/envs/pytorch/lib -Wl,--no-as-needed -Wl,--sysroot=/ /data/songpeng/project/guided_summarization/bart/build/temp.linux-x86_64-3.8/fairseq/clib/libnat/edit_dist.o -L/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-3.8/fairseq/libnat.cpython-38-x86_64-linux-gnu.so building 'fairseq.libnat_cuda' extension Traceback (most recent call last): File "", line 1, in File "/data/songpeng/project/guided_summarization/bart/setup.py", line 118, in setup( File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/setuptools/init.py", line 153, in setup return distutils.core.setup(**attrs) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/distutils/core.py", line 148, in setup dist.run_commands() File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run self.install_for_development() File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/setuptools/command/develop.py", line 136, in install_for_development self.run_command('build_ext') File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run _build_ext.run(self) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 708, in build_extensions build_ext.build_extensions(self) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions _build_ext.build_ext.build_extensions(self) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions self._build_extensions_serial() File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial self.build_extension(ext) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 196, in build_extension _build_ext.build_extension(self, ext) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension objects = self.compiler.compile(sources, File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 524, in unix_wrap_ninja_compile cuda_post_cflags = unix_cuda_flags(cuda_post_cflags) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 423, in unix_cuda_flags cflags + _get_cuda_arch_flags(cflags)) File "/data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1561, in _get_cuda_arch_flags arch_list[-1] += '+PTX' IndexError: list index out of range

Rolling back uninstall of fairseq Moving to /data/songpeng/.conda/envs/pytorch/bin/fairseq-eval-lm from /tmp/pip-uninstall-etrt6wr1/fairseq-eval-lm Moving to /data/songpeng/.conda/envs/pytorch/bin/fairseq-generate from /tmp/pip-uninstall-etrt6wr1/fairseq-generate Moving to /data/songpeng/.conda/envs/pytorch/bin/fairseq-hydra-train from /tmp/pip-uninstall-etrt6wr1/fairseq-hydra-train Moving to /data/songpeng/.conda/envs/pytorch/bin/fairseq-interactive from /tmp/pip-uninstall-etrt6wr1/fairseq-interactive Moving to /data/songpeng/.conda/envs/pytorch/bin/fairseq-preprocess from /tmp/pip-uninstall-etrt6wr1/fairseq-preprocess Moving to /data/songpeng/.conda/envs/pytorch/bin/fairseq-score from /tmp/pip-uninstall-etrt6wr1/fairseq-score Moving to /data/songpeng/.conda/envs/pytorch/bin/fairseq-train from /tmp/pip-uninstall-etrt6wr1/fairseq-train Moving to /data/songpeng/.conda/envs/pytorch/bin/fairseq-validate from /tmp/pip-uninstall-etrt6wr1/fairseq-validate Moving to /data/songpeng/.conda/envs/pytorch/lib/python3.8/site-packages/fairseq.egg-link from /tmp/pip-uninstall-un3pd2ki/fairseq.egg-link ERROR: Command errored out with exit status 1: /data/songpeng/.conda/envs/pytorch/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/data/songpeng/project/guided_summarization/bart/setup.py'"'"'; file='"'"'/data/songpeng/project/guided_summarization/bart/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

zdou0830 commented 3 years ago

Hi, sorry that I'm not familiar with the fairseq installing procedure either and may just be able to give you some general suggestions:

I saw the error message No CUDA runtime is found, using CUDA_HOME='/cm/shared/apps/cuda11.1/toolkit/11.1.0'. If you import torch and run torch.cuda.is_available() inside container, does it detect the GPU?

Maybe you can try not to install apex?

SONGPENG commented 3 years ago

I migrate the guidxxx.py files to the corresponding directory of an available fairseq. It seems to work temporarily.

Helicqin commented 3 years ago

I migrate the guidxxx.py files to the corresponding directory of an available fairseq. It seems to work temporarily.

Hi, I'm running into the same issue. May I ask what is "migrate the guidxxx.py files to the corresponding directory of an available fairseq"? I guess you first install a new fairseq, then move Gsum-related code to the installed fairseq? Thanks!

SONGPENG commented 3 years ago

I migrate the guidxxx.py files to the corresponding directory of an available fairseq. It seems to work temporarily.

Hi, I'm running into the same issue. May I ask what is "migrate the guidxxx.py files to the corresponding directory of an available fairseq"? I guess you first install a new fairseq, then move Gsum-related code to the installed fairseq? Thanks!

Hi, that's exactly what I did. And it may help you https://github.com/neulab/guided_summarization/issues/14#issuecomment-858437170

JaniceXiong commented 3 years ago

I migrate the guidxxx.py files to the corresponding directory of an available fairseq. It seems to work temporarily.

Hi, I meet the same issue. And Following #14, I firstly install fairseq and try to copy Gsum-releated code to location where fairseq installed. Though I copy data/guided_language_pair_dataset.py, I still meet the error below image Do you have any idea about how to fix it? Thanks a lot! I copy:

SONGPENG commented 3 years ago

I migrate the guidxxx.py files to the corresponding directory of an available fairseq. It seems to work temporarily.

Hi, I meet the same issue. And Following #14, I firstly install fairseq and try to copy Gsum-releated code to location where fairseq installed. Though I copy data/guided_language_pair_dataset.py, I still meet the error below image Do you have any idea about how to fix it? Thanks a lot! I copy:

  • models/bart/guided_hub_interface.py
  • models/bart/guided_model.py
  • tasks/guided_translation.py
  • data/guided_language_pair_dataset.py

Perhaps you should edit the file fairseq\fairseq\data_init_.py

add code : from .guided_language_pair_dataset import GuidedLanguagePairDataset

and add 'GuidedLanguagePairDataset' into list all like below

all = [ 'AppendTokenDataset', 'BacktranslationDataset', 'BaseWrapperDataset', 'ColorizeDataset', 'ConcatDataset', 'ConcatSentencesDataset', 'CountingIterator', 'DenoisingDataset', 'Dictionary', 'EpochBatchIterator', 'FairseqDataset', 'FairseqIterableDataset', 'GroupedIterator', 'IdDataset', 'IndexedCachedDataset', 'IndexedDataset', 'IndexedRawTextDataset', 'LanguagePairDataset', 'LeftPadDataset', 'ListDataset', 'LMContextWindowDataset', 'LRUCacheDataset', 'MaskTokensDataset', 'MMapIndexedDataset', 'MonolingualDataset', 'MultiCorpusSampledDataset', 'NestedDictionaryDataset', 'NoisingDataset', 'NumelDataset', 'NumSamplesDataset', 'OffsetTokensDataset', 'PadDataset', 'PrependDataset', 'PrependTokenDataset', 'ReplaceDataset', 'RollDataset', 'FileAudioDataset', 'RawLabelDataset', 'ResamplingDataset', 'RightPadDataset', 'RoundRobinZipDatasets', 'ShardedDataset', 'ShardedIterator', 'SortDataset', 'StripTokenDataset', 'SubsampleDataset', 'TokenBlockDataset', 'TransformEosDataset', 'TransformEosLangPairDataset', 'TruncateDataset', 'TruncatedDictionary', 'GuidedLanguagePairDataset' ]

JaniceXiong commented 3 years ago

I migrate the guidxxx.py files to the corresponding directory of an available fairseq. It seems to work temporarily.

Hi, I meet the same issue. And Following #14, I firstly install fairseq and try to copy Gsum-releated code to location where fairseq installed. Though I copy data/guided_language_pair_dataset.py, I still meet the error below image Do you have any idea about how to fix it? Thanks a lot! I copy:

  • models/bart/guided_hub_interface.py
  • models/bart/guided_model.py
  • tasks/guided_translation.py
  • data/guided_language_pair_dataset.py

Perhaps you should edit the file fairseq\fairseq\datainit.py

add code : from .guided_language_pair_dataset import GuidedLanguagePairDataset

and add 'GuidedLanguagePairDataset' into list all like below

all = [ 'AppendTokenDataset', 'BacktranslationDataset', 'BaseWrapperDataset', 'ColorizeDataset', 'ConcatDataset', 'ConcatSentencesDataset', 'CountingIterator', 'DenoisingDataset', 'Dictionary', 'EpochBatchIterator', 'FairseqDataset', 'FairseqIterableDataset', 'GroupedIterator', 'IdDataset', 'IndexedCachedDataset', 'IndexedDataset', 'IndexedRawTextDataset', 'LanguagePairDataset', 'LeftPadDataset', 'ListDataset', 'LMContextWindowDataset', 'LRUCacheDataset', 'MaskTokensDataset', 'MMapIndexedDataset', 'MonolingualDataset', 'MultiCorpusSampledDataset', 'NestedDictionaryDataset', 'NoisingDataset', 'NumelDataset', 'NumSamplesDataset', 'OffsetTokensDataset', 'PadDataset', 'PrependDataset', 'PrependTokenDataset', 'ReplaceDataset', 'RollDataset', 'FileAudioDataset', 'RawLabelDataset', 'ResamplingDataset', 'RightPadDataset', 'RoundRobinZipDatasets', 'ShardedDataset', 'ShardedIterator', 'SortDataset', 'StripTokenDataset', 'SubsampleDataset', 'TokenBlockDataset', 'TransformEosDataset', 'TransformEosLangPairDataset', 'TruncateDataset', 'TruncatedDictionary', 'GuidedLanguagePairDataset' ]

wow, Thanks, it works! @SONGPENG But I meet anothor OOM error when loading pretrained model and finetuning on my own dataset. image Many issues in original fairseq show changing UPDATE_FREQ and TOTAL_NUM_UPDATES can reduce the batch size to solve this problem. Did you change the settings in z_train.sh?

SONGPENG commented 3 years ago

I migrate the guidxxx.py files to the corresponding directory of an available fairseq. It seems to work temporarily.

Hi, I meet the same issue. And Following #14, I firstly install fairseq and try to copy Gsum-releated code to location where fairseq installed. Though I copy data/guided_language_pair_dataset.py, I still meet the error below image Do you have any idea about how to fix it? Thanks a lot! I copy:

  • models/bart/guided_hub_interface.py
  • models/bart/guided_model.py
  • tasks/guided_translation.py
  • data/guided_language_pair_dataset.py

Perhaps you should edit the file fairseq\fairseq\datainit.py add code : from .guided_language_pair_dataset import GuidedLanguagePairDataset and add 'GuidedLanguagePairDataset' into list all like below all = [ 'AppendTokenDataset', 'BacktranslationDataset', 'BaseWrapperDataset', 'ColorizeDataset', 'ConcatDataset', 'ConcatSentencesDataset', 'CountingIterator', 'DenoisingDataset', 'Dictionary', 'EpochBatchIterator', 'FairseqDataset', 'FairseqIterableDataset', 'GroupedIterator', 'IdDataset', 'IndexedCachedDataset', 'IndexedDataset', 'IndexedRawTextDataset', 'LanguagePairDataset', 'LeftPadDataset', 'ListDataset', 'LMContextWindowDataset', 'LRUCacheDataset', 'MaskTokensDataset', 'MMapIndexedDataset', 'MonolingualDataset', 'MultiCorpusSampledDataset', 'NestedDictionaryDataset', 'NoisingDataset', 'NumelDataset', 'NumSamplesDataset', 'OffsetTokensDataset', 'PadDataset', 'PrependDataset', 'PrependTokenDataset', 'ReplaceDataset', 'RollDataset', 'FileAudioDataset', 'RawLabelDataset', 'ResamplingDataset', 'RightPadDataset', 'RoundRobinZipDatasets', 'ShardedDataset', 'ShardedIterator', 'SortDataset', 'StripTokenDataset', 'SubsampleDataset', 'TokenBlockDataset', 'TransformEosDataset', 'TransformEosLangPairDataset', 'TruncateDataset', 'TruncatedDictionary', 'GuidedLanguagePairDataset' ]

wow, Thanks, it works! @SONGPENG But I meet anothor OOM error when loading pretrained model and finetuning on my own dataset. image Many issues in original fairseq show changing UPDATE_FREQ and TOTAL_NUM_UPDATES can reduce the batch size to solve this problem. Did you change the settings in z_train.sh?

Sorry, that's not what happened to me. But I think changing the settings in z_train.sh is workable and you can give it a try. : D