Closed mostafa-saad closed 3 years ago
Hi. Your issue comes from treesitter, which we use to parse the source code. It seems to be close to this issue, which they fixed on github but I think not on the version we install with pip https://github.com/tree-sitter/py-tree-sitter/issues/43. It seems to affect new versions of python. Although our code probably works with other versions of python, I updated the environment installation script and the CI to use python 3.6.9, which is the one we used to train the models, to avoid this kind of issues in the future. I double checked and tree-sitter works on a clean 3.6.9 installation.
You can either run the install_env script again to install everything with python 3.6.9 or try to install the new version of tree-sitter which probably fixes this issue.
I tried both solutions, but the same issue @brozi
/usr/bin/ld: cannot find -lc++ collect2: error: ld returned 1 exit status Traceback (most recent call last): File "/home/moustafa/system-installs1/anaconda3/envs/codeGen_env_3/lib/python3.6/distutils/unixccompiler.py", line 197, in link self.spawn(linker + ld_args) File "/home/moustafa/system-installs1/anaconda3/envs/codeGen_env_3/lib/python3.6/distutils/ccompiler.py", line 909, in spawn spawn(cmd, dry_run=self.dry_run) File "/home/moustafa/system-installs1/anaconda3/envs/codeGen_env_3/lib/python3.6/distutils/spawn.py", line 36, in spawn _spawn_posix(cmd, search_path, dry_run=dry_run) File "/home/moustafa/system-installs1/anaconda3/envs/codeGen_env_3/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix % (cmd, exit_status)) distutils.errors.DistutilsExecError: command 'cc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/moustafa/system-installs1/anaconda3/envs/codeGen_env_3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/moustafa/system-installs1/anaconda3/envs/codeGen_env_3/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/moustafa/workspaces/git_web/facebook_transcoder/CodeGen/codegen_sources/model/translate.py", line 258, in
my conda env
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
antlr4-python3-runtime 4.8 pypi_0 pypi
appdirs 1.4.4 pypi_0 pypi
astunparse 1.6.3 pyhd8ed1ab_0 conda-forge
attrs 21.2.0 pypi_0 pypi
black 19.10b0 pypi_0 pypi
blas 1.0 mkl
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2021.10.8 ha878542_0 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.4 pyh9f0ad1d_0 conda-forge
cudatoolkit 11.1.74 h6bb024c_0 nvidia
cython 0.29.24 pypi_0 pypi
dataclasses 0.8 pyh787bdff_2 conda-forge
fastbpe 0.1.0 pypi_0 pypi
ffmpeg 4.3 hf484d3e_0 pytorch
filelock 3.4.0 pyhd8ed1ab_0 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
huggingface_hub 0.2.0 pyhd8ed1ab_0 conda-forge
hydra-core 1.1.1 pypi_0 pypi
importlib-metadata 3.10.1 py36h5fab9bb_0 conda-forge
importlib-resources 5.4.0 pypi_0 pypi
importlib_metadata 3.10.1 hd8ed1ab_0 conda-forge
intel-openmp 2021.4.0 h06a4308_3561
javalang 0.13.0 pypi_0 pypi
jbig 2.1 h7f98852_2003 conda-forge
joblib 1.1.0 pyhd8ed1ab_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
lame 3.100 h7f98852_1001 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
lerc 3.0 h9c3ff4c_0 conda-forge
libblas 3.9.0 12_linux64_mkl conda-forge
libcblas 3.9.0 12_linux64_mkl conda-forge
libdeflate 1.8 h7f98852_0 conda-forge
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-ng 11.2.0 h1d223b6_11 conda-forge
libgfortran-ng 11.2.0 h69a702a_11 conda-forge
libgfortran5 11.2.0 h5c6108e_11 conda-forge
libgomp 11.2.0 h1d223b6_11 conda-forge
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 12_linux64_mkl conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge
libtiff 4.3.0 h6f004c6_2 conda-forge
libuv 1.42.0 h7f98852_0 conda-forge
libwebp-base 1.2.1 h7f98852_0 conda-forge
libzlib 1.2.11 h36c2ea0_1013 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
mkl 2021.4.0 h06a4308_640
ncurses 6.2 h58526e2_4 conda-forge
nettle 3.6 he412f7d_0 conda-forge
numpy 1.19.5 py36hfc0c790_2 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
omegaconf 2.1.1 pypi_0 pypi
openh264 2.1.1 h780b84a_0 conda-forge
openssl 1.1.1l h7f98852_0 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pathspec 0.9.0 pypi_0 pypi
pillow 7.2.0 py36h8328e55_2 conda-forge
pip 21.3.1 pyhd8ed1ab_0 conda-forge
ply 3.11 py_1 conda-forge
psutil 5.8.0 pypi_0 pypi
pyparsing 3.0.6 pyhd8ed1ab_0 conda-forge
python 3.6.9 h9d8adfe_0_cpython conda-forge
python_abi 3.6 2_cp36m conda-forge
pytorch 1.10.0 py3.6_cuda11.1_cudnn8.0.5_0 pytorch
pytorch-mutex 1.0 cuda pytorch
pyyaml 5.4.1 py36h8f6f2f9_1 conda-forge
readline 8.1 h46c0cb4_0 conda-forge
regex 2021.8.3 py36h7f8727e_0
requests 2.12.5 py36_0 conda-forge
sacrebleu 1.2.11 pypi_0 pypi
sacremoses 0.0.46 pyhd8ed1ab_0 conda-forge
scikit-learn 0.24.2 py36hc89565f_1 conda-forge
scipy 1.5.3 py36h81d768a_1 conda-forge
setuptools 58.0.4 py36h5fab9bb_2 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
slimit 0.8.1 py36h9f0ad1d_2 conda-forge
sqlite 3.37.0 h9cd32fc_0 conda-forge
stringcase 1.2.0 py_0 conda-forge
submitit 1.2.1 pyh44b312d_0 conda-forge
threadpoolctl 3.0.0 pyh8a188c0_0 conda-forge
tk 8.6.11 h27826a3_1 conda-forge
tokenizers 0.10.3 py36h589faa6_1 conda-forge
toml 0.10.2 pypi_0 pypi
torchaudio 0.10.0 py36_cu111 pytorch
torchvision 0.11.1 py36_cu111 pytorch
tqdm 4.62.3 pyhd8ed1ab_0 conda-forge
transformers 4.12.5 pyhd8ed1ab_0 conda-forge
tree-sitter 0.19.0 pypi_0 pypi
typed-ast 1.5.0 pypi_0 pypi
typing 3.7.4.3 pypi_0 pypi
typing-extensions 4.0.1 hd8ed1ab_0 conda-forge
typing_extensions 4.0.1 pyha770c72_0 conda-forge
wheel 0.37.0 pyhd8ed1ab_1 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
zipp 3.6.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h36c2ea0_1013 conda-forge
zstd 1.5.0 ha95c52a_0 conda-forge
Ok, I found my way
ldconfig -p | grep libc++ sudo ln -s /usr/lib/x86_64-linux-gnu/libc++.so.1 /usr/lib/x86_64-linux-gnu/libc++.so
I am trying to use the pretrained model to transform from C++ to Java, but with no luck after installations with this error
` python -m codegen_sources.model.translate --src_lang cpp --tgt_lang java --model_path models/TransCoder_model_1.pth --beam_size 1 < zcpp_sample.cpp adding to path <>/workspaces/git_web/facebook_transcoder/CodeGen INFO - 10/02/21 08:26:08 - 0:00:04 - ============ Model Reloading INFO - 10/02/21 08:26:08 - 0:00:04 - Reloading encoder from models/TransCoder_model_1.pth ... WARNING - 10/02/21 08:26:10 - 0:00:06 - Lang cpp_sa matched to pretrained cpp_sa lang embedding. WARNING - 10/02/21 08:26:10 - 0:00:06 - Lang java_sa matched to pretrained java_sa lang embedding. WARNING - 10/02/21 08:26:10 - 0:00:06 - Lang python_sa matched to pretrained python_sa lang embedding. WARNING - 10/02/21 08:26:10 - 0:00:06 - The size of position embeddings in current model is 2048, the size of reloaded is 1024. need to repeat last positions 1024 times. INFO - 10/02/21 08:26:10 - 0:00:07 - Reloading decoders from models/TransCoder_model_1.pth ... WARNING - 10/02/21 08:26:11 - 0:00:08 - Lang cpp_sa matched to pretrained cpp_sa lang embedding. WARNING - 10/02/21 08:26:11 - 0:00:08 - Lang java_sa matched to pretrained java_sa lang embedding. WARNING - 10/02/21 08:26:11 - 0:00:08 - Lang python_sa matched to pretrained python_sa lang embedding. WARNING - 10/02/21 08:26:11 - 0:00:08 - The size of position embeddings in current model is 2048, the size of reloaded is 1024. need to repeat last positions 1024 times. INFO - 10/02/21 08:26:11 - 0:00:08 - Number of parameters (encoder): 143239641 INFO - 10/02/21 08:26:11 - 0:00:08 - Number of parameters (decoders): 168442329 INFO - 10/02/21 08:26:11 - 0:00:08 - Number of decoders: 1
Input cpp function:
include
using namespace std;
int main() { cout<<"My First Program. Helllllllo."<<endl;
} /usr/bin/ld: cannot find -lc++ collect2: error: ld returned 1 exit status Traceback (most recent call last): File "<>/anaconda3/envs/pyt/lib/python3.9/distutils/unixccompiler.py", line 206, in link self.spawn(linker + ld_args) File "<>/anaconda3/envs/pyt/lib/python3.9/distutils/ccompiler.py", line 910, in spawn spawn(cmd, dry_run=self.dry_run) File "<>/anaconda3/envs/pyt/lib/python3.9/distutils/spawn.py", line 91, in spawn raise DistutilsExecError( distutils.errors.DistutilsExecError: command '/usr/bin/cc' failed with exit code 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "<>/anaconda3/envs/pyt/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "<>/anaconda3/envs/pyt/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "<>/workspaces/git_web/facebook_transcoder/CodeGen/codegen_sources/model/translate.py", line 254, in
output = translator.translate(
File "<>/workspaces/git_web/facebook_transcoder/CodeGen/codegen_sources/model/translate.py", line 139, in translate
src_lang_processor = LangProcessor.processors[lang1](
File "<>/workspaces/git_web/facebook_transcoder/CodeGen/codegen_sources/preprocessing/lang_processors/cpp_processor.py", line 25, in init
super().init(
File "<>/workspaces/git_web/facebook_transcoder/CodeGen/codegen_sources/preprocessing/lang_processors/tree_sitter_processor.py", line 40, in init
self.create_treesiter_parser()
File "<>/workspaces/git_web/facebook_transcoder/CodeGen/codegen_sources/preprocessing/lang_processors/tree_sitter_processor.py", line 48, in create_treesiter_parser
Language.build_library(
File "<>/anaconda3/envs/pyt/lib/python3.9/site-packages/tree_sitter/init.py", line 72, in build_library
compiler.link_shared_object(object_paths, output_path)
File "<>/anaconda3/envs/pyt/lib/python3.9/distutils/ccompiler.py", line 713, in link_shared_object
self.link(CCompiler.SHARED_OBJECT, objects,
File "<>/anaconda3/envs/pyt/lib/python3.9/distutils/unixccompiler.py", line 208, in link
raise LinkError(msg)
distutils.errors.LinkError: command '/usr/bin/cc' failed with exit code 1
`
I run the following command in my environment ' cc --version cc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 '