search

RUCAIBox / TextBox

TextBox 2.0 is a text generation library with pre-trained language models
https://github.com/RUCAIBox/TextBox
MIT License
1.08k stars 117 forks source link

2.0.0版本运行install.sh报错 #319

Closed Foehnc closed 1 year ago

Foehnc commented 1 year ago

ERROR: Could not find a version that satisfies the requirement rouge-score>=0.1.2 (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4)

rouge-score装不上0.1.2版本请问怎么解决

StevenTang1998 commented 1 year ago

你可以尝试换个pip的源，比如清华源

Foehnc commented 1 year ago

你可以尝试换个pip的源，比如清华源

感谢回复，但是我又遇到了新的error请问下这个怎么解决

rm: cannot remove ‘/home/cqy/.files2rouge/data/WordNet-2.0-Exceptions/WordNet-2.0.exc.db’: No such file or directory Can't locate DB_File.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /home/cqy/.files2rouge/data/WordNet-2.0-Exceptions/buildExeptionDB.pl line 2. BEGIN failed--compilation aborted at /home/cqy/.files2rouge/data/WordNet-2.0-Exceptions/buildExeptionDB.pl line 2. chmod: cannot operate on dangling symlink ‘/home/cqy/.files2rouge/data/WordNet-2.0.exc.db’ chmod: cannot access ‘/home/cqy/.files2rouge/data/WordNet-2.0-Exceptions/WordNet-2.0.exc.db’: No such file or directory

StevenTang1998 commented 1 year ago

请直接运行install.sh进行安装，而不要只运行其中的一部分

Foehnc commented 1 year ago

请直接运行install.sh进行安装，而不要只运行其中的一部分

我就是直接运行的install.sh，只截取了报错的部分

StevenTang1998 commented 1 year ago

那请展示从运行之后所有的输出

StevenTang1998 commented 1 year ago

或者如果files2rouge已经正确安装，则可以忽略上述命令

Foehnc commented 1 year ago

-> ./install.sh

████████╗███████╗██╗ ██╗████████╗██████╗ ██████╗ ██╗ ██╗ ╚══██╔══╝██╔════╝╚██╗██╔╝╚══██╔══╝██╔══██╗██╔═══██╗╚██╗██╔╝ ██║ █████╗ ╚███╔╝ ██║ ██████╔╝██║ ██║ ╚███╔╝ ██║ ██╔══╝ ██╔██╗ ██║ ██╔══██╗██║ ██║ ██╔██╗ ██║ ███████╗██╔╝ ██╗ ██║ ██████╔╝╚██████╔╝██╔╝ ██╗ ╚═╝ ╚══════╝╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝

A modified version of transformers will be installed to python environment. Create a new conda environment (TextBox)? (y/n) y Creating conda environment named TextBox (python=3.8) ... Collecting package metadata (current_repodata.json): done Solving environment: done

==> WARNING: A newer version of conda exists. <== current version: 4.12.0 latest version: 22.11.1

Please update conda by running

$ conda update -n base -c defaults conda

Package Plan

environment location: /home/cqy/anaconda3/envs/TextBox

added / updated specs:

The following NEW packages will be INSTALLED:

_libgcc_mutex anaconda/pkgs/main/linux-64::_libgcc_mutex-0.1-main _openmp_mutex anaconda/pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu ca-certificates anaconda/pkgs/main/linux-64::ca-certificates-2022.10.11-h06a4308_0 certifi anaconda/pkgs/main/linux-64::certifi-2022.12.7-py38h06a4308_0 ld_impl_linux-64 anaconda/pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1 libffi anaconda/pkgs/main/linux-64::libffi-3.4.2-h6a678d5_6 libgcc-ng anaconda/pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1 libgomp anaconda/pkgs/main/linux-64::libgomp-11.2.0-h1234567_1 libstdcxx-ng anaconda/pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1 ncurses anaconda/pkgs/main/linux-64::ncurses-6.3-h5eee18b_3 openssl anaconda/pkgs/main/linux-64::openssl-1.1.1s-h7f8727e_0 pip anaconda/pkgs/main/linux-64::pip-22.3.1-py38h06a4308_0 python anaconda/pkgs/main/linux-64::python-3.8.15-h7a1cb2a_2 readline anaconda/pkgs/main/linux-64::readline-8.2-h5eee18b_0 setuptools anaconda/pkgs/main/linux-64::setuptools-65.6.3-py38h06a4308_0 sqlite anaconda/pkgs/main/linux-64::sqlite-3.40.1-h5082296_0 tk anaconda/pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0 wheel anaconda/pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0 xz anaconda/pkgs/main/linux-64::xz-5.2.8-h5eee18b_0 zlib anaconda/pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0

Proceed ([y]/n)?

Preparing transaction: done Verifying transaction: done Executing transaction: done

To activate this environment, use

$ conda activate TextBox

To deactivate an active environment, use

$ conda deactivate

no change /home/cqy/anaconda3/condabin/conda no change /home/cqy/anaconda3/bin/conda no change /home/cqy/anaconda3/bin/conda-env no change /home/cqy/anaconda3/bin/activate no change /home/cqy/anaconda3/bin/deactivate no change /home/cqy/anaconda3/etc/profile.d/conda.sh no change /home/cqy/anaconda3/etc/fish/conf.d/conda.fish no change /home/cqy/anaconda3/shell/condabin/Conda.psm1 no change /home/cqy/anaconda3/shell/condabin/conda-hook.ps1 no change /home/cqy/anaconda3/lib/python3.8/site-packages/xontrib/conda.xsh no change /home/cqy/anaconda3/etc/profile.d/conda.csh no change /home/cqy/.bashrc No action taken. Installation may take a few minutes. Installing torch ... Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: done

==> WARNING: A newer version of conda exists. <== current version: 4.12.0 latest version: 22.11.1

Please update conda by running

$ conda update -n base -c defaults conda

Package Plan

environment location: /home/cqy/anaconda3/envs/TextBox

added / updated specs:

The following NEW packages will be INSTALLED:

blas anaconda/pkgs/main/linux-64::blas-1.0-mkl brotlipy anaconda/pkgs/main/linux-64::brotlipy-0.7.0-py38h27cfd23_1003 bzip2 anaconda/pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0 cffi anaconda/pkgs/main/linux-64::cffi-1.15.1-py38h5eee18b_3 charset-normalizer anaconda/pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0 cryptography anaconda/pkgs/main/linux-64::cryptography-38.0.4-py38h9ce1e76_0 cudatoolkit anaconda/pkgs/main/linux-64::cudatoolkit-10.2.89-hfd86e86_1 ffmpeg pytorch/linux-64::ffmpeg-4.3-hf484d3e_0 flit-core anaconda/pkgs/main/noarch::flit-core-3.6.0-pyhd3eb1b0_0 freetype anaconda/pkgs/main/linux-64::freetype-2.12.1-h4a9f257_0 giflib anaconda/pkgs/main/linux-64::giflib-5.2.1-h7b6447c_0 gmp anaconda/pkgs/main/linux-64::gmp-6.2.1-h295c915_3 gnutls anaconda/pkgs/main/linux-64::gnutls-3.6.15-he1e5248_0 idna anaconda/pkgs/main/linux-64::idna-3.4-py38h06a4308_0 intel-openmp anaconda/pkgs/main/linux-64::intel-openmp-2021.4.0-h06a4308_3561 jpeg anaconda/pkgs/main/linux-64::jpeg-9e-h7f8727e_0 lame anaconda/pkgs/main/linux-64::lame-3.100-h7b6447c_0 lcms2 anaconda/pkgs/main/linux-64::lcms2-2.12-h3be6417_0 lerc anaconda/pkgs/main/linux-64::lerc-3.0-h295c915_0 libdeflate anaconda/pkgs/main/linux-64::libdeflate-1.8-h7f8727e_5 libiconv anaconda/pkgs/main/linux-64::libiconv-1.16-h7f8727e_2 libidn2 anaconda/pkgs/main/linux-64::libidn2-2.3.2-h7f8727e_0 libpng anaconda/pkgs/main/linux-64::libpng-1.6.37-hbc83047_0 libtasn1 anaconda/pkgs/main/linux-64::libtasn1-4.16.0-h27cfd23_0 libtiff anaconda/pkgs/main/linux-64::libtiff-4.4.0-hecacb30_2 libunistring anaconda/pkgs/main/linux-64::libunistring-0.9.10-h27cfd23_0 libwebp anaconda/pkgs/main/linux-64::libwebp-1.2.4-h11a3e52_0 libwebp-base anaconda/pkgs/main/linux-64::libwebp-base-1.2.4-h5eee18b_0 lz4-c anaconda/pkgs/main/linux-64::lz4-c-1.9.4-h6a678d5_0 mkl anaconda/pkgs/main/linux-64::mkl-2021.4.0-h06a4308_640 mkl-service anaconda/pkgs/main/linux-64::mkl-service-2.4.0-py38h7f8727e_0 mkl_fft anaconda/pkgs/main/linux-64::mkl_fft-1.3.1-py38hd3c417c_0 mkl_random anaconda/pkgs/main/linux-64::mkl_random-1.2.2-py38h51133e4_0 nettle anaconda/pkgs/main/linux-64::nettle-3.7.3-hbbd107a_1 numpy anaconda/pkgs/main/linux-64::numpy-1.23.5-py38h14f4228_0 numpy-base anaconda/pkgs/main/linux-64::numpy-base-1.23.5-py38h31eccc5_0 openh264 anaconda/pkgs/main/linux-64::openh264-2.1.1-h4ff587b_0 pillow anaconda/pkgs/main/linux-64::pillow-9.3.0-py38hace64e9_1 pycparser anaconda/pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0 pyopenssl anaconda/pkgs/main/noarch::pyopenssl-22.0.0-pyhd3eb1b0_0 pysocks anaconda/pkgs/main/linux-64::pysocks-1.7.1-py38h06a4308_0 pytorch pytorch/linux-64::pytorch-1.12.1-py3.8_cuda10.2_cudnn7.6.5_0 pytorch-mutex pytorch/noarch::pytorch-mutex-1.0-cuda requests anaconda/pkgs/main/linux-64::requests-2.28.1-py38h06a4308_0 six anaconda/pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1 torchaudio pytorch/linux-64::torchaudio-0.12.1-py38_cu102 torchvision pytorch/linux-64::torchvision-0.13.1-py38_cu102 typing_extensions anaconda/pkgs/main/linux-64::typing_extensions-4.4.0-py38h06a4308_0 urllib3 anaconda/pkgs/main/linux-64::urllib3-1.26.13-py38h06a4308_0 zstd anaconda/pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0

Proceed ([y]/n)?

Preparing transaction: done Verifying transaction: done Executing transaction: done Installing requirements ... Looking in indexes: https://mirror.baidu.com/pypi/simple Requirement already satisfied: numpy in /home/cqy/anaconda3/envs/TextBox/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (1.23.5) Collecting matplotlib Using cached https://mirror.baidu.com/pypi/packages/c4/49/80590115bcae4bd8ffb30fc52df459d785dd0f1542312c8c996b7206747c/matplotlib-3.6.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.4 MB) Collecting tqdm Using cached https://mirror.baidu.com/pypi/packages/47/bb/849011636c4da2e44f1253cd927cfb20ada4374d8b3a4e425416e84900cc/tqdm-4.64.1-py2.py3-none-any.whl (78 kB) Collecting nltk Using cached https://mirror.baidu.com/pypi/packages/a6/0a/0d20d2c0f16be91b9fa32a77b76c60f9baf6eba419e5ef5deca17af9c582/nltk-3.8.1-py3-none-any.whl (1.5 MB) Collecting pyyaml Using cached https://mirror.baidu.com/pypi/packages/d7/42/7ad4b6d67a16229496d4f6e74201bdbebcf4bc1e87d5a70c9297d4961bd2/PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB) Collecting sentencepiece Using cached https://mirror.baidu.com/pypi/packages/68/91/ded0f64f90abfc5413c620fc345a0aef1e7ff5addda8704cc6b3bf589c64/sentencepiece-0.1.96-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) Collecting pycocoevalcap Using cached https://mirror.baidu.com/pypi/packages/08/f9/466f289f1628296b5e368940f89e3cfcfb066d15ddc02ff536dc532b1c93/pycocoevalcap-1.2-py3-none-any.whl (104.3 MB) ERROR: Could not find a version that satisfies the requirement rouge-score>=0.1.2 (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4) ERROR: No matching distribution found for rouge-score>=0.1.2 Installing requirements (fast-bleu) ... Installing requirements (fastseq) ... Running command git clone --filter=blob:none --quiet https://github.com/microsoft/fastseq.git /tmp/pip-req-build-pztflk0s Installing requirements (rouge) ... Running command git clone --filter=blob:none --quiet https://github.com/pltrdy/pyrouge.git /tmp/pip-req-build-3kha0188 Cloning into 'files2rouge'... remote: Enumerating objects: 278, done. remote: Counting objects: 100% (20/20), done. remote: Compressing objects: 100% (14/14), done. remote: Total 278 (delta 9), reused 11 (delta 6), pack-reused 258 Receiving objects: 100% (278/278), 212.16 KiB | 1.02 MiB/s, done. Resolving deltas: 100% (133/133), done. /home/cqy/anaconda3/envs/TextBox/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( /home/cqy/anaconda3/envs/TextBox/lib/python3.8/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools. warnings.warn( zip_safe flag not set; analyzing archive contents... files2rouge.pycache.settings.cpython-38: module references file rm: cannot remove ‘/home/cqy/.files2rouge/data/WordNet-2.0-Exceptions/WordNet-2.0.exc.db’: No such file or directory Can't locate DB_File.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /home/cqy/.files2rouge/data/WordNet-2.0-Exceptions/buildExeptionDB.pl line 2. BEGIN failed--compilation aborted at /home/cqy/.files2rouge/data/WordNet-2.0-Exceptions/buildExeptionDB.pl line 2. chmod: cannot operate on dangling symlink ‘/home/cqy/.files2rouge/data/WordNet-2.0.exc.db’ chmod: cannot access ‘/home/cqy/.files2rouge/data/WordNet-2.0-Exceptions/WordNet-2.0.exc.db’: No such file or directory WARNING: Skipping py-rouge as it is not installed.

我看这报错是不是files2rouge不能用setup.py安装啊？

StevenTang1998 commented 1 year ago

如果files2rouge已经正确安装，则可以忽略上述命令

Foehnc commented 1 year ago

如果files2rouge已经正确安装，则可以忽略上述命令

请问下怎么判断files2rouge正确安装了啊，只要有那个文件夹就行了嘛？还有就是~/.files2rouge用不到嘛

StevenTang1998 commented 1 year ago

命令行中输入files2rouge，显示如下字段

usage: files2rouge [-h] [-v] [-n ROUGE_N] [-nol] [-a ARGS] [-s SAVETO] [-e EOS] [-m] [-nm]
                   [-ir] [-is]
                   reference summary
Foehnc commented 1 year ago

命令行中输入files2rouge，显示如下字段

usage: files2rouge [-h] [-v] [-n ROUGE_N] [-nol] [-a ARGS] [-s SAVETO] [-e EOS] [-m] [-nm]
                   [-ir] [-is]
                   reference summary
usage: files2rouge [-h] [-v] [-n ROUGE_N] [-nol] [-a ARGS] [-s SAVETO] [-e EOS] [-m] [-nm] [-ir] [-is] reference summary
files2rouge: error: the following arguments are required: reference, summary

error可以忽略嘛

StevenTang1998 commented 1 year ago

对

Foehnc commented 1 year ago

对

感谢回复，那环境就先这样了，跑跑demo看看会不会出bug

Foehnc commented 1 year ago

在运行官方训练代码python run_textbox.py --model=BART --dataset=samsum --model_path=facebook/bart-base还是会有error，这个问题有办法解决吗

Can't locate XML/Parser.pm in @INC (@INC contains: /home/cqy/.files2rouge /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /home/cqy/.files2rouge/XML/DOM.pm line 41.
BEGIN failed--compilation aborted at /home/cqy/.files2rouge/XML/DOM.pm line 70.
Compilation failed in require at /home/cqy/.files2rouge/ROUGE-1.5.5.pl line 177.
BEGIN failed--compilation aborted at /home/cqy/.files2rouge/ROUGE-1.5.5.pl line 177.
12 Jan 18:00    ERROR Traceback (most recent call last):
  File "/home/cqy/workspace/InterestGraph/video_understanding/TextBox/textbox/utils/dashboard.py", line 312, in new_experiment
    yield True
  File "/home/cqy/workspace/InterestGraph/video_understanding/TextBox/textbox/quick_start/experiment.py", line 136, in run
    self._do_train_and_valid()
  File "/home/cqy/workspace/InterestGraph/video_understanding/TextBox/textbox/quick_start/experiment.py", line 111, in _do_train_and_valid
    self.valid_result = self.trainer.fit(train_data, valid_data)
  File "/home/cqy/workspace/InterestGraph/video_understanding/TextBox/textbox/trainer/trainer.py", line 455, in fit
    self.stopped |= self._valid(valid_data, 'epoch')
  File "/home/cqy/anaconda3/envs/TextBox/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/cqy/workspace/InterestGraph/video_understanding/TextBox/textbox/trainer/trainer.py", line 294, in _valid
    valid_results = self.evaluate(valid_data, is_valid=True)
  File "/home/cqy/anaconda3/envs/TextBox/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/cqy/workspace/InterestGraph/video_understanding/TextBox/textbox/trainer/trainer.py", line 565, in evaluate
    result = self.evaluator.evaluate(generate_corpus, reference_dataset)
  File "/home/cqy/workspace/InterestGraph/video_understanding/TextBox/textbox/evaluator/base_evaluator.py", line 156, in evaluate
    metric_result = evaluator.evaluate(generate_corpus, reference_corpus, avg=avg)
  File "/home/cqy/workspace/InterestGraph/video_understanding/TextBox/textbox/evaluator/abstract_evaluator.py", line 31, in evaluate
    metric_dict = self._calc_metrics_info(generate_corpus=generate_corpus, reference_corpus=reference_corpus)
  File "/home/cqy/workspace/InterestGraph/video_understanding/TextBox/textbox/evaluator/rouge_evaluator.py", line 57, in _calc_metrics_info
    output = r.convert_and_evaluate(rouge_args=rouge_args_str)
  File "/home/cqy/anaconda3/envs/TextBox/lib/python3.8/site-packages/pyrouge/Rouge155.py", line 368, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/home/cqy/anaconda3/envs/TextBox/lib/python3.8/site-packages/pyrouge/Rouge155.py", line 343, in evaluate
    rouge_output = check_output(command, env=env).decode("UTF-8")
  File "/home/cqy/anaconda3/envs/TextBox/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/cqy/anaconda3/envs/TextBox/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/cqy/.files2rouge/ROUGE-1.5.5.pl', '-e', '/home/cqy/.files2rouge/data', '-c', '95', '-r', '1000', '-n', '2', '-w', '1.2', '-a', '-m', '/tmp/tmpszg0n7fm/rouge_conf.xml']' returned non-zero exit status 2.
StevenTang1998 commented 1 year ago

安装一下perl

sudo apt-get install libxml-parser-perl
Foehnc commented 1 year ago

安装一下perl

sudo apt-get install libxml-parser-perl

我这服务器是centos的，找到了类似方案装xml::parser，但是貌似是源的问题总是timeout，心累...

StevenTang1998 commented 1 year ago

这个似乎没有办法解决，建议自寻可以突破的办法吧

StevenTang1998 commented 1 year ago

或者如果你不需要进行摘要任务，可以忽略错误

Foehnc commented 1 year ago

或者如果你不需要进行摘要任务，可以忽略错误

我这边也需要做类似摘要的生成任务，这个error看来必须得解决了

Foehnc commented 1 year ago

或者如果你不需要进行摘要任务，可以忽略错误

我从前天晚上弄这个环境，弄到现在凌晨，那个rouge的问题还是没解决，centos还要用perl装XML::Parser，这个XML::Parser又有依赖装不上，真的劝退了，装环境从来没装的这么痛苦过

StevenTang1998 commented 1 year ago

抱歉，这个是files2rouge的要求

StevenTang1998 commented 1 year ago

要准确的评价摘要任务需要这个包的支持，建议使用Ubuntu系统，我们确实没有在centos系统上进行过测试

StevenTang1998 commented 1 year ago

请问你需要进行什么任务的评测？中文还是英文？

Foehnc commented 1 year ago

请问你需要进行什么任务的评测？中文还是英文？

中文任务，类似摘要的生成任务

StevenTang1998 commented 1 year ago

中文的话可以不使用files2rouge进行评测，你可以下载https://huggingface.co/datasets/RUCAIBox/Chinese-Generation/blob/main/csl.tgz 这个数据集进行测试

Foehnc commented 1 year ago

中文的话可以不使用files2rouge进行评测，你可以下载https://huggingface.co/datasets/RUCAIBox/Chinese-Generation/blob/main/csl.tgz 这个数据集进行测试

好的，我明天试下

Foehnc commented 1 year ago

我跑csl数据集没有再报错了，但大致看了一下源码之后有两个问题希望解答一下： （1）我在设置了2张卡之后，训练时只用了设置的第一张卡，在init_device里面看了下设置多卡的方式我应该没写错并且print看了下CUDA_VISIBLE_DEVICE都是两张卡

python run_textbox.py \
    --gpu_id=\(2,3\) \
    --model=Chinese-BART \
    --dataset=csl \
    --model_path=fnlp/bart-base-chinese \
    --saved_dir=./saved/DEBUG/ \
    --epochs=3 \
    --train_batch_size=32 \
    --eval_batch_size=64 \
    --max_save=1 \
    --wandb=offline \

（2）好像只保存inner model相关的参数config.json，通过配置文件、cmd传入的参数不会保存是吗

StevenTang1998 commented 1 year ago

（1）你是要多卡DDP训练吗？ （2）config参数在project.log中

Foehnc commented 1 year ago

（1）是的，单机多卡；（2）个人感觉这些外部参数可以保存到一个文件里，看起来也方便

StevenTang1998 commented 1 year ago

（1）单机多卡请参考：https://github.com/RUCAIBox/TextBox/blob/2.0.0/asset/efficient_training.md#distributed-data-parallel （2）之后可能会考虑你的建议

Foehnc commented 1 year ago

（1）单机多卡请参考：https://github.com/RUCAIBox/TextBox/blob/2.0.0/asset/efficient_training.md#distributed-data-parallel （2）之后可能会考虑你的建议

好的，谢谢

Foehnc commented 1 year ago

请问一下，我用chinese-bart，target里面添加了两个新的token，在配置文件里面也添加了'additional_special_tokens': ['[EMP]', '[DEL]']，但是预测的结果没有这两个token，这个是为什么啊

StevenTang1998 commented 1 year ago

https://github.com/RUCAIBox/TextBox/blob/2e061f2215813c7c8dc5d01bda3aa0d701a79a3b/textbox/model/abstract_model.py#L103 按照hugging face，我们在生成的时候会跳过特殊字符（例如[SOS]，[PAD]），如果你需要添加普通的字符，请通过tokenizer_add_tokens添加 https://github.com/RUCAIBox/TextBox/blob/2e061f2215813c7c8dc5d01bda3aa0d701a79a3b/textbox/properties/overall.yaml#L30

此外，如果安装没有问题，请关闭这个issue，与安装无关的问题麻烦新开issue

Foehnc commented 1 year ago

好的，谢谢