search

Voice-Privacy-Challenge / Voice-Privacy-Challenge-2024

Baseline Recipe for VoicePrivacy Challenge 2024: anonymization systems and evaluation software
Other
40 stars 2 forks source link

Calculate ASR error #59

Closed FuShengDuoBuYu closed 5 months ago

FuShengDuoBuYu commented 5 months ago

The reason for the error is: the ind2lab dictionary is empty. Where is the value assigned? It is not found in the code? In addition, the files provided by the ASR model have been updated. What are the changes?

FuShengDuoBuYu commented 5 months ago

base64 The printed error content is as above. The main reason is that the ind2lab dictionary is empty. Where should I assign values to the dictionary?

pchampio commented 5 months ago

Hello. I think your issue comes from a upgrading from v1 to v2 or using v1 instead of v2. Question: Does a fresh install work? (git clone and all the install/run scripts) Did you updated to v2 recently?

Does running 00_install.sh and 01_download_data_model.sh fixes your issue? What is your commit id? (share output of git log) Thanks!

FuShengDuoBuYu commented 5 months ago

Hello， My commit id is 70c5080.

I have not updated the environment. My environment configuration does not directly run 00_install.sh and download_data_module.sh. Since my pytorch version is different from the one in the file, and I did not download micromamba, I established a conda virtual environment in the docker container and configured the environment according to the .sh file.

At present, my environment can run McAdams Baseline normally, and can perform normal evaluation of ASV and SER on anonymized audio, but ASR will report an error. The reason for the error is that ind2lab is empty. I did not find where to assign a value to ind2lab in the code.

Can you tell me where to assign the value to ind2lab in the ASR evaluation part of the run_evaluation.py code? Thanks！

pchampio commented 5 months ago

I can not tell you directly where to assign the value to ind2lab since it is done internally in the SpeechBrain dependency. This is supposed to be handled by speechbrain, not by us. This raises the question about your speechbrain version. What is your speechbrain version?

$ python3 -c "import speechbrain; print(speechbrain.__version__)"
0.5.16
pchampio commented 5 months ago

Having different version of python/pytorch and different installation scripts than the one we provide could be the main cause of this issue. Could you provide the output of pip freeze It should be the following:

absl-py==2.1.0
audioread==3.0.1
certifi==2022.12.7
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1696001724357/work
charset-normalizer==2.1.1
cmake==3.25.0
ConfigArgParse==1.7
cycler==0.12.1
Cython==3.0.10
decorator==5.1.1
ffmpeg==1.4
filelock==3.9.0
fsspec==2024.3.1
future==1.0.0
grpcio==1.62.1
huggingface-hub==0.22.2
HyperPyYAML==1.2.2
idna==3.4
Jinja2==3.1.2
joblib==1.4.0
kaldiio==2.18.0
kiwisolver==1.4.5
librosa==0.9.2
lit==15.0.7
llvmlite==0.42.0
Markdown==3.6
MarkupSafe==2.1.3
matplotlib==3.4.3
mpmath==1.3.0
networkx==3.2.1
noisereduce==3.0.0
numba==0.59.1
numpy==1.26.3
packaging==24.0
pandas==2.2.2
pillow==10.2.0
platformdirs==4.2.0
pooch==1.8.1
protobuf==5.26.1
pyarrow==15.0.2
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1711811537435/work
pyloudnorm==0.1.1
pyparsing==3.1.2
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
regex==2023.12.25
requests==2.28.1
resampy==0.4.3
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8
safetensors==0.4.2
scikit-learn==1.4.2
scipy==1.12.0
sentencepiece==0.2.0
six==1.16.0
sounddevice @ file:///home/conda/feedstock_root/build_artifacts/python-sounddevice_1681927788393/work
soundfile==0.12.1
speechbrain==0.5.16
sympy==1.12
tensorboard==2.16.2
tensorboard-data-server==0.7.2
threadpoolctl==3.4.0
tokenizers==0.15.2
torch==2.0.1+cu117
torchaudio==2.0.2+cu117
torchvision==0.15.2+cu117
tqdm==4.66.2
transformers==4.39.3
triton==2.0.0
typing_extensions==4.8.0
tzdata==2024.1
urllib3==1.26.13
Werkzeug==3.0.2
FuShengDuoBuYu commented 5 months ago

Thanks for your reply,

my speechbrain version is as foloows:

 (myenv) root@gpu02:/opt/Voice-Privacy-Challenge-2024# python3 -c "import speechbrain; print(speechbrain.__version__)"

0.5.16

my pip freeze shows:

absl-py==2.1.0
aiohttp @ file:///croot/aiohttp_1707342283163/work
aiosignal @ file:///tmp/build/80754af9/aiosignal_1637843061372/work
async-timeout @ file:///croot/async-timeout_1703096998144/work
attrs @ file:///croot/attrs_1695717823297/work
audioread==3.0.1
blessed @ file:///croot/blessed_1701887495364/work
Bottleneck @ file:///croot/bottleneck_1707864210935/work
certifi @ file:///croot/certifi_1707229174982/work/certifi
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1696001684923/work
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
click @ file:///croot/click_1698129812380/work
cmake==3.25.0
ConfigArgParse==1.7
cycler==0.12.1
Cython==3.0.10
datasets @ file:///home/conda/feedstock_root/build_artifacts/datasets_1709395865330/work
decorator==5.1.1
dill @ file:///croot/dill_1692271232022/work
ffmpeg==1.4
filelock @ file:///croot/filelock_1700591183607/work
frozenlist @ file:///croot/frozenlist_1698702560391/work
fsspec @ file:///croot/fsspec_1701286474621/work
future==1.0.0
gpustat @ file:///croot/gpustat_1701888826609/work
grpcio==1.62.1
huggingface-hub @ file:///croot/huggingface_hub_1708634519519/work
HyperPyYAML==1.2.2
idna @ file:///croot/idna_1666125576474/work
joblib @ file:///croot/joblib_1685113087166/work
kaldiio==2.18.0
kiwisolver==1.4.5
librosa==0.9.2
llvmlite==0.42.0
Markdown==3.6
MarkupSafe==2.1.5
matplotlib==3.4.3
mkl-fft @ file:///croot/mkl_fft_1695058164594/work
mkl-random @ file:///croot/mkl_random_1695059800811/work
mkl-service==2.4.0
mpmath==1.3.0
multidict @ file:///croot/multidict_1701096859099/work
multiprocess @ file:///croot/multiprocess_1692294385131/work
networkx==3.2.1
nltk @ file:///croot/nltk_1688114146592/work
noisereduce==3.0.0
numba==0.59.1
numexpr @ file:///croot/numexpr_1696515281613/work
numpy @ file:///croot/numpy_and_numpy_base_1708638617955/work/dist/numpy-1.26.4-cp310-cp310-linux_x86_64.whl#sha256=d8cd837ed43e87f77e6efaa08e8de927ca030a1c9c5d04624432d6fb9a74a5ee
nvidia-ml-py @ file:///croot/nvidia-ml-py_1701875136273/work
packaging @ file:///croot/packaging_1710807400464/work
pandas @ file:///croot/pandas_1709590491089/work/dist/pandas-2.2.1-cp310-cp310-linux_x86_64.whl#sha256=af0f237a7f090add6f15eb4b5d0fd2d9d8ded4cfe6979b18563580353c113c9f
pillow @ file:///croot/pillow_1707233021655/work
platformdirs==4.2.0
pooch==1.8.1
protobuf==5.26.1
psutil @ file:///opt/conda/conda-bld/psutil_1656431268089/work
pyarrow @ file:///croot/pyarrow_1707330824290/work/python
pyarrow-hotfix @ file:///home/conda/feedstock_root/build_artifacts/pyarrow-hotfix_1700596371886/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
pyloudnorm==0.1.1
pyparsing==3.1.2
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pytz @ file:///croot/pytz_1695131579487/work
PyYAML @ file:///croot/pyyaml_1698096049011/work
regex @ file:///croot/regex_1696515298636/work
requests @ file:///croot/requests_1707355572290/work
resampy==0.4.3
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8
safetensors @ file:///croot/safetensors_1708633833937/work
scikit-learn==1.4.1.post1
scipy==1.12.0
sentencepiece==0.2.0
six @ file:///tmp/build/80754af9/six_1644875935023/work
sounddevice @ file:///home/conda/feedstock_root/build_artifacts/python-sounddevice_1681927788393/work
soundfile==0.12.1
speechbrain==0.5.16
tensorboard==2.16.2
tensorboard-data-server==0.7.2
threadpoolctl==3.4.0
tokenizers @ file:///croot/tokenizers_1708633814160/work
torch==1.12.1
torchaudio==0.12.1
torchvision==0.13.1
tqdm @ file:///croot/tqdm_1679561862951/work
transformers==4.39.2
typing_extensions @ file:///croot/typing_extensions_1705599297034/work
tzdata @ file:///croot/python-tzdata_1690578112552/work
urllib3 @ file:///croot/urllib3_1700840488771/work
wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work
Werkzeug==3.0.1
xxhash @ file:///croot/python-xxhash_1667919521615/work
yarl @ file:///croot/yarl_1701105127787/work

I checked and all the required packages were available.

I think this may not be an environment problem, because the baseline and the ASV and SER in the evaluation can run normally.

Do you think there are any possible problems?

Thanks！

pchampio commented 5 months ago

Do you think there are any possible problems?

Yes, in your screenshot, the line of code of /opt/conda/envs/myenv/lib/python3.10/site-packages/speechbrain/dataio/encoder.py does not match the one of speechbrain at the v0.5.16 tag https://github.com/speechbrain/speechbrain/blob/v0.5.16/speechbrain/dataio/encoder.py#L570 As such, it is very weird that your version is 0.5.16 too, it does not match the github tag.

Your torch version is 1.12.1 and ours is 2.0.1.

It is really complicated for me to debug your issue since it is a dependency issue IMHO, your dev environment is not based on the one we provide. There is no way for me to reproduce your error at the moment. How hard would it for you to install everything with our script? (To validate that the issue is a dependency one). Can you also check the diff between the github 0.5.16 tag and yours?:

wget https://raw.githubusercontent.com/speechbrain/speechbrain/v0.5.16/speechbrain/dataio/encoder.py 
diff /opt/conda/envs/myenv/lib/python3.10/site-packages/speechbrain/dataio/encoder.py encoder.py
pchampio commented 5 months ago

This issue/question was closed because it has been inactive for 14 days. Feel free to re-open it if you are still facing the described issue with the provided installation script.