Open unilight opened 7 months ago
Will reply soon
On Wed, Jan 24, 2024, 15:07 Wen-Chin Huang (unilight) < @.***> wrote:
From the README ( https://github.com/DigitalPhonetics/VoicePAT?tab=readme-ov-file#anonymization) it says that to anonymize my own data I should modify the following fields in the config file:
data_dir: # path to original data in Kaldi-format for anonymization results_dir: # path to location for all (intermediate) results of the anonymization models_dir: # path to models location
Just wondering what exactly is the Kaldi-format. I guess it refers to a text file with each line of the format
, but just want to double-check. The README also says: Pretrained models for this anonymization can be found at https://github. com/DigitalPhonetics/speaker-anonymization/releases/tag/v2.0 https://github.com/DigitalPhonetics/speaker-anonymization/releases/tag/v2.0 and earlier releases.
But the link contains several zip files to download and it is very unclear what should be done here.
Would appreciate it if some more details could be provided. I totally understand this toolkit is under construction -- just raising my questions here.
— Reply to this email directly, view it on GitHub https://github.com/DigitalPhonetics/VoicePAT/issues/10, or unsubscribe https://github.com/notifications/unsubscribe-auth/AECMKFLLT2IPOQUPK45LJ4DYQEIQPAVCNFSM6AAAAABCI2HQ7WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA4TQMZUGYZDGNA . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Regarding the first part of your question (Kaldi format),
Kaldi is a Swiss army knife for various speech processing tasks. Kaldi format is the dataset organization used by this toolkit, for a complete reference viewing https://kaldi-asr.org/doc/data_prep.html helps. Alternatively you can request access to the VoicePrivacy Challenge datasets, the instructions for this are available at https://github.com/Voice-Privacy-Challenge/Voice-Privacy-Challenge-2022.
This framework primarily utilizes the files
wav.scp
: a text file, at each line containing space-separated UtteranceID and file path (or a command that dumps the binary file to stdout)
enrolls
: a text file, at each line containing UtteranceIDs that will be used as enrollment utterances for ASV tasks
trials
: a text file, at each line containing UtteranceIDs that will be used as trial utterances for ASV tasks
spk2gender
: a text file, at each line containng space-separated SpeakerID and speaker gender (m: male, f: female)
utt2spk
: a text file, at each line containing space-separated UtteranceID and SpeakerID
and few others that are similar to the last two I explained.
The function read_kaldi_format
in utils.dataio
would give you some hints about how these files are parsed.
Regarding the second part of your question (installation), currently Bash script 01_download_data_model.sh
is provided. I am working on consolidating these into a Makefile, and plan to submit a PR, to be approved by the project maintainers. This set of changes will also include a better documentation of these :)
Hi @unilight, thanks for the question. Yeah, the toolkit is still under construction and will update soon.
If you would like to use your own anonymized data, simply use the VoicePAT pretrained ASV and ASR models for evaluation:
1) please prepare your own anonymized data folder anon_data_dir
, which includes
2) run 00_install.sh
and 01_download_data_model.sh
3) modify the anon_data_dir
filed in config/eval_pre_from_anon_datadir.yaml
4) run python run_evaluation.py --config eval_pre_from_anon_datadir.yaml
Thank you for the replies! So for now it would be better to use the vpc
branch, right? (I guess all your instructions assume that I am on that branch)
please prepare your own anonymized data folder anon_data_dir, which includes...
Do you mean that I should prepare one or several folders each containing the wav files I want to anonymize?
So for now it would be better to use the
vpc
branch, right? (I guess all your instructions assume that I am on that branch)
Yes.
Do you mean that I should prepare one or several folders each containing the wav files I want to anonymize?
Yes, but currently, the toolkit only supports 12 dev+test datasets provided by VoicePrivacy challenge, These datasets include wav.scp/trial/utt2spk/spk2utt files, and they are also indicated in the config file, https://github.com/DigitalPhonetics/VoicePAT/blob/vpc/configs/eval_pre_from_anon_datadir.yaml#L7-L27
The first step of the evaluation script is to prepare wav.scp/trial/utt2spk/spk2utt for anonymized data and evaluation subdatasets. You can find the implementation details here https://github.com/DigitalPhonetics/VoicePAT/blob/vpc/run_evaluation.py#L154-L175
We strictly follow the VPC data design, I understand this is complicated at the beginning. Sorry for any confusion.
If you would like to use your own anonymized data instead of the VPC datasets, you will need to prepare your own data(including wav.scp/trial/utt2spk/spk2utt) and modify(or skip) https://github.com/DigitalPhonetics/VoicePAT/blob/vpc/run_evaluation.py#L154-L175. We plan to support this function at a later time.
@xiaoxiaomiao323 Thank you for the kind replies (and sorry for getting back so late).
So I tried running the VPC2022 baseline repo and I think I have a better understanding of how the directories should look right now.
I have another question. Say I have developed an anonymization system by myself and I want to use this toolkit to do evaluation on the VPC2022 dataset. After anonymizing the official 12 dev/test sets, is there anything else I should do other than just putting them in separate folders? Do I need to prepare the wav.scp/uttspk/...etc? Also can you suggest what modifications I should make to the config files?
I have another question. Say I have developed an anonymization system by myself and I want to use this toolkit to do evaluation on the VPC2022 dataset. After anonymizing the official 12 dev/test sets, is there anything else I should do other than just putting them in separate folders? Do I need to prepare the wav.scp/uttspk/...etc? Also can you suggest what modifications I should make to the config files?
Hi, glad to know! We updated the readme. No need to prepare the wav.scp/utt2spk/..etc, the run_evaluation.py will create them automatically.
prepare anonymized folders each containing the anonymized wav files (which you have done)
libri_dev_enrolls/*wav
libri_dev_trials_m/*wav
libri_dev_trials_f/*wav
libri_test_enrolls/*wav
libri_test_trials_m/*wav
libri_test_trials_f/*wav
~train-clean-360/*wav~ ignore this if you don't train $ASR{eval}^{anon}$, $ASV{eval}^{anon}$
modify entries in configs/eval_pre_from_anon_datadir.yaml and ~configs/eval_post_scratch_from_anon_datadir.yaml~ ( ignore this if you don't train $ASR{eval}^{anon}$, $ASV{eval}^{anon}$):
anon_data_dir: !PLACEHOLDER # TODO path to anonymized data (raw audios), e.g. <anon_data_dir>/libri_test_enrolls/*wav etc.
anon_data_suffix: !PLACEHOLDER # suffix for dataset to signal that it is anonymized, e.g. b2, b1b, or gan
Noted: the VPC2024 plan (the challenge plan will release soon) to remove VCTK dev/test datasets, so the entry datasets
in configs/eval_pre_from_anon_datadir.yaml is
datasets:
- name: libri_dev
data: libri
set: dev
enrolls: [enrolls]
trials: [trials_f, trials_m]
- name: libri_test
data: libri
set: test
enrolls: [enrolls]
trials: [trials_f, trials_m]
If you still want to include VCTK, please modify the entry datasets
in
configs/eval_pre_from_anon_datadir.yaml to add vctk datasets.
datasets:
- name: libri_dev
data: libri
set: dev
enrolls: [enrolls]
trials: [trials_f, trials_m]
- name: libri_test
data: libri
set: test
enrolls: [enrolls]
trials: [trials_f, trials_m]
- name: vctk_dev
data: vctk
set: dev
enrolls: [enrolls]
trials: [trials_f_all, trials_m_all]
- name: vctk_test
data: vctk
set: test
enrolls: [enrolls]
trials: [trials_f_all, trials_m_all]
python run_evaluation.py --config eval_pre_from_anon_datadir.yaml
~python run_evaluation.py --config eval_post_scratch_from_anondatadir.yaml~ ignore this if you don't train $ASR{eval}^{anon}$, $ASV_{eval}^{anon}$
@xiaoxiaomiao323 Thank you for the reply!
I mainly want to align with the evaluation protocol in VPC2022, which, in my understanding, requires to train $ASR{eval}^{anon}$ and $ASV{eval}^{anon}$. Just wondering if this understanding is correct, and whether the training takes a long time...?
Just wondering if this understanding is correct, and whether the training takes a long time...?
No problem! After many testing runs, we found that this really depends on the hard drives. For 1 V100: If you have an SSD or a high-performance drive, $ASV{eval}^{anon}$ takes <3h, while $ASR{eval}^{anon}$ takes ~35h. But if the drive is old and slow, in a worse case, $ASV{eval}^{anon}$ takes ~10h, $ASR{eval}^{anon}$ takes ~150h Increase num_workers in ASV training config and ASR training config may help to speed up the processing.
@xiaoxiaomiao323 Thank you for the reply!
(I know you probably do not have the official answer but) just wondering whether training $ASR_{eval}^{anon}$ is really necessary. In my understanding, VPC adopted this process because it is claimed that the WER gets greatly improved, but if the anonymized speech cannot yield a good WER on an ASR model trained with natural speech, doesn't it mean that the anonymized speech is not natural enough? Or are there some other reasons?
(I know you probably do not have the official answer but) just wondering whether training $ASR_{eval}^{anon}$ is really necessary. In my understanding, VPC adopted this process because it is claimed that the WER gets greatly improved, but if the anonymized speech cannot yield a good WER on an ASR model trained with natural speech, doesn't it mean that the anonymized speech is not natural enough? Or are there some other reasons?
I agree with your opinion. Actually we already decided to not use $ASR_{eval}^{anon}$ for VPC2024 :)
@xiaoxiaomiao323 I see, thank you for the answers! :)
So to understand the evaluation process I synced the latest vpc
branch and ran 02_run.sh
. Then I encountered the following error:
...
100%|█████████▉| 761/762 [02:19<00:00, 27.26it/s]
100%|██████████| 762/762 [02:19<00:00, 5.47it/s]
Done
Processing libri_train_360...
Traceback (most recent call last):
File "run_anonymization_dsp.py", line 19, in <module>
pipeline.run_anonymization_pipeline(datasets)
File "/data/group1/z44476r/Experiments/VoicePAT/anonymization/pipelines/dsp_pipeline.py", line 34, in run_anonymization_pipeline
process_data(dataset_path=self.libri_360_data_dir,
File "/data/group1/z44476r/Experiments/VoicePAT/anonymization/modules/dsp/anonymise_dir_mcadams_rand_seed.py", line 69, in process_data
shutil.copytree(dataset_path, output_path)
File "/home/z44476r/data/Experiments/VoicePAT/venv/lib/python3.8/shutil.py", line 555, in copytree
with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: 'data/train-clean-360'
2024-02-20 16:39:05,022 - __main__- INFO - Preparing datadir according to the Kaldi format.
2024-02-20 16:39:06,516 - root- INFO - Perform ASV evaluation
...
which leads to error in python run_evaluation.py --config eval_post_scratch_from_anon_datadir.yaml
.
However I do find the anonymized waveform files in data/train-clean-360-asv_dsp/wav/*.wav
.
Is there some variables that need to be changed in the configs?
@unilight, sorry I think this is just a name mistake, when did you download "data.zip"? this should no problem if you download the new version of "data.zip" https://github.com/DigitalPhonetics/VoicePAT/blob/vpc/01_download_data_model.sh#L65 we used to use "train-clean-360-asv", but now we changed to "train-clean-360" without "-asv", for now please rename
"data/train-clean-360-asv_dsp" -> "data/train-clean-360_dsp"
"data/train-clean-360-asv" -> "data/train-clean-360"
And rerun.
@xiaoxiaomiao323 Thank you for the reply! Yes, I indeed downloaded the files earlier (in late December). I'll rename and rerun, and let you know if I have more problems. (Sorry for running the scripts while you are still working in progress...)
@unilight , no problem, thanks for your patience. Feel free to post any questions.
@xiaoxiaomiao323 I have finished running 02_run.sh
. (FYI training the ASV model for 10 epochs on my server's single Tesla V100 32GM RAM took 18 hours. Maybe we have a not-so-efficient file system here...)
I am trying to interpret the numbers and compare them with those I find on the VoicePAT paper but I am having a hard time. I have the following results:
dataset split gender enrollment trial EER
0 libri dev f original original 16.760
1 libri dev f original anon 13.744
2 libri dev f anon anon 13.070
3 libri dev m original original 1.710
4 libri dev m original anon 2.010
5 libri dev m anon anon 1.087
6 libri test f original original 8.074
7 libri test f original anon 8.210
8 libri test f anon anon 8.576
9 libri test m original original 1.800
10 libri test m original anon 2.197
11 libri test m anon anon 0.916
root - --- EER computation time: 10.313560 min ---
Can you kindly tell me which numbers to look?
Also I am very interested in GVD, and am wondering which variables in the config should I modify to get those GVD numbers?
(FYI training the ASV model for 10 epochs on my server's single Tesla V100 32GM RAM took 18 hours. Maybe we have a not-so-efficient file system here...)
Yeah other people also pointed it out. Really depends on I/O speed of the machine. Try increase num_workers next time, it helps.
For EER results: 1)$ASV_{eval}$ provided model trained using original LibriSpeech-360
enrollment trial
original original (unprotected->OO)
original anon (Ignorant->AA)
anon anon (Lazy-Informed->AA-lazy)
2)$ASV_{eval}^{anon}$ you trained using anonymized LibriSpeech-360
enrollment trial
anon anon (Semi-Informed->AA-semi)
The results you listed are obtained from $ASV_{eval}^{anon}$, so just need to look at the results
enrollment=anon trial=anon
And we didn't test DSP system when we wrote the VoicePAT paper.
which variables in the config should I modify to get those GVD numbers?
GVD computed by $ASV_{eval}$, so change the config https://github.com/DigitalPhonetics/VoicePAT/blob/vpc/configs/eval_pre_from_anon_datadir.yaml
utility:
- asr
- gvd
comment asr, if you want to skip,
utility:
#- asr
- gvd
From the README (https://github.com/DigitalPhonetics/VoicePAT?tab=readme-ov-file#anonymization) it says that to anonymize my own data I should modify the following fields in the config file:
Just wondering what exactly is the Kaldi-format. I guess it refers to a text file with each line of the format
<id> <wav path>
, but just want to double-check. The README also says:But the link contains several zip files to download and it is very unclear what should be done here.
Would appreciate it if some more details could be provided. I totally understand this toolkit is under construction -- just raising my questions here.