inference_gui2.py
.
Inference GUI 2
header.On Windows, try the script under releases.
Otherwise: pip install -r requirements.txt
in a Python 3.8/3.9 (conda) environment. Additional features may be available based on other dependencies:
pip install pyrubberband
. Note that installing pyrubberband installs PySoundFile which needs to be uninstalled, and SoundFile will need to be reinstalled.pip install requests
and also install this ControllableTalkNet fork.pip install tensorflow crepe
pip install pygame
pip install keyboard
. Requires sudo on Linux. Please note that keyboard
uses OS keyboard hooks. This allows it to read your keystrokes without window
focus. This also could result in unexpected keyboard behavior in the event of
a crash.pip install pedalboard
. Please note that
pedalboard's support for VST3 is fairly temperamental and crashes may result.
Additionally, plugins are not guaranteed to function correctly.Models should be placed in separate folders within a folder called models
, in the same directory as inference_gui2.py
by default. Specifically, the file structure should be:
so-vits-svc-eff\
models\
TwilightSparkle
G_*****.pth
D_*****.pth
kmeans_*****.pt {may or may not be present for some models}
config.json
If the proper libraries are installed, the GUI can be run simply by running inference_gui2.py
. If everything goes well you should see something like this (some features may not be available depending on what extra libraries you have installed):
All basic workflow occurs under the leftmost UI panel.
Speaker:
.Files to Convert
. Alternatively, click on Files to Convert
or Recent Directories
to open a file dialog.Transpose
.Convert
. The resulting file should appear under results
.The right UI panel allows for recording audio directly into the GUI for quick fixes and tests. Simply select the proper audio device and click Record
to begin recording. Recordings will automatically be saved to a recordings
folder. The resulting recording can be transferred to the so-vits-svc panel by pressing Push last output to so-vits-svc
.
TypeError: Invalid file: WindowsPath('...')
Ensure that PySoundFile is not installed (pip show pysoundfile
). This may occur after installing rubberband. PySoundFile is a deprecated version of SoundFile. After uninstalling pysoundfile, run pip install soundfile==0.10.3.post1 --force-reinstall
Couldn't parse TalkNet response.
Ensure that you are running alt_server.py
in the TalkNet fork, and not controllable_talknet.py
(or related).Use clustering
.Select custom cluster model...
Clustering ratio
.to_pitch_ac
instead of
to_pitch_cc
. MAY be preferred for speaking inputs with rapidly varying pitch.custom_merge
.For TalkNet support, you need to pip install requests
and also install this ControllableTalkNet fork. Instead of running talknet_offline.py
, run alt_server.py
(if you use a batch script or conda environment to run TalkNet, you should use it to run alt_server.py
). This will start a server that can interface with Inference GUI 2. The TalkNet server should be started before Inference GUI 2.
Next, starting Inference GUI 2 should show a UI like this:
The rightmost panel shows controls for TalkNet which are similar to those used in the web interface. Some items special to this interface:
Provide input audio
through clicking for a file dialog or drag-and-drop.Disable reference audio
Manual push TalkNet output to so-vits-svc section
; alternatively, automatically check Auto push TalkNet output to so-vits-svc
.Auto push TalkNet output to so-vits-svc
ignores so-vits-svc parameters on the left side panel, since the intended use of this panel is to improve the quality of so-vits-svc generations; however, this can be changed with Apply left-side parameters to so-vits-svc gens
A singing voice coversion (SVC) model, using the SoftVC encoder to extract features from the input audio, sent into VITS along with the F0 to replace the original input to acheive a voice conversion effect. Additionally, changing the vocoder to NSF HiFiGAN to fix the issue with unwanted staccato.
eff
branch of this
repository, with a few extra features relating to 4.0 models
(such as automatic pitch prediction and clustering). For instructions on using the GUI see the eff
branchhubert
.logs/44k
.wget -P logs/44k/ https://huggingface.co/therealvul/so-vits-svc-4.0-init/resolve/main/G_0.pth
wget -P logs/44k/ https://huggingface.co/therealvul/so-vits-svc-4.0-init/resolve/main/D_0.pth
Note that the following notebooks are not maintained by me.
All that is required is that the data be put under the dataset_raw
folder in the structure format provided below.
dataset_raw
├───speaker0
│ ├───xxx1-xxx1.wav
│ ├───...
│ └───Lxx-0xx8.wav
└───speaker1
├───xx2-0xxx2.wav
├───...
└───xxx7-xxx007.wav
python resample.py
python preprocess_flist_config.py
python preprocess_hubert_f0.py
After running the step above, the dataset
folder will contain all the pre-processed data, you can delete the dataset_raw
folder after that.
python train.py -c configs/config.json -m 44k
Note: The old model will be automatically cleared during training, and only the latest 5 models will be kept. If you want to prevent overfitting, you need to manually back up the model record points, or modify the configuration file keep_ckpts 0 to never clear.
To train a cluster model, train a so-vits-svc 4.0 model first (as above), then execute python cluster/train_cluster.py
.
For instructions on using the GUI see the eff
branch
Otherwise use inference_main.py
Command line support has been added for inference
# Example
python inference_main.py -m "logs/44k/G_30400.pth" -c "configs/config.json" -n "君の知らない物語-src.wav" -t 0 -s "nen"
Required fields
raw
folderOptional fields
The 4.0 model training process will train an f0 predictor. For voice conversion you can enable automatic pitch prediction. Do not enable this function when converting singing voices unless you want it to be out of tune.
Clustering is used to make the model trained more like the target timbre at the cost of articulation/intelligibility. The model can linearly control the proportion of non-clustering scheme (more intelligible, 0) vs. clustering scheme (more speaker-like, 1).
Use onnx_export.py
checkpoints
and open itcheckpoints
folder and name it after your project such as aziplayer
model.pth
,rename the config file to config.json
,and place it in the project folder (aziplayer
)path = "NyaruTaffy"
to your project name e.g. path = "aziplayer"
model.onnx
file will be generated in your project folder, which is the exported model