coqui-ai / TTS

πŸΈπŸ’¬ - a deep learning toolkit for Text-to-Speech, battle-tested in research and production
http://coqui.ai
Mozilla Public License 2.0
35.82k stars 4.38k forks source link

[Bug] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) #3599

Open asadullahnaeem opened 9 months ago

asadullahnaeem commented 9 months ago

Describe the bug

I am using tts_models/multilingual/multi-dataset/your_tts with multiple gunicorn workers but most of the time I get the following Error traceback. The code sometimes run without any issue. Seems like a bug when multiple workers are used because on single worker it works fine.

Fix Tried

I thought that because of some thread file lock, when multiple workers try to access same config.json file, this issue occurs. So added delay in loading the model (code given) but didn't work.

To Reproduce

class CoquiTTS:
    def __init__(self):

        # adding random time delay
        delay_time = random.randint(1, 5)

        time.sleep(random.randint(1, 5))

        gpu_id = 0

        self.device = torch.device(f"cuda:{gpu_id}" if torch.cuda.is_available() else "cpu")

        self.tts_clone = TTS(
            model_name="tts_models/multilingual/multi-dataset/your_tts"
        ).to(self.device)

Expected behavior

No response

Logs

Feb 21 08:32:07 gunicorn[19710]: [2024-02-21 08:32:07 +0000] [19710] [ERROR] Exception in worker process
Feb 21 08:32:07 gunicorn[19710]: Traceback (most recent call last):
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
Feb 21 08:32:07 gunicorn[19710]:     worker.init_process()
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
Feb 21 08:32:07 gunicorn[19710]:     super(UvicornWorker, self).init_process()
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
Feb 21 08:32:07 gunicorn[19710]:     self.load_wsgi()
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Feb 21 08:32:07 gunicorn[19710]:     self.wsgi = self.app.wsgi()
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
Feb 21 08:32:07 gunicorn[19710]:     self.callable = self.load()
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Feb 21 08:32:07 gunicorn[19710]:     return self.load_wsgiapp()
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Feb 21 08:32:07 gunicorn[19710]:     return util.import_app(self.app_uri)
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
Feb 21 08:32:07 gunicorn[19710]:     mod = importlib.import_module(module)
Feb 21 08:32:07 gunicorn[19710]:   File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
Feb 21 08:32:07 gunicorn[19710]:     return _bootstrap._gcd_import(name[level:], package, level)
Feb 21 08:32:07 gunicorn[19710]:   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
Feb 21 08:32:07 gunicorn[19710]:   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
Feb 21 08:32:07 gunicorn[19710]:   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
Feb 21 08:32:07 gunicorn[19710]:   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
Feb 21 08:32:07 gunicorn[19710]:   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
Feb 21 08:32:07 gunicorn[19710]:   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_avatar/app.py", line 5, in <module>
Feb 21 08:32:07 gunicorn[19710]:     from avatar import avatar_router
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_avatar/avatar/__init__.py", line 1, in <module>
Feb 21 08:32:07 gunicorn[19710]:     from avatar.avatar_router import avatar_router
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_avatar/avatar/avatar_router.py", line 9, in <module>
Feb 21 08:32:07 gunicorn[19710]:     from voice.utils import upload_to_s3
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_avatar/voice/__init__.py", line 1, in <module>
Feb 21 08:32:07 gunicorn[19710]:     from voice.voice_router import voice_router
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_avatar/voice/voice_router.py", line 14, in <module>
Feb 21 08:32:07 gunicorn[19710]:     voice_driver = VoiceDriver()
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_avatar/voice/voice_driver.py", line 20, in __init__
Feb 21 08:32:07 gunicorn[19710]:     self.tts_driver = model_dict[config["voice_model"]](config=config)
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_avatar/voice/coqui_tts/driver.py", line 31, in __init__
Feb 21 08:32:07 gunicorn[19710]:     self.tts_clone = TTS(
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/TTS/api.py", line 74, in __init__
Feb 21 08:32:07 gunicorn[19710]:     self.load_tts_model_by_name(model_name, gpu)
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/TTS/api.py", line 171, in load_tts_model_by_name
Feb 21 08:32:07 gunicorn[19710]:     model_path, config_path, vocoder_path, vocoder_config_path, model_dir = self.download_model_by_name(
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/TTS/api.py", line 129, in download_model_by_name
Feb 21 08:32:07 gunicorn[19710]:     model_path, config_path, model_item = self.manager.download_model(model_name)
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/TTS/utils/manage.py", line 421, in download_model
Feb 21 08:32:07 gunicorn[19710]:     self._update_paths(output_path, output_config_path)
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/TTS/utils/manage.py", line 495, in _update_paths
Feb 21 08:32:07 gunicorn[19710]:     self._update_path("model_args.speaker_encoder_model_path", speaker_encoder_model_path, config_path)
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/TTS/utils/manage.py", line 503, in _update_path
Feb 21 08:32:07 gunicorn[19710]:     config = load_config(config_path)
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/TTS/config/__init__.py", line 92, in load_config
Feb 21 08:32:07 gunicorn[19710]:     data = read_json_with_comments(config_path)
Feb 21 08:32:07 gunicorn[19710]:   File "/home/ubuntu/ai_venv/lib/python3.10/site-packages/TTS/config/__init__.py", line 21, in read_json_with_comments
Feb 21 08:32:07 gunicorn[19710]:     return json.loads(input_str)
Feb 21 08:32:07 gunicorn[19710]:   File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
Feb 21 08:32:07 gunicorn[19710]:     return _default_decoder.decode(s)
Feb 21 08:32:07 gunicorn[19710]:   File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
Feb 21 08:32:07 gunicorn[19710]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Feb 21 08:32:07 gunicorn[19710]:   File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
Feb 21 08:32:07 gunicorn[19710]:     raise JSONDecodeError("Expecting value", s, err.value) from None
Feb 21 08:32:07 gunicorn[19710]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Feb 21 08:32:07 gunicorn[19710]: [2024-02-21 08:32:07 +0000] [19710] [INFO] Worker exiting (pid: 19710)
Feb 21 08:32:07 gunicorn[19710]:  > tts_models/multilingual/multi-dataset/your_tts is already downloaded.Feb 21 08:32:08 gunicorn[19709]: [2024-02-21 08:32:08 +0000] [19709] [ERROR] Worker (pid:19710) exited with code 3
Feb 21 08:32:09 gunicorn[19709]: [2024-02-21 08:32:09 +0000] [19709] [ERROR] Worker (pid:19711) was sent SIGTERM!
Feb 21 08:32:09 gunicorn[19709]: [2024-02-21 08:32:09 +0000] [19709] [ERROR] Shutting down: Master
Feb 21 08:32:09 gunicorn[19709]: [2024-02-21 08:32:09 +0000] [19709] [ERROR] Reason: Worker failed to boot.

Environment

- Python 3.10
- TTS Version: 0.22.0
- Ubuntu 22.04.3 LTS
- Nvidia A10G 24 GB GPU.
- Pytorch Version: 2.1.0+cu121
- CUDA Version: 12.3

Additional context

No response

ClericalAid commented 9 months ago

I am having the same issue running it locally on a single machine.

Code

import torch
from TTS.api import TTS

# Get device
device = "cuda" if torch.cuda.is_available() else "cpu"

# List available 🐸TTS models
print(TTS().list_models())

# Init TTS
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to(device)

# Run TTS
# ❗ Since this model is multi-lingual voice cloning model, we must set the target speaker_wav and language
# Text to speech list of amplitude values as output
wav = tts.tts(text="Hello world!", speaker_wav="~/Downloads/uk_female_high_voice.wav", language="en")
# Text to speech to a file
tts.tts_to_file(text="Hello world!", speaker_wav="~/Downloads/uk_female_high_voice.wav", language="en", file_path="output.wav")

Output

<TTS.utils.manage.ModelManager object at 0x7fbd95c5bee0>
 > tts_models/multilingual/multi-dataset/xtts_v2 has been updated, clearing model cache...
 > Downloading model to /home/chrish/.local/share/tts/tts_models--multilingual--multi-dataset--xtts_v2
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 3.05k/3.05k [00:00<00:00, 8.33kiB/s]
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 3.05k/3.05k [00:00<00:00, 8.47kiB/s]
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 3.05k/3.05k [00:00<00:00, 8.19kiB/s]
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 3.05k/3.05k [00:00<00:00, 7.74kiB/s]
 > Model's license - CPML
 > Check https://coqui.ai/cpml.txt for more info.
Traceback (most recent call last):
  File "/home/chrish/workspace/talkie/example.py", line 11, in <module>
    tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to(device)
  File "/home/chrish/miniconda3/envs/talkie/lib/python3.10/site-packages/TTS/api.py", line 74, in __init__
    self.load_tts_model_by_name(model_name, gpu)
  File "/home/chrish/miniconda3/envs/talkie/lib/python3.10/site-packages/TTS/api.py", line 177, in load_tts_model_by_name
    self.synthesizer = Synthesizer(
  File "/home/chrish/miniconda3/envs/talkie/lib/python3.10/site-packages/TTS/utils/synthesizer.py", line 109, in __init__
    self._load_tts_from_dir(model_dir, use_cuda)
  File "/home/chrish/miniconda3/envs/talkie/lib/python3.10/site-packages/TTS/utils/synthesizer.py", line 161, in _load_tts_from_dir
    config = load_config(os.path.join(model_dir, "config.json"))
  File "/home/chrish/miniconda3/envs/talkie/lib/python3.10/site-packages/TTS/config/__init__.py", line 92, in load_config
    data = read_json_with_comments(config_path)
  File "/home/chrish/miniconda3/envs/talkie/lib/python3.10/site-packages/TTS/config/__init__.py", line 21, in read_json_with_comments
    return json.loads(input_str)
  File "/home/chrish/miniconda3/envs/talkie/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/home/chrish/miniconda3/envs/talkie/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/chrish/miniconda3/envs/talkie/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Update

Wait, I just realized that HuggingFace is down, so the model failed to download. In the folder ~/.local/share/tts/tts_models--multilingual--multi-dataset--xtts_v2, the file config.json reads like the HTML file received because HuggingFace returned a 503:

config.json ```html Hugging Face - The AI community building the future.

503

Hugging Face is in maintenance

```

@asadullahnaeem - seeing as we got a similar error, is it possible that when you run multiple workers they encounter an issue because they are being disconnected from the network? Maybe you're being blocked from creating multiple connections. Meanwhile, when you run a single worker, ti downloads the model fine and moves on.

asadullahnaeem commented 9 months ago

Files are already downloaded.

stale[bot] commented 7 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.