Shared-Reality-Lab / IMAGE-server

IMAGE project server components
Other
2 stars 7 forks source link

espnet-tts fails to start #353

Closed jeffbl closed 2 years ago

jeffbl commented 2 years ago

Updated unicorn, and espnet-tts is failing to start (keeps restarting the container, and does not work). It appears that the images were improperly built due to some dependencies on things in google drive, e.g., : https://drive.google.com/uc?id=1Kb7q5zBeQ30Wsnma0X23G08zvgDG5oen

atp-espnet-tts                 | 2022-02-16 19:34:12,061 Device: cuda
atp-orchestrator               | 2022-02-16T19:34:12.234472276Z Received request
atp-unicorn-cim-mcgill-ca      | 2022-02-16T19:34:12.235185920Z 172.18.0.2 - - [16/Feb/2022:19:34:12 +0000] "POST /image/render HTTP/1.1" 400 642 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
atp-espnet-tts                 | 2022-02-16 19:34:12,953 Fallback to conformer_pos_enc_layer_type = 'legacy_rel_pos' due to the compatibility. If you want to use the new one, please use conformer_pos_enc_layer_type = 'latest'.
atp-espnet-tts                 | 2022-02-16 19:34:12,953 Fallback to conformer_self_attn_layer_type = 'legacy_rel_selfattn' due to the compatibility. If you want to use the new one, please use conformer_pos_enc_layer_type = 'latest'.
atp-espnet-tts                 | Access denied with the following error:
atp-espnet-tts                 | 
atp-espnet-tts                 |        Cannot retrieve the public link of the file. You may need to change
atp-espnet-tts                 |        the permission to 'Anyone with the link', or have had many accesses. 
atp-espnet-tts                 | 
atp-espnet-tts                 | You may still be able to access the file from the browser:
atp-espnet-tts                 | 
atp-espnet-tts                 |         https://drive.google.com/uc?id=1Kb7q5zBeQ30Wsnma0X23G08zvgDG5oen 
atp-espnet-tts                 | 
atp-espnet-tts                 | [2022-02-16 19:34:16 +0000] [9] [ERROR] Exception in worker process
atp-espnet-tts                 | Traceback (most recent call last):
atp-espnet-tts                 |   File "/home/python/.local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
atp-espnet-tts                 |     worker.init_process()
atp-espnet-tts                 |   File "/home/python/.local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 134, in init_process
atp-espnet-tts                 |     self.load_wsgi()
atp-espnet-tts                 |   File "/home/python/.local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
atp-espnet-tts                 |     self.wsgi = self.app.wsgi()
atp-espnet-tts                 |   File "/home/python/.local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
atp-espnet-tts                 |     self.callable = self.load()
atp-espnet-tts                 |   File "/home/python/.local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
atp-espnet-tts                 |     return self.load_wsgiapp()
atp-espnet-tts                 |   File "/home/python/.local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
atp-espnet-tts                 |     return util.import_app(self.app_uri)
atp-espnet-tts                 |   File "/home/python/.local/lib/python3.7/site-packages/gunicorn/util.py", line 359, in import_app
atp-espnet-tts                 |     mod = importlib.import_module(module)
atp-espnet-tts                 |   File "/opt/conda/lib/python3.7/importlib/__init__.py", line 127, in import_module
atp-espnet-tts                 |     return _bootstrap._gcd_import(name[level:], package, level)
atp-espnet-tts                 |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
atp-espnet-tts                 |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
atp-espnet-tts                 |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
atp-espnet-tts                 |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
atp-espnet-tts                 |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
atp-espnet-tts                 |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
atp-espnet-tts                 |   File "/run/tts/app.py", line 23, in <module>
atp-espnet-tts                 |     from espnet_util import tts, fs
atp-espnet-tts                 |   File "/run/tts/espnet_util.py", line 52, in <module>
atp-espnet-tts                 |     vocoder = load_model(download_pretrained_model(vocoder_tag)).to(device).eval()
atp-espnet-tts                 |   File "/home/python/.local/lib/python3.7/site-packages/parallel_wavegan/utils/utils.py", line 341, in download_pretrained_model
atp-espnet-tts                 |     with tarfile.open(output_path, 'r:*') as tar:
atp-espnet-tts                 |   File "/opt/conda/lib/python3.7/tarfile.py", line 1575, in open
atp-espnet-tts                 |     return func(name, "r", fileobj, **kwargs)
atp-espnet-tts                 |   File "/opt/conda/lib/python3.7/tarfile.py", line 1639, in gzopen
atp-espnet-tts                 |     fileobj = GzipFile(name, mode + "b", compresslevel, fileobj)
atp-espnet-tts                 |   File "/opt/conda/lib/python3.7/gzip.py", line 168, in __init__
atp-espnet-tts                 |     fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
atp-espnet-tts                 | FileNotFoundError: [Errno 2] No such file or directory: '/home/python/.cache/parallel_wavegan/ljspeech_full_band_melgan.v2.tar.gz'
atp-espnet-tts                 | [2022-02-16 19:34:16 +0000] [9] [INFO] Worker exiting (pid: 9)
atp-espnet-tts                 | [2022-02-16 19:34:16 +0000] [1] [INFO] Shutting down: Master
atp-espnet-tts                 | [2022-02-16 19:34:16 +0000] [1] [INFO] Reason: Worker failed to boot.
jeffbl commented 2 years ago

Triaging as production issue given we need to deploy current server-side build as soon as updated browser extension goes live.