Open syedmustafa54 opened 1 year ago
Is port 8000 the right port? Usually gradio defaults to port 7860 - could you check with:
API_URL=http://0.0.0.0:7860/generate/ API_URL_FROM_FEATURES=http://0.0.0.0:7860/generate_from_features/ python app.py
(also generate_from_features
was spelled, wrong, which could have been the culprit)
Either way, I agree that we should have a better error message! Will open a PR for this
Using the main branch now should give you a more detailed Gradio error message @syedmustafa54!
@sanchit-gandhi Thanks for the update but now I am facing a different issue on GPU it's not giving output its stuck I have modified demo.launch(server_name="0.0.0.0",server_port=9504, show_api=False)
and just ran this command python direct_app.py
To create a public link, set `share=True` in `launch()`.
2023-04-24 14:51:03;INFO;Loading audio file...
2023-04-24 14:51:04;INFO;Pre-processing audio file...
its stuck for long time there
lly gradio defaults to port 7860
port 8000 is for sh launch_app.sh
and for app.py gradio its 9504 I have modified it
Cool! The model is probably JIT compiling (which takes a few minutes the first time you run it, but will be fast thereafter). See https://github.com/sanchit-gandhi/whisper-jax#pipeline-usage for details.
@sanchit-gandhi any guess on T4 I have waited for 25-30 mins but still no results is this normal?
Which checkpoint are you using? And how long is the audio file? If it's stuck on the pre-processing stage then it might be that you're maxed out on CPU resource for pre-processing the audio file.
I just ran direct_app.py. the audio file is 3 sec. I have checked the CPU usage its only 1 process is using 100% rest is at 0%
Hey @syedmustafa54 - could you first check that the Flax Whisper Pipeline works standalone with this audio file, i.e. not as a gradio app? This will give you better error logs for any environment / resource errors. You can follow the instructions from https://github.com/sanchit-gandhi/whisper-jax#pipeline-usage
Hey I tried to run the below code by adding audio.mp3 in the folder but still its not giving any error just not showing any output
from whisper_jax import FlaxWhisperPipline
# instantiate pipeline
pipeline = FlaxWhisperPipline("openai/whisper-large-v2")
# JIT compile the forward call - slow, but we only do once
text = pipeline("audio.mp3")
# used cached function thereafter - super fast!!
text = pipeline("audio.mp3")```
Hello @sanchit-gandhi I have started a new Instance install cuda-11.8 cudnn-8.8 and install all the requirements and ran the below code
from whisper_jax import FlaxWhisperPipline
# instantiate pipeline
pipeline = FlaxWhisperPipline("openai/whisper-large-v2")
# JIT compile the forward call - slow, but we only do once
text = pipeline("audio.mp3")
# used cached function thereafter - super fast!!
text = pipeline("audio.mp3")
print(text)
Now this code is working fine its taking 1.5 mins to give the output on 3 sec audio on T4 GPU.
After confirming the code is working I ran python direct_app.py
it has pass 800 sec still no results output was
2023-04-25 10:40:23;INFO;Loading audio file...
2023-04-25 10:40:23;INFO;Pre-processing audio file...
according to me when I debug the issue is in the below for loop logic it has printed Transcribing... {'stride': [(25728, 0, 0)], 'input_features': array([[[-0.58101606, -0.58101606, -0.38316262, ..., -0.58101606,
and stuck at there after that for loop I have also added a print but that is not getting a trigger.
This is the for loop
for batch, _ in zip(dataloader, progress.tqdm(dummy_batches, desc="Transcribing...")):
print("Transcribing...", batch.shape)
model_outputs.append(
pipeline.forward(batch, batch_size=BATCH_SIZE, task=task, return_timestamps=return_timestamps)
)
Hey @syedmustafa54 - could you maybe try and pinpoint the exact line the app is failing on? Perhaps you can quit the app when it hangs (CTRL + C) and paste the traceback here? For reference, I'm using direct_app.py
exactly as is to host the Whisper JAX demo on the HF Hub!
There are some more verbose instructions for creating a demo here: https://github.com/sanchit-gandhi/whisper-jax#creating-an-endpoint
@sanchit-gandhi for your case you are just hosting the front end on hugging face and back end at Google TPU I think that's why you are not getting any issues. this are the logs after CTRL+C
Initialized persistent compilation cache at ./jax_cache
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:319: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your components from gradio.components
warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/deprecation.py:40: UserWarning: `optional` parameter is deprecated, and it has no effect
warnings.warn(value)
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:183: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your component from gradio.components
warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:121: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your component from gradio.components
warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/outputs.py:22: UserWarning: Usage of gradio.outputs is deprecated, and will not be supported in the future, please import your components from gradio.components
warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:27: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your component from gradio.components
warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/deprecation.py:40: UserWarning: `numeric` parameter is deprecated, and it has no effect
warnings.warn(value)
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py:863: UserWarning: api_name predict already exists, using predict_1
warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py:863: UserWarning: api_name predict already exists, using predict_2
warnings.warn(
Running on local URL: http://0.0.0.0:7860
To create a public link, set `share=True` in `launch()`.
2023-04-26 12:00:30;INFO;Loading audio file...
2023-04-26 12:00:31;INFO;Pre-processing audio file...
^CKeyboard interruption in main thread... closing server.
Process ForkPoolWorker-4:
Process ForkPoolWorker-6:
Process ForkPoolWorker-8:
Process ForkPoolWorker-7:
Process ForkPoolWorker-5:
Process ForkPoolWorker-3:
Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
task = get()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
with self._rlock:
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
Traceback (most recent call last):
Traceback (most recent call last):
KeyboardInterrupt
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
task = get()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
with self._rlock:
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
task = get()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 366, in get
res = self._reader.recv_bytes()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
buf = self._recv(4)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
task = get()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
with self._rlock:
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
task = get()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
with self._rlock:
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
task = get()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
with self._rlock:
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
task = get()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
task = get()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
with self._rlock:
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
with self._rlock:
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
^CTraceback (most recent call last):
File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1984, in block_thread
time.sleep(0.1)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/whisper-jax/app/direct_app.py", line 236, in <module>
demo.launch(server_name="0.0.0.0", show_api=False)
File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1901, in launch
self.block_thread()
File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1987, in block_thread
self.server.close()
File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/networking.py", line 43, in close
self.thread.join()
File "/opt/conda/envs/whisper/lib/python3.9/threading.py", line 1029, in join
self._wait_for_tstate_lock()
File "/opt/conda/envs/whisper/lib/python3.9/threading.py", line 1045, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
^CProcess ForkPoolWorker-9:
Traceback (most recent call last):
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
task = get()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 366, in get
res = self._reader.recv_bytes()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
buf = self._recv(4)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 692, in _terminate_pool
cls._help_stuff_finish(inqueue, task_handler, len(pool))
File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 672, in _help_stuff_finish
inqueue._rlock.acquire()
KeyboardInterrupt
^C
Hey @syedmustafa54 - the application is agnostic to whether we host on HF / locally. How many preprocessing workers are you using? Is this more than the number of CPUs you have available? If so, then multiprocessing will hang indefinitely.
Could you check the number of CPUs you have through:
lscpu
And make sure that this number is less than the product of NUM_PROC
:
https://github.com/sanchit-gandhi/whisper-jax/blob/7159222d7ec994153a527f0220cd58ea8f4c7546/app/app.py#L22
And concurrency_count
:
https://github.com/sanchit-gandhi/whisper-jax/blob/7159222d7ec994153a527f0220cd58ea8f4c7546/app/app.py#L244
(so lscpu
should be less than NUM_PROC * concurrency_count
)
Hello @sanchit-gandhi Thanks for sharing this repo. I installed all the dependencies and ran this command in terminal 1
bash launch_app.sh
in terminal 2 I ranAPI_URL=http://0.0.0.0:8000/generate/ API_URL_FROM_FEATURES=http://0.0.0.0:8000/gnerate_from_features/ python app.py
when I select you tube url getting this errorComplete error