search

nateraw / openai-vision-api-for-videos

Extract information, summarize, ask questions, and search videos using OpenAI's Vision API 🚀🎦
61 stars 9 forks source link

Session crashed when trying to run it #1

Open vonsander opened 11 months ago

vonsander commented 11 months ago

Hey!

Used the following parameters for an 18 second video (I'm looking to get 1 fps):

start_sec = 1  # @param {type:"integer"}
duration = 18   # @param {type:"integer"}
end_sec = start_sec + duration
num_frames_to_sample = 18  # @param {type:"integer"}
short_side_size = 512  # @param {type:"integer"}
max_tokens = 512  # @param {type:"integer"}
prompt = "These are frames from a video. Generate a description/summary that explains what is happening based on what you can see.\nYour response should **STRICTLY and ONLY** be the description, no titles, headers, or anything else." # @param {type: "string",description:"leav blank"}
show_frame_grid = True  # @param {type:"boolean"}

The session crashed, here's the log:

{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.146 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files","time":"2023-11-08T13:04:27.146Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.146 NotebookApp] Looking for jupyter_config in /etc/jupyter","time":"2023-11-08T13:04:27.154Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.150 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files","time":"2023-11-08T13:04:27.151Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.147 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter","time":"2023-11-08T13:04:27.155Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.150 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter","time":"2023-11-08T13:04:27.155Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.150 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter","time":"2023-11-08T13:04:27.155Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.150 NotebookApp] Looking for jupyter_config in /root/.jupyter","time":"2023-11-08T13:04:27.157Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.158 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter","time":"2023-11-08T13:04:27.163Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.158 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py","time":"2023-11-08T13:04:27.163Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.159 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter","time":"2023-11-08T13:04:27.163Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.159 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.163Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.159 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter","time":"2023-11-08T13:04:27.164Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.159 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter","time":"2023-11-08T13:04:27.165Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.159 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter","time":"2023-11-08T13:04:27.165Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 13:04:27.162 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py","time":"2023-11-08T13:04:27.167Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.150 NotebookApp] Looking for jupyter_config in /etc/jupyter","time":"2023-11-08T13:04:27.161Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.151 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter","time":"2023-11-08T13:04:27.162Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.151 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter","time":"2023-11-08T13:04:27.162Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.151 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter","time":"2023-11-08T13:04:27.164Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.151 NotebookApp] Looking for jupyter_config in /root/.jupyter","time":"2023-11-08T13:04:27.164Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.152 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter","time":"2023-11-08T13:04:27.164Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.154 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py","time":"2023-11-08T13:04:27.165Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.154 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter","time":"2023-11-08T13:04:27.166Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.155 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.168Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.155 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter","time":"2023-11-08T13:04:27.168Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.155 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter","time":"2023-11-08T13:04:27.168Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.156 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter","time":"2023-11-08T13:04:27.168Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 13:04:27.158 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py","time":"2023-11-08T13:04:27.169Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/etc/jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.767Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json","time":"2023-11-08T13:04:27.769Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.769Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/etc/jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.770Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/etc/jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.770Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/root/.local/etc/jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.771Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/root/.jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.772Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json","time":"2023-11-08T13:04:27.773Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.773Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/usr/etc/jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.774Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/root/.local/etc/jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.776Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/root/.jupyter/jupyter_notebook_config.json","time":"2023-11-08T13:04:27.777Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret","time":"2023-11-08T13:04:27.786Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Authentication of /metrics is OFF, since other authentication is disabled.","time":"2023-11-08T13:04:27.786Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret","time":"2023-11-08T13:04:27.787Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Authentication of /metrics is OFF, since other authentication is disabled.","time":"2023-11-08T13:04:27.788Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"google.colab serverextension initialized.","time":"2023-11-08T13:04:27.849Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"google.colab serverextension initialized.","time":"2023-11-08T13:04:27.854Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Serving notebooks from local directory: /","time":"2023-11-08T13:04:30.970Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Jupyter Notebook 6.5.5 is running at:","time":"2023-11-08T13:04:30.971Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"http://172.28.0.12:9000/","time":"2023-11-08T13:04:30.975Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).","time":"2023-11-08T13:04:30.976Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Serving notebooks from local directory: /","time":"2023-11-08T13:04:30.996Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Jupyter Notebook 6.5.5 is running at:","time":"2023-11-08T13:04:30.996Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"http://172.28.0.2:9000/","time":"2023-11-08T13:04:30.998Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).","time":"2023-11-08T13:04:30.998Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Kernel started: 04e7ae55-03fa-4cd7-ae2d-b90013400c58, name: python3","time":"2023-11-08T13:09:34.751Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"KernelRestarter: restarting kernel (1/5), keep random ports","time":"2023-11-08T13:38:01.751Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"WARNING:root:kernel 04e7ae55-03fa-4cd7-ae2d-b90013400c58 restarted","time":"2023-11-08T13:38:01.752Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"KernelRestarter: restarting kernel (1/5), keep random ports","time":"2023-11-08T13:40:16.759Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"WARNING:root:kernel 04e7ae55-03fa-4cd7-ae2d-b90013400c58 restarted","time":"2023-11-08T13:40:16.759Z","v":0}

What should I do? Thanks in forward! :)

nateraw commented 11 months ago

Uncheck the "show grid" box (or set it to False in the code). Maybe that helps? If not, it's likely because you're going OOM.

What runtime were you on? I'm using colab pro+ and used high memory machine.

In meantime...

Easy win here is to ffmpeg the video and rescale/downsample frame rate to 1-3 beforehand so it doesn't end up in memory. Or just use decord as I suggest in the notebook and select idxs efficiently. I'll update to do one of these 2 soon, as I think the notebook is too hard to read rn.

vonsander commented 11 months ago

Unchecked the "show grid", re-ran it and got the "Your session crashed after using all available RAM" message. I guess I need to upgrade to a paid Colab subscription :D Either that or try rescaling/downsampling the frame rate or use Decord.

P.S. Do you by any chance happen to sell consulting as well? Eg. on an hourly basis.

nateraw commented 11 months ago

Ah yea that makes sense. I will update the notebook as soon as I can here so its more accessible for folks without colab pro+. Thanks for the heads up.

As for consulting, you can shoot me an email at naterawdata@gmail.com to discuss. Most of my time is dedicated to another client right now, but open to conversation.