search

replicate / cog

Containers for machine learning
https://cog.run
Apache License 2.0
8.01k stars 560 forks source link

Infinite loops happening without logs #1665

Closed MaximeDde closed 4 months ago

MaximeDde commented 5 months ago

Hello there ! I'm opening an issue here as I wasn't able to find a solution on the mode l've been using in itself, and I am not sure it's related to it...

The model I use is https://replicate.com/thomasmol/whisper-diarization Working perfectly for several months, no issues with that. But recently, for a reason I cannot find, I find myself with predicts that go in infinite loops, without logs nor warnings, and obviously cost me quite a lot (sometimes looping for one or two hours before I notice it...).

I just had one with the prediction ID 0e120sfsasrgp0cffm4bvtnb5c By any chance, could someone help me identify if there may be something wrong on the Cog side that creates this behavior ?

Best regards,

mattt commented 4 months ago

Hi @MaximeDde. Sorry that you're having problems with this model.

thomasmol/whisper-diarization is a community model by a Replicate user. I noticed that several new versions were pushed in the last few months, so if your code was running the latest version, that could explain the changes you observed. If you pin to a specific version SHA, that should ensure consistent behavior.

It looks like the source code for the model is available here. So if you wanted to make any other changes, you could push your own version and use that instead.