Model takes several minutes to download occasionally

[bertagknowles]

When running my json workflow through api in my application, most of the times it downloads the required models and runs within a minute. However at times it takes several minutes and when I checked the logs, it showed that it took 4 minutes to download a model. Why does this happen occasionally and is there a way to ensure that if the models take more than a minute to download, the prediction is cancelled or errored out? This extra time adds to my billing :(

[aichromator]

Yes, same issue. especially when multiple models are options available for a custom node, it loads all. How do we solve it?

[fofr]

My best guess here is that this is a cache miss on Replicate's side, meaning the weights download slower. Any subsequent requests shouldn't have that problem.

[bertagknowles]

Hmm, then I think I'll create an automatic cancel after a set time in the application side.

[bertagknowles]

The problem is that if I use a timer to cancel requests after one minute, then the requests that are waiting in queue will also be cancelled since sometimes in public models the queue can also take more time. Will it be possible for you to add in the backend that if the download time exceeds one minute or if the cache is missed then the request should time out/fail? I think most models even ones around 23gb like flux etc should not take more than a minute to download from a parallel cache. This will save unnecessary billing costs for everyone.

[bertagknowles]

Got this error and was billed unnecessarily for 30 minutes :( ....something is broken....this is not reliable for api

[fofr]

@bertagknowles FYI, you are not billed for failed predictions. You won't be billed for this.

[bertagknowles]

Oh, that's a big relief...thanks for letting me know.

[bertagknowles]

However, there is something wrong with the model lately and the output file is not being shown (blank).