Empty list response for long-running requests

[btlorch]

Dear Replicate team,

since the upgrade to 1.0.2, I am experiencing the problem that replicate.run() often returns an empty list. I believe this occurs when the request takes longer response times. Apparently, the client does not wait long enough for the response. I did not pass any wait parameter, but I can see that run by default sets params["wait"] = True and is_blocking = True.

To understand the issue, I customized the run method to print out the request id. After the run method returned with an empty list, I looked up the request id in the recent predictions list on replicate.com. I can see that the status of the prediction is "Running". A minute later, the status on the webpage changed to "Succeeded". Looking at a specific example, the queued time was 22.9s and the runtime was 27.7s, total duration 50.6s.

Is there an easy wait to make the client wait for the prediction to succeed?

[btlorch]

I worked around it by implementing my own poll loop, similar to an example from the README. Would still be interested to know why run did not wait.

def poll(payload):
    """
    Create a prediction and poll for status changes.
    :param payload: input to the model
    :return: url of the response image, or None in case of an error
    """

    # Create prediction
    prediction = replicate.predictions.create(
        version=model_version,
        input=payload,
    )

    # Poll for status changes
    while prediction.status in {"starting", "processing"}:
        time.sleep(POLL_INTERVAL)
        prediction.reload()

    # Handle error status
    if prediction.status in {"failed", "canceled"}:
        log.error(f"Prediction returned with status {prediction.status}")
        return None

    # When the prediction succeeded, the response should be a list with a single URL
    if len(prediction.output) != 1:
        log.error(f"Expected 1 output URL, got {len(prediction.output)}")
        return None

    # Take single URL
    output_url = prediction.output[0]
    return output_url

[nickstenning]

Thank you for the report. This is very similar to https://github.com/replicate/replicate-javascript/issues/328, and we've been looking at that one today. It seems likely that they share a cause. We identified a bug that was causing us to return predictions with no output to the client libraries, and have hopefully fixed the most common scenarios in which that happens.

If you could let us know if you're still seeing this that would be super helpful.

[nickstenning]

This was a problem with the API itself sometimes returning a prediction object in an invalid state. That issue should now be fixed. Thanks again for the report!