[Question❓] ValueError: Conjugate gradient descent did not converge within 10000 iterations

[TatianaSnauwaert]

Hello!

First of all, thank you for your great work, I am using your library a lot and it's very efficient in removing background!

My issue is that recently I started getting this error after processing just a few images (2 out of 100, 18 out of 16k+ images): ValueError: Conjugate gradient descent did not converge within 10000 iterations

Like I said, I used it very often before and never faced this issue but now all of a sudden (nothing changed in terms of soft and hardware) it doesn't converge. I tried increasing the number of iterations to 20000 but it still fails. Is there anything else I can try or do you have any ideas what causes this error? Let me know if you need more details or what information can help you figure it out!

Thank you in advance! Best regards, Tatiana

[99991]

Hello Tatiana, a minimal reproducible example would be great.

Best regards, Thomas

[TatianaSnauwaert]

Hi Thomas,

Thanks for your reply! Sure, here is an example of a script I'm using.

I should mention that the size of images I'm trying to process is 1024*1024 px. I think I mostly used pymatting for smaller images up to 672 px before. Could the size do anything with this issue?

Thank you in advance and let me know if you need anything else! Regards, Tatiana

[99991]

Could you please add an image where the error occurs?

[TatianaSnauwaert]

For example, here is an image on which the matting stopped and threw the error. However, when I processed it again separately, it worked just fine...

[99991]

However, when I processed it again separately, it worked just fine...

That's odd. There should not be any state that is stored between function calls which could cause such an issue.

I should mention that the size of images I'm trying to process is 1024*1024 px. I think I mostly used pymatting for smaller images up to 672 px before. Could the size do anything with this issue?

I have not tested it with very very large images because it would take a long time to converge (although 1024x1024 is not that terribly large and should still work fine).

My only guess so far would be that perhaps the convergence condition was too restrictive. You could try passing a callback to the conjugate gradient method and print the norm of the residual vector and terminate earlier if it becomes "small enough". The meaning of "small enough" depends on the application, so I can not offer an exact value. I picked 400 for no good reason.

def remove_greenscreen(image):
    # Some hacky code to generate a trimap for green background
    r, g, b = image.transpose(2, 0, 1)
    is_bg = (r < 0.4) & (g > 0.3) & (b < 0.3)
    is_fg = np.logical_not(is_bg)
    x = np.linspace(-1, 1, 21)
    x, y = np.meshgrid(x, x)
    structure = x*x + y*y < 1.0
    is_fg = scipy.ndimage.morphology.binary_erosion(is_fg, structure=structure, border_value=1)
    is_bg = scipy.ndimage.morphology.binary_erosion(is_bg, structure=structure, border_value=1)
    trimap = 0.5 + 0.5 * is_fg - 0.5 * is_bg

    print("Computing alpha matte. This might take a while.")

    absolute_tolerance = 400.0

    class MyCallback:
        def __init__(self):
            self.n = 0

        def __call__(self, A, x, b, norm_b, r, norm_r):
            self.n += 1

            print("iteration %7d - %e (%.20f)" % (self.n, norm_r, norm_r))

            alpha = x.reshape(image.shape[:2]).copy()
            alpha[is_fg] = 1
            alpha[is_bg] = 0

            save_image(f"iterations/{self.n}.bmp", alpha)

    alpha = estimate_alpha_lkm(image, trimap, laplacian_kwargs={"radius": 30},
        cg_kwargs=dict(callback=MyCallback(), atol=absolute_tolerance))

    foreground = estimate_foreground_ml(image, alpha)

    cutout = stack_images(foreground, alpha)

    return cutout

Alternatively, you could throw an exception after a fixed number of iterations and use the alpha matte calculated up to this point. The API for that is admittedly a bit awkward. I think the easiest way would be to save the alpha matte from the callback and then throw an exception from the callback and catch it around the estimate_alpha_lkm call to stop it.

def remove_greenscreen(image):
    # Some hacky code to generate a trimap for green background
    r, g, b = image.transpose(2, 0, 1)
    is_bg = (r < 0.4) & (g > 0.3) & (b < 0.3)
    is_fg = np.logical_not(is_bg)
    x = np.linspace(-1, 1, 21)
    x, y = np.meshgrid(x, x)
    structure = x*x + y*y < 1.0
    is_fg = scipy.ndimage.morphology.binary_erosion(is_fg, structure=structure, border_value=1)
    is_bg = scipy.ndimage.morphology.binary_erosion(is_bg, structure=structure, border_value=1)
    trimap = 0.5 + 0.5 * is_fg - 0.5 * is_bg

    print("Computing alpha matte. This might take a while.")

    class MyCallback:
        def __init__(self):
            self.n = 0

        def __call__(self, A, x, b, norm_b, r, norm_r):
            self.n += 1

            print("iteration %7d - %e (%.20f)" % (self.n, norm_r, norm_r))

            if self.n >= 50:
                alpha = x.reshape(image.shape[:2]).copy()
                alpha[is_fg] = 1
                alpha[is_bg] = 0

                e = StopIteration()
                e.alpha = alpha
                raise e

    try:
        alpha = estimate_alpha_lkm(image, trimap, laplacian_kwargs={"radius": 30},
            cg_kwargs=dict(callback=MyCallback()))
    except StopIteration as e:
        alpha = e.alpha

    foreground = estimate_foreground_ml(image, alpha)

    cutout = stack_images(foreground, alpha)

    return cutout

[TatianaSnauwaert]

Hey Thomas, sorry, got busy with something else! Thanks a lot for your help and suggestions, I'll have a look soon and try to implement it!

[99991]

Sure! Let me know if you find a way to reproduce the issue you were having.