Removing old rapid images doesn't work when layers were used before

Description:

When installing a new version of SAM cli, old existing rapid images are deleted (since #3092).

$ sam local invoke
Invoking app.lambda_handler (python3.8)
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-python3.8
...

However, if in the previous version of SAM cli I used layers for this runtime, then that older version of SAM cli created another image (like samcli/lambda:python3.8-d69b815d5b5c18e87ebbd6a1c) that depends on the rapid image that is being deleted, therefore an exception is thrown: conflict: unable to delete f9da007faf40 (cannot be forced) - image has dependent child images. Full stack trace:

Failed to remove rapid image with ID: sha256:f9da007faf408be9ed55f05b33686cbc672513a07620b47e81c0fa1f1b985b11
Traceback (most recent call last):
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.35/images/sha256:f9da007faf408be9ed55f05b33686cbc672513a07620b47e81c0fa1f1b985b11?force=False&noprune=False

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/samcli/local/docker/lambda_image.py", line 332, in _remove_rapid_images
    self.docker_client.images.remove(image.id)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/models/images.py", line 463, in remove
    self.client.api.remove_image(*args, **kwargs)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/api/image.py", line 495, in remove_image
    return self._result(res, True)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/api/client.py", line 267, in _result
    self._raise_for_status(response)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 409 Client Error: Conflict ("conflict: unable to delete f9da007faf40 (cannot be forced) - image has dependent child images")

This process of deleting the old images happens only during the first invoke (per runtime), so the exception doesn't appear on future invokes, but the first time it certainly looks like there was a big problem.

Steps to reproduce:

Install an older version of SAM cli (1.28.0 or later, to get images from ECR)
Create an function that uses a layer, build it and invoke it locally.
Install a newer version of SAM cli (1.30.0 or later)
Create a function using the same runtime as before and invoke it locally.
The function gets invoked correctly, but there's a big error message like the one displayed above.

Observed result:

(sam local invoke --debug output added at the end, but it's the same exception as above)

Expected result:

I'm not sure if this should delete all the layer images before deleting the rapid image, but at least it should handle that case gracefully instead of throwing the big exception, probably letting people know that that older image wasn't deleted successfully.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: AL2 (testing in an EC2 instance)
sam --version: SAM CLI, version 1.33.0, but it probably happens with every version after 1.30.0
AWS region: N/A

$ sam local invoke --debug
2021-10-26 14:57:41,683 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-10-26 14:57:41,684 | Using config file: samconfig.toml, config environment: default
2021-10-26 14:57:41,684 | Expand command line arguments to:
2021-10-26 14:57:41,684 | --template_file=/local/home/valerena/test/test-py8/.aws-sam/build/template.yaml --no_event --layer_cache_basedir=/home/valerena/.aws-sam/layers-pkg --container_host=localhost --container_host_
interface=127.0.0.1
2021-10-26 14:57:41,684 | local invoke command is called
2021-10-26 14:57:41,688 | No Parameters detected in the template
2021-10-26 14:57:41,711 | 2 stacks found in the template
2021-10-26 14:57:41,711 | No Parameters detected in the template
2021-10-26 14:57:41,730 | 2 resources found in the stack
2021-10-26 14:57:41,730 | No Parameters detected in the template
2021-10-26 14:57:41,749 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2021-10-26 14:57:41,749 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to /local/home/valerena/test/test-py8/.aws-sam/build/template.yaml
2021-10-26 14:57:41,755 | Found one Lambda function with name 'HelloWorldFunction'
2021-10-26 14:57:41,755 | Invoking app.lambda_handler (python3.8)
2021-10-26 14:57:41,755 | No environment variables found for function 'HelloWorldFunction'
2021-10-26 14:57:41,755 | Environment variables overrides data is standard format
2021-10-26 14:57:41,755 | Loading AWS credentials from session with profile 'None'
2021-10-26 14:57:41,767 | Resolving code path. Cwd=/local/home/valerena/test/test-py8/.aws-sam/build, CodeUri=/local/home/valerena/test/test-py8/.aws-sam/build/HelloWorldFunction
2021-10-26 14:57:41,767 | Resolved absolute path to code is /local/home/valerena/test/test-py8/.aws-sam/build/HelloWorldFunction
2021-10-26 14:57:41,767 | Code /local/home/valerena/test/test-py8/.aws-sam/build/HelloWorldFunction is not a zip/jar file
2021-10-26 14:57:41,768 | Image was not found.
2021-10-26 14:57:41,768 | Removing rapid images for repo public.ecr.aws/sam/emulation-python3.8
2021-10-26 14:57:41,796 | Failed to remove rapid image with ID: sha256:f9da007faf408be9ed55f05b33686cbc672513a07620b47e81c0fa1f1b985b11
Traceback (most recent call last):
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.35/images/sha256:f9da007faf408be9ed55f05b33686cbc672513a07620b47e81c0fa1f1b985b11?force=False&noprune=False

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/samcli/local/docker/lambda_image.py", line 332, in _remove_rapid_images
    self.docker_client.images.remove(image.id)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/models/images.py", line 463, in remove
    self.client.api.remove_image(*args, **kwargs)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/api/image.py", line 495, in remove_image
    return self._result(res, True)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/api/client.py", line 267, in _result
    self._raise_for_status(response)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/local/home/valerena/.pyenv/versions/3.6.8/envs/new-sam/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 409 Client Error: Conflict ("conflict: unable to delete f9da007faf40 (cannot be forced) - image has dependent child images")
Building image....................
2021-10-26 14:57:43,943 | Skip pulling image and use local one: public.ecr.aws/sam/emulation-python3.8:rapid-1.33.0-x86_64.

2021-10-26 14:57:43,943 | Mounting /local/home/valerena/test/test-py8/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
2021-10-26 14:57:44,433 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
START RequestId: 7909bd39-bf61-493a-84b2-d51daf058265 Version: $LATEST
END RequestId: 7909bd39-bf61-493a-84b2-d51daf058265
REPORT RequestId: 7909bd39-bf61-493a-84b2-d51daf058265  Init Duration: 0.13 ms  Duration: 63.26 ms      Billed Duration: 64 ms  Memory Size: 128 MB     Max Memory Used: 128 MB
{"statusCode": 200, "body": "{\"message\": \"hello world\"}"}2021-10-26 14:57:44,778 | Cleaning all decompressed code dirs

aws / aws-sam-cli