search

microsoft / service-fabric

Service Fabric is a distributed systems platform for packaging, deploying, and managing stateless and stateful distributed applications and containers at large scale.
https://docs.microsoft.com/en-us/azure/service-fabric/
MIT License
3.03k stars 401 forks source link

Remove-ServiceFabricComposeDeployment does not remove docker image #660

Open ifle opened 5 years ago

ifle commented 5 years ago

Remove-ServiceFabricComposeDeployment does remove the deployment, but does not remove the docker image. That cause to out of memory in case of the nightly builds.

jiqh commented 5 years ago

Hi @ifle, what's your cluster version you have this problem? And in fact it takes a while for the command to remove the image. Could you take a look if you wait some time whether the image is deleted? Thanks!

ifle commented 5 years ago

We use 6.4.622.9590 version of cluster based on aspnet:4.7.1-windowsservercore-1709 image. Take a look on following image. All older images did nor removed. What is interesting that SF does not update image with latest tag.

image

jiqh commented 5 years ago

@ifle Yes by default we won't delete image. If you call create/upgrade with a image downloaded from remote repo, that image will remain in local. Remove will only remove container created from the image. You can use "docker container -a" to see containers are removed.

As for tag, if you specify tag in the compose file, like this way: image: aspnet:4.7.1-windowsservercore-1709:20190106.6 it will download the image with the specific tag 20190106.6. If you don't specify tag, it will only download the one with latest tag by docker default.

ifle commented 5 years ago

Yes by default we won't delete image

Thats is strange behaviour. If SF download it, why is does not delete it? How can I remove older images during devops?

If you don't specify tag, it will only download the one with latest tag by docker default.

The problem that SF does not download the new version of image with default tag like :latest

Thanks

jiqh commented 5 years ago

I confirmed with my team member, and images can be removed. Hosting will check and remove unused images periodically. The default interval is 60 minutes. You can modify this config to set the pruning interval: INTERNAL_CONFIG_ENTRY(Common::TimeSpan, L"Hosting", CacheCleanupScanInterval, Common::TimeSpan::FromSeconds(3600), Common::ConfigEntryUpgradePolicy::Static);

For the tag, I'm sorry I didn't get your problem. Do you mean you were not downloading the newest version for image aspnet:4.7.1-windowsservercore-1709?

ifle commented 5 years ago

Sorry, but I don't understand how CacheCleanupScanInterval can help me.

For the tag, I'm sorry I didn't get your problem. Do you mean you were not downloading the newest version for image aspnet:4.7.1-windowsservercore-1709?

Yes, service fabric does not download the new version of image with the same tag

jiqh commented 5 years ago

When you call Remove-ServiceFabricComposeDeployment, the image won't be deleted. The Hosting component in Service Fabric will periodically check whether the image is used by any application. If not used, Hosting will remove the image. The time interval that Hosting does the check is defined by CacheCleanupScanInterval. So you can change it to a smaller value to delete the image in time.

ifle commented 5 years ago

In my SF nodes images did not removed and it going to out of the memory. How can to check what is wrong? image

jiqh commented 5 years ago

Could you provide me some information about how do you create compose deployment? I see same image repository with different tags are presented. Thanks!

ifle commented 5 years ago

We use to azure devops. The docker compose file contains 3 services. The name of containers do not change during build\release pipeline The tag of container is represent the buildId of build pipeline The release pipeline has two steps related to compose deployment:

  1. Remove compose deployment
  2. Create new one

The DeploymentName is the same in each build\release

image