search

devopshq / artifactory-cleanup

Extended cleanup tool for JFrog Artifactory
MIT License
125 stars 66 forks source link
artifactory artifactory-api artifactory-cleanup cleanup hacktoberfest jfrog jfrog-artifactory

readme

Artifactory cleanup

artifactory-cleanup is an extended and flexible cleanup tool for JFrog Artifactory.

The tool has simple YAML-defined cleanup configuration and can be extended with your own rules on Python. Everything must be as a code, even cleanup policies!

Tables of Contents

Installation

As simple as one command!

# docker
docker pull devopshq/artifactory-cleanup
docker run --rm devopshq/artifactory-cleanup artifactory-cleanup --help

# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup
artifactory-cleanup --help

Usage

Suppose you want to remove all artifacts older than N days from reponame repository. You should take the following steps:

  1. Install artifactory-cleanup (see above)
  2. Create a configuration file artifactory-cleanup.yaml. variables.
# artifactory-cleanup.yaml
artifactory-cleanup:
  server: https://repo.example.com/artifactory
  # $VAR is auto populated from environment variables
  user: $ARTIFACTORY_USERNAME
  password: $ARTIFACTORY_PASSWORD

  policies:
    - name: Remove all files from repo-name-here older than 7 days
      rules:
        - rule: Repo
          name: "reponame"
        - rule: DeleteOlderThan
          days: 7
  1. Run the command TO SHOW (not remove) artifacts that will be deleted. By default artifactory-cleanup uses "dry mode".
# Set the credentials with delete permissions
export ARTIFACTORY_USERNAME=usernamehere
export ARTIFACTORY_PASSWORD=password

# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup

# cli
artifactory-cleanup
  1. Verify that right artifacts will be removed and add --destroy flag TO REMOVE artifacts:
# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup --destroy

# cli
artifactory-cleanup --destroy

Looking for more examples? Check examples folder!

Notes

# docker
docker pull devopshq/artifactory-cleanup:1.0.0
docker run --rm devopshq/artifactory-cleanup:1.0.0 artifactory-cleanup --version

# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup==1.0.0
artifactory-cleanup --help

Commands

# Debug - "dry run" mode by default
# debug run - only print artifacts. it does not delete any artifacts
artifactory-cleanup

# Debug run only for policytestname.
artifactory-cleanup --policy-name policytestname

# REMOVE
# For remove artifacts use --destroy
artifactory-cleanup --destroy

# For remove artifacts use environment variable
export ARTIFACTORY_CLEANUP_DESTROY=True
artifactory-cleanup

# Specify config filename
artifactory-cleanup --config artifactory-cleanup.yaml

# Specify config filename using environment variable
export ARTIFACTORY_CLEANUP_CONFIG_FILE=artifactory-cleanup.yaml
artifactory-cleanup --config artifactory-cleanup.yaml

# Look in the future - shows what the tool WILL remove after 10 days
artifactory-cleanup --days-in-future=10

# Not satisfied with built-in rules? Write your own rules in python and connect them!
artifactory-cleanup --load-rules=myrule.py
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py

# Save the table summary in a file
artifactory-cleanup --output=myfile.txt

# Save the summary in a json file
artifactory-cleanup --output=myfile.txt --output-format=json

# Save the summary in a json file and append the list of all removed artifacts
artifactory-cleanup --output=myfile.json --output-format json --output-artifacts

Rules

Common

- rule: Repo
  name: reponame
# OR - if you have a single policy for the repo - you can name the policy as reponame
# Both configurations are equal
policies:
  - name: reponame
    rules:
      - rule: Repo
- rule: RepoList
  repos:
    - repo1
    - repo2
    - repo3
- rule: RepoByMask
  mask: "*.banned"
- rule: PropertyEq
  property_key: key-name
  property_value: 1
- rule: PropertyNeq
  property_key: key-name
  property_value: 1

Delete

- rule: DeleteOlderThan
  days: 1
- rule: DeleteWithoutDownloads
- rule: DeleteOlderThanNDaysWithoutDownloads
  days: 1
- rule: DeleteNotUsedSince
  days: 1
- rule: DeleteEmptyFolders
- rule: DeleteByRegexpName
  regex_pattern: "\d"

Keep

- rule: KeepLatestNFiles
  count: 1
- rule: KeepLatestNFilesInFolder
  count: 1
- rule: KeepLatestVersionNFilesInFolder
  count: 1
  custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"
- rule: KeepLatestNupkgNVersions
  count: 1

Docker

- rule: DeleteDockerImagesOlderThan
  days: 1
- rule: DeleteDockerImagesOlderThanNDaysWithoutDownloads
  days: 1
- rule: DeleteDockerImagesNotUsed
  days: 1
- rule: IncludeDockerImages
  masks: "*singlemask*"
- rule: IncludeDockerImages
  masks:
    - "*production*"
    - "*release*"
- rule: ExcludeDockerImages
  masks:
    - "*production*"
    - "*release*"
- rule: KeepLatestNVersionImagesByProperty
  count: 1
  custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"

Filters

- rule: IncludePath
  masks: "*production*"
- rule: IncludePath
  masks:
   - "*production*"
   - "*develop*"
- rule: IncludeFilename
  masks:
   - "*production*"
   - "*develop*"
- rule: ExcludePath
  masks:
   - "*production*"
   - "*develop*"
- rule: ExcludeFilename
  masks:
    - "*.tag.gz"
    - "*.zip"

Create your own rule

If you want to create your own rule, you can do it!

The basic flow how the tool calls Rules:

  1. Rule.check(*args, **kwargs) - verify that the Rule configured right. Call other services to get more information.
  2. Rule.aql_add_filter(filters) - add Artifactory Query Language expressions
  3. Rule.aql_add_text(aql) - add text to the result aql query
  4. artifactory-cleanup calls Artifactory with AQL and pass the result to the next step
  5. Rule.filter(artifacts) - filter out artifacts. The method returns artifacts that will be removed!.
    • To keep artifacts use artifacts.keep(artifact) method

Create myrule.py file at the same folder as artifactory-cleanup.yaml:

# myrule.py
from typing import List

from artifactory_cleanup import register
from artifactory_cleanup.rules import Rule, ArtifactsList

class MySimpleRule(Rule):
    """
    This doc string is used as rule title

    For more methods look at Rule source code
    """

    def __init__(self, my_param: str, value: int):
        self.my_param = my_param
        self.value = value

    def aql_add_filter(self, filters: List) -> List:
        print(f"Today is {self.today}")
        print(self.my_param)
        print(self.value)
        return filters

    def filter(self, artifacts: ArtifactsList) -> ArtifactsList:
        """I'm here just to print the list"""
        print(self.my_param)
        print(self.value)
        # You can make requests to artifactory by using self.session:
        # url = f"/api/storage/{self.repo}"
        # r = self.session.get(url)
        # r.raise_for_status()
        return artifacts

# Register your rule in the system
register(MySimpleRule)

Use rule: MySimpleRule in configuration:

# artifactory-cleanup.yaml
- rule: MySimpleRule
  my_param: "Hello, world!"
  value: 42

Specify --load-rules to the command:

# docker
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py

# cli
artifactory-cleanup --load-rules=myrule.py

How to

How to connect self-signed certificates for docker?

In case you have set up your Artifactory self-signed certificates, place all certificates of the chain of trust into the certificates folder and add additional argument to the command:

docker run -v "$(pwd)":/app -v "$(pwd)/certificates":/mnt/self-signed-certs/ devopshq/artifactory-cleanup artifactory-cleanup

How to clean up Conan repository?

We can handle conan's metadata by creating two policies:

  1. First one removes files but keep all metadata.
  2. Second one look at folders and if it contains only medata files - removes it (because there's no associated with metadata files)

The idea came from https://github.com/devopshq/artifactory-cleanup/issues/47

# artifactory-cleanup.yaml
artifactory-cleanup:
  server: https://repo.example.com/artifactory
  user: $ARTIFACTORY_USERNAME
  password: $ARTIFACTORY_PASSWORD

  policies:
    - name: Conan - delete files older than 60 days
      rules:
        - rule: Repo
          name: "conan-testing"
        - rule: DeleteNotUsedSince
          days: 60
        - rule: ExcludeFilename
          masks:
            - ".timestamp"
            - "index.json"
    - name: Conan - delete empty folders (to fix the index)
      rules:
        - rule: Repo
          name: "conan-testing"
        - rule: DeleteEmptyFolders
        - rule: ExcludeFilename
          masks:
            - ".timestamp"
            - "index.json"

How to keep latest N docker images?

We can combine docker rules with usual "files" rules!

The idea came from https://github.com/devopshq/artifactory-cleanup/issues/61

# artifactory-cleanup.yaml
artifactory-cleanup:
  server: https://repo.example.com/artifactory
  user: $ARTIFACTORY_USERNAME
  password: $ARTIFACTORY_PASSWORD

  policies:
    - name: Remove docker images, but keep last 3
      rules:
        # Select repo
        - rule: Repo
          name: docker-demo
        # Delete docker images older than 30 days
        - rule: DeleteDockerImagesOlderThan
          days: 30
        # Keep these tags for all images
        - rule: ExcludeDockerImages
          masks:
            - "*:latest"
            - "*:release*"
        # Exclude these docker tags
        - rule: ExcludePath
          masks: "*base-tools*"
        # Keep 3 docker tags for all images
        - rule: KeepLatestNFilesInFolder
          count: 3

Release

In order to provide a new release of artifactory-cleanup, there are two steps involved.

  1. Bump the version in the setup.py
  2. Bump the version in the init.py
  3. Create a Git release tag (in format 1.0.1) by creating a release on GitHub