Non-empty parent folders being marked for purge

[benmarsden]

Example scenario:

I have the following structure in artifactory:

<local-repo>/<folderX>/<artifact>

This works fine when filtering the purge to item_type="file". However, if I want to also remove empty folders, Lavatory flags <folderX> for removal, as nested files/folders do not update the modified timestamp of the parent folder.

How have others resolved this? It seems like less of a Lavatory issue than an issue with Artifactory's indexing of the folders, but I'm hoping someone here may have run into this problem too.

[sijis]

Would you be able to provide a small snippet to reproduce this?

[benmarsden]

Hey @sijis! Sure... I've renamed/stripped content I cannot share, but this is the gist:

Artifactory repository:

Jfrog CLI: jfrog rt s example-repo --include-dirs Output:

[
  {
    "path": "example-repo/",
    "type": "folder",
    "created": "2020-05-13T13:39:54.818Z",
    "modified": "2020-05-13T13:39:54.818Z"
  },
  {
    "path": "example-repo/example",
    "type": "folder",
    "created": "2020-05-19T09:30:40.100Z",
    "modified": "2020-05-19T09:30:40.100Z"
  },
  {
    "path": "example-repo/example/example.yaml",
    "type": "file",
    "size": 7096,
    "created": "2020-07-30T06:43:26.855Z",
    "modified": "2020-07-30T06:43:26.836Z"
  }
]

Policy in use:

def purgelist(artifactory):
    """Purge artifacts older than 7 days"""
    purgable = artifactory.time_based_retention(keep_days=7, item_type="any")
    return purgable

Purge

Command: lavatory -v purge --no-default --dryrun --policies-path ./policies --repo example-repo

Output (snippet):

[INFO] lavatory.commands.purge Applying retention policies to example-repo
[INFO] lavatory.commands.purge Policy Docs: Purge artifacts older than 7 days
[DEBUG] lavatory.utils.artifactory AQL: {'$and': [{'created': {'$lt': '2020-07-23T09:17:55Z'}}, {'path': {'$nmatch': '*/repodata'}}, {'repo': {'$eq': 'example-repo'}}, {'type': {'$eq': 'any'}}]}
[INFO] lavatory.utils.artifactory Running mode: DRYRUN
[INFO] lavatory.utils.artifactory DRYRUN purge example-repo/./.
[INFO] lavatory.utils.artifactory DRYRUN purge example-repo/./example
[INFO] lavatory.utils.artifactory DRYRUN purge example-repo/example/example.zip

Issue

From the INFO logs, it would seem that Lavatory wishes to delete the /example folder, despite the fact that example-repo/example/example.yaml exists in this folder with a creation timestamp that does not meet the retention policy criteria for purge.

FYI later today I can also test the actual purge output and provide the logs, in the case actual behaviour differs to what I'm interpreting from the dry run logs... In the meantime any comments on this would be awesome!

[benmarsden]

FYI for anyone in a similar position: I reverted Lavatory to only remove files (item_type="file"), then combined it with Artifactory's deleteEmptyDirs plugin to remove the empty folders left over by Lavatory. Example Jenkins stage demonstrating its use:

stage("Purge Empty Directories"){
            steps{
                echo "======== Executing Empty Folder Purge ========"
                sh '''
                    export ARTIFACTORY_LOCAL_REPOS=\$(curl -v -u $ARTIFACTORY_USERNAME:$ARTIFACTORY_PASSWORD $ARTIFACTORY_URL/api/repositories?type=local | jq -c '.[] | .key' | jq -r -s 'join(",")')
                    echo "Running Empty Directory cleanup over the following local repositories: $ARTIFACTORY_LOCAL_REPOS..."
                    curl -X POST -v -u $ARTIFACTORY_USERNAME:$ARTIFACTORY_PASSWORD $ARTIFACTORY_URL/api/plugins/execute/deleteEmptyDirsPlugin?params=paths=$ARTIFACTORY_LOCAL_REPOS
                '''
            }
        }

Probably not the most elegant of solutions, but did the job. While I don't consider the issue fixed per se, I have no capacity to investigate it further, nor a need with the above workaround. Therefore @sijis feel welcome to close the issue.