devopshq / artifactory-cleanup

Extended cleanup tool for JFrog Artifactory
MIT License
123 stars 65 forks source link

Problems with trashcan and conan #110

Open Stunkymonkey opened 1 year ago

Stunkymonkey commented 1 year ago

using the example policy from README will fail if the trashcan is enabled for artifactory:

  policies:
    - name: Conan - delete empty folders (to fix the index)
      rules:
        - rule: Repo
          name: "conan-testing"
        - rule: DeleteEmptyFolders
        - rule: ExcludeFilename
          masks:
            - ".timestamp"
            - "index.json"

looks for me like: https://jfrog.atlassian.net/browse/RTFACT-11898 (but is already marked as resolved)

Checking 'ops-conan-dev-local' repository exists.
The ops-conan-dev-local repository exists.
Add AQL Filter - rule: Repo - Apply the policy to one repository.
Before AQL query: []
After AQL query: [{'repo': {'$eq': 'ops-conan-dev-local'}}]
Add AQL Filter - rule: DeleteEmptyFolders - Remove empty folders.
Before AQL query: [{'repo': {'$eq': 'ops-conan-dev-local'}}]
After AQL query: [{'repo': {'$eq': 'ops-conan-dev-local'}}, {'path': {'$match': '**'}, 'type': {'$eq': 'any'}}]
Add AQL Filter - rule: ExcludeFilename - Exclude artifacts by filename.
Before AQL query: [{'repo': {'$eq': 'ops-conan-dev-local'}}, {'path': {'$match': '**'}, 'type': {'$eq': 'any'}}]
After AQL query: [{'repo': {'$eq': 'ops-conan-dev-local'}}, {'path': {'$match': '**'}, 'type': {'$eq': 'any'}}, {'$and': [{'name': {'$nmatch': '.timestamp'}}, {'name': {'$nmatch': 'index.json'}}]}]
Add AQL Text - rule: Repo - Apply the policy to one repository.
Add AQL Text - rule: DeleteEmptyFolders - Remove empty folders.
Add AQL Text - rule: ExcludeFilename - Exclude artifacts by filename.
********************************************************************************
Result AQL Query:
items.find({"$and": [{"repo": {"$eq": "ops-conan-dev-local"}}, {"path": {"$match": "**"}, "type": {"$eq": "any"}}, {"$and": [{"name": {"$nmatch": ".timestamp"}}, {"name": {"$nmatch": "index.json"}}]}]}).include("*", "property", "stat")
********************************************************************************
Found 75638 artifacts
Filter artifacts - rule: Repo - Apply the policy to one repository.
Filter artifacts - rule: DeleteEmptyFolders - Remove empty folders.
Before count: 75638
After count: 6976
Filter artifacts - rule: ExcludeFilename - Exclude artifacts by filename.
Found 6976 artifacts AFTER filtering
DESTROY MODE - delete 'ops-conan-dev-local/Demo - 0B'
Traceback (most recent call last):
  File "/usr/local/bin/artifactory-cleanup", line 8, in <module>
    sys.exit(ArtifactoryCleanupCLI())
  File "/usr/local/lib/python3.8/dist-packages/plumbum/cli/application.py", line 177, in __new__
    return cls.run()
  File "/usr/local/lib/python3.8/dist-packages/plumbum/cli/application.py", line 634, in run
    retcode = inst.main(*tailargs)
  File "/usr/local/lib/python3.8/dist-packages/artifactory_cleanup/cli.py", line 164, in main
    for summary in cleanup.cleanup(
  File "/usr/local/lib/python3.8/dist-packages/artifactory_cleanup/artifactorycleanup.py", line 59, in cleanup
    policy.delete(artifact, destroy=self.destroy)
  File "/usr/local/lib/python3.8/dist-packages/artifactory_cleanup/rules/base.py", line 313, in delete
    r.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error:  for url: https://artifactory.mysecretdomain.de/artifactory/ops-conan-dev-local/Demo

the file is already in the trashcan. So something is not correct with the AQL, or the AQL itself is wrong.

I tried something like, but with no success:

        - rule: PropertyNeq
          property_key: module.artifact.item.repo
          property_value: auto-trashcan
Stunkymonkey commented 1 year ago

I added:

        - rule: ExcludePath
          masks: "*Demo*"

this fixes my problem for now. Not sure where this file is from. Not visible via UI.

we can close it from my side. EDIT: nope same error as before

allburov commented 1 year ago

Hi! Thank you for sharing the solution! Could one add a suggested solution to the Readme? I think we could add it as commented rule with a short explanation for trash can

Stunkymonkey commented 1 year ago

@allburov sorry eddited my response just 5 min after i posted it. Still a bug.