daemon/purge not working - python/api error

[swooton]

So my config seems to be working properly as it is pulling the backups list from API but it seems to be failing to delete with the daemon. I had it set up as an @reboot crontab so I figured that was causing it to not work, but testing via purge manually gives the following output:

ciscodnacbackupctl purge --keep 14 BACKUP_ID BACKUP_SIZE COMPATIBLE DESCRIPTION END_TIMESTAMP MAGLEV_ADDRESS_FAMILY START_TIMESTAMP STATUS a90725f2-8f6f-4e39-955e-0fee49fede6b 121G TRUE daily-dnac-backups 2022-06-14 01:06:47 ipv4 2022-06-14 01:00:35 SUCCESS f81ccc91-7076-4f77-aba8-7f021e610903 108G TRUE daily-dnac-backups 2022-06-11 01:07:44 ipv4 2022-06-11 01:00:31 SUCCESS 84db850f-1ec0-403e-ba8c-332bf2f51724 103G TRUE daily-dnac-backups 2022-06-10 01:34:06 ipv4 2022-06-10 01:00:33 SUCCESS 11e5af51-a93a-4586-bfe9-eb323e15390f 69G TRUE secondbackup622022 2022-06-02 11:37:09 ipv4 2022-06-02 11:17:44 SUCCESS 6fec1cf4-4e8d-4481-a050-ba93212b3b7d 7G TRUE initialbackups622022 2022-06-02 10:17:08 ipv4 2022-06-02 10:15:56 SUCCESS Warning: Confirm if you want to delete these backups (y/n): y Deleting... (this could take a while - as it's synchronous API calls) Traceback (most recent call last): File "/home/netops/.local/bin/ciscodnacbackupctl", line 8, in sys.exit(entry()) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/cli.py", line 367, in entry cli(obj={}) File "/home/netops/.local/lib/python3.9/site-packages/click/core.py", line 1130, in call return self.main(args, kwargs) File "/home/netops/.local/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/netops/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/netops/.local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/netops/.local/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(args, *kwargs) File "/home/netops/.local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), args, **kwargs) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/cli.py", line 283, in purge purge = cli.purge(keep=keep, incompatible=incompatible, force=force) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/init.py", line 574, in purge self.delete(backup_id) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/init.py", line 357, in delete task(id) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/init.py", line 344, in task data = self.api._request(type="delete", url=url) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/init.py", line 234, in _request "Error: ({})".format(data["response"].get("error", "Not Available")) KeyError: 'response'

[robertcsapo]

Could you share the version you are using?

ciscodnacbackupctl --version

[swooton]

$ ciscodnacbackupctl --version ciscodnacbackupctl, version 0.2.10

[robertcsapo]

ciscodnacbackupctl --debug will display more details about the error. (remove sensitive data)

[swooton]

Looks like a 504 Gateway time-out. Super weird, I use the API heavily and have never received that, just get rate-limited every once in a while but the sdk usually handles it for me.

[netops@asdf-dnacbackups01pa ~]$ ciscodnacbackupctl --debug purge --keep 14 Debug mode: True send: b'POST /dna/system/api/v1/auth/token HTTP/1.1\r\nHost: dnac.asdf.edu\r\nUser-Agent: python-requests/2.28.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json\r\nConnection: keep-alive\r\nContent-Type: application/json\r\nContent-Length: 0\r\nAuthorization: Basic asdf=\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: application/json header: Content-Length: 742 header: Connection: keep-alive header: Date: Wed, 29 Jun 2022 15:46:17 GMT header: X-Password-Expiry-Days: -1, -1, -1 header: Server: webserver header: x-request-id: aebf51bf2b731c9cc66c5b83e619b7fd header: Vary: Origin header: Access-Control-Allow-Origin: dnac.asdf.edu header: Via: api-gateway header: Cache-Control: no-store header: Pragma: no-cache header: Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' blob: data: header: X-Content-Type-Options: nosniff header: X-XSS-Protection: 1 header: Strict-Transport-Security: max-age=31536000; includeSubDomains header: X-Frame-Options: SAMEORIGIN send: b'GET /api/system/v1/maglev/backup HTTP/1.1\r\nHost: dnac.asdf.edu\r\nUser-Agent: python-requests/2.28.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json\r\nConnection: keep-alive\r\nX-Auth-Token: asdf\r\nContent-Type: application/json\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: application/json; charset=utf-8 header: Content-Length: 969135 header: Connection: keep-alive header: last-modified: Wed, 29 Jun 2022 15:46:19 -0000 header: DATE: Thu, 09 Jun 2022 11:03:01 GMT header: Server: webserver header: x-request-id: 2f9f886f76da35897a83a8f7f1e20edf header: Vary: Origin header: Access-Control-Allow-Origin: dnac.asdf.edu header: Via: api-gateway header: Cache-Control: no-store header: Pragma: no-cache header: Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' blob: data: header: X-Content-Type-Options: nosniff header: X-XSS-Protection: 1 header: Strict-Transport-Security: max-age=31536000; includeSubDomains header: X-Frame-Options: SAMEORIGIN BACKUP_ID BACKUP_SIZE COMPATIBLE DESCRIPTION END_TIMESTAMP MAGLEV_ADDRESS_FAMILY START_TIMESTAMP STATUS d17b13f5-7064-4f31-ab94-cba3246379a1 126G TRUE daily-dnac-backups 2022-06-15 01:07:18 ipv4 2022-06-15 01:00:36 SUCCESS 84db850f-1ec0-403e-ba8c-332bf2f51724 103G TRUE daily-dnac-backups 2022-06-10 01:34:06 ipv4 2022-06-10 01:00:33 SUCCESS 11e5af51-a93a-4586-bfe9-eb323e15390f 69G TRUE secondbackup622022 2022-06-02 11:37:09 ipv4 2022-06-02 11:17:44 SUCCESS 6fec1cf4-4e8d-4481-a050-ba93212b3b7d 7G TRUE initialbackups622022 2022-06-02 10:17:08 ipv4 2022-06-02 10:15:56 SUCCESS Warning: Confirm if you want to delete these backups (y/n): y Deleting... (this could take a while - as it's synchronous API calls) send: b'GET /api/system/v1/maglev/backup HTTP/1.1\r\nHost: dnac.asdf.edu\r\nUser-Agent: python-requests/2.28.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json\r\nConnection: keep-alive\r\nX-Auth-Token: asdf\r\nContent-Type: application/json\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: application/json; charset=utf-8 header: Content-Length: 969135 header: Connection: keep-alive header: last-modified: Wed, 29 Jun 2022 15:46:22 -0000 header: DATE: Thu, 09 Jun 2022 11:03:03 GMT header: Server: webserver header: x-request-id: 4c215486a38551ef346ba664adaa6fcf header: Vary: Origin header: Access-Control-Allow-Origin: dnac.asdf.edu header: Via: api-gateway header: Cache-Control: no-store header: Pragma: no-cache header: Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' blob: data: header: X-Content-Type-Options: nosniff header: X-XSS-Protection: 1 header: Strict-Transport-Security: max-age=31536000; includeSubDomains header: X-Frame-Options: SAMEORIGIN send: b'DELETE /api/system/v1/maglev/backup/d17b13f5-7064-4f31-ab94-cba3246379a1 HTTP/1.1\r\nHost: dnac.asdf.edu\r\nUser-Agent: python-requests/2.28.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json\r\nConnection: keep-alive\r\nX-Auth-Token: asdf\r\nContent-Type: application/json\r\nContent-Length: 0\r\n\r\n' reply: 'HTTP/1.1 504 Gateway Time-out\r\n' header: Date: Wed, 29 Jun 2022 15:47:22 GMT header: Content-Type: application/json header: Transfer-Encoding: chunked header: Connection: keep-alive header: x-request-id: 99abd7dc4564c00679373df509b65fdc header: Vary: Origin header: Access-Control-Allow-Origin: dnac.asdf.edu header: Server: kong/0.14.1 header: Cache-Control: no-store header: Pragma: no-cache header: Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' blob: data: header: X-Content-Type-Options: nosniff header: X-XSS-Protection: 1 header: Strict-Transport-Security: max-age=31536000; includeSubDomains header: X-Frame-Options: SAMEORIGIN Traceback (most recent call last): File "/home/netops/.local/bin/ciscodnacbackupctl", line 8, in sys.exit(entry()) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/cli.py", line 367, in entry cli(obj={}) File "/home/netops/.local/lib/python3.9/site-packages/click/core.py", line 1130, in call return self.main(args, kwargs) File "/home/netops/.local/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/netops/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/netops/.local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/netops/.local/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(args, *kwargs) File "/home/netops/.local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), args, **kwargs) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/cli.py", line 283, in purge purge = cli.purge(keep=keep, incompatible=incompatible, force=force) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/init.py", line 574, in purge self.delete(backup_id) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/init.py", line 357, in delete task(id) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/init.py", line 344, in task data = self.api._request(type="delete", url=url) File "/home/netops/.local/lib/python3.9/site-packages/ciscodnacbackupctl/init.py", line 234, in _request "Error: ({})".format(data["response"].get("error", "Not Available")) KeyError: 'response'

[robertcsapo]

14 was also facing 504 timeouts

is this assurance (NFS) backup or automation (rsync) backup?

[swooton]

This might be the exact same issue then. In my case, the first backup is not deleted, it doesn't delete anything.

This is both assurance and automation

[robertcsapo]

Assurance backups isn't supported. I'll add that to the README.