search

nautobot / nautobot-app-golden-config

Golden Configuration App for Nautobot.
https://docs.nautobot.com/projects/golden-config/en/latest/
Other
100 stars 57 forks source link

Execute All Jobs - Multiple Device doesn't get to compliance step if any device fails in previous jobs #692

Closed jeffkala closed 8 months ago

jeffkala commented 10 months ago

Environment

Expected Behavior

If any single device fails one of other tasks (backup config, generate config) the remaining successful devices should still be passed to the compliance job to run a new compliance test.

Observed Behavior

If any device fails backup, generation of config, subsequent jobs in the chain never get executed.

Steps to Reproduce

  1. Have two devices in NB 1 that is reachable and 1 that is not reachable
  2. Run the Execute All Golden Configuration Jobs - Multiple Device with both devices
  3. Notice that one the successful device the compliance job never gets run due to the failure that occurred for the other device.
nautobot-golden-config-worker-1    | nxos#
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,854: DEBUG/ForkPoolWorker-6] write_channel: b'exit\n'
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,866: DEBUG/ForkPoolWorker-6] EOF in transport thread
nautobot-golden-config-worker-1    | `E1004:` Could not connect to IP: `10.1.1.8` and port: `22`, preemptively failed.
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,977: ERROR/ForkPoolWorker-6] `E1004:` Could not connect to IP: `10.1.1.8` and port: `22`, preemptively failed.
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,980: ERROR/ForkPoolWorker-6] Host 'demo-cisco-xe': task 'check_connectivity' failed with traceback:
nautobot-golden-config-worker-1    | Traceback (most recent call last):
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/nornir/core/task.py", line 99, in start
nautobot-golden-config-worker-1    |     r = self.task(self, **self.params)
nautobot-golden-config-worker-1    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/nornir_nautobot/plugins/tasks/dispatcher/default.py", line 84, in check_connectivity
nautobot-golden-config-worker-1    |     raise NornirNautobotException(error_msg)
nautobot-golden-config-worker-1    | nornir_nautobot.exceptions.NornirNautobotException: `E1004:` Could not connect to IP: `10.1.1.8` and port: `22`, preemptively failed.
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,980: INFO/ForkPoolWorker-6] demo-cisco-xe | Task instance subtask check_connectivity has completed.
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,980: ERROR/ForkPoolWorker-6] Host 'demo-cisco-xe': task 'TEST CONNECTIVITY' failed with traceback:
nautobot-golden-config-worker-1    | Traceback (most recent call last):
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/nornir/core/task.py", line 99, in start
nautobot-golden-config-worker-1    |     r = self.task(self, **self.params)
nautobot-golden-config-worker-1    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/nornir_nautobot/plugins/tasks/dispatcher/__init__.py", line 62, in dispatcher
nautobot-golden-config-worker-1    |     result = task.run(task=driver_task, logger=logger, obj=obj, *args, **kwargs)
nautobot-golden-config-worker-1    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/nornir/core/task.py", line 174, in run
nautobot-golden-config-worker-1    |     raise NornirSubTaskError(task=run_task, result=r)
nautobot-golden-config-worker-1    | nornir.core.exceptions.NornirSubTaskError: Subtask: check_connectivity (failed)
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,980: INFO/ForkPoolWorker-6] demo-cisco-xe | Task instance subtask TEST CONNECTIVITY has completed.
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,981: ERROR/ForkPoolWorker-6] Host 'demo-cisco-xe': task 'BACKUP CONFIG' failed with traceback:
nautobot-golden-config-worker-1    | Traceback (most recent call last):
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/nornir/core/task.py", line 99, in start
nautobot-golden-config-worker-1    |     r = self.task(self, **self.params)
nautobot-golden-config-worker-1    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-golden-config-worker-1    |   File "/source/nautobot_golden_config/utilities/db_management.py", line 15, in inner
nautobot-golden-config-worker-1    |     func(*args, **kwargs)
nautobot-golden-config-worker-1    |   File "/source/nautobot_golden_config/nornir_plays/config_backup.py", line 61, in run_backup
nautobot-golden-config-worker-1    |     task.run(
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/nornir/core/task.py", line 174, in run
nautobot-golden-config-worker-1    |     raise NornirSubTaskError(task=run_task, result=r)
nautobot-golden-config-worker-1    | nornir.core.exceptions.NornirSubTaskError: Subtask: TEST CONNECTIVITY (failed)
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | BACKUP CONFIG failed: Subtask: TEST CONNECTIVITY (failed)
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,981: ERROR/ForkPoolWorker-6] BACKUP CONFIG failed: Subtask: TEST CONNECTIVITY (failed)
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | demo-cisco-xe, BACKUP CONFIG failed: Subtask: TEST CONNECTIVITY (failed)
nautobot-golden-config-worker-1    |  Subtask: TEST CONNECTIVITY (failed)
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,988: WARNING/ForkPoolWorker-6] demo-cisco-xe, BACKUP CONFIG failed: Subtask: TEST CONNECTIVITY (failed)
nautobot-golden-config-worker-1    |  Subtask: TEST CONNECTIVITY (failed)
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | BACKUP CONFIG failed: AggregatedResult (BACKUP CONFIG): {'demo-cisco-xe': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"]], 'nxos-0': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"], MultiResult: [Result: "SAVE BACKUP CONFIGURATION TO FILE", MultiResult: [Result: "get_config", Result: "netmiko_send_command"]]]}
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,990: ERROR/ForkPoolWorker-6] BACKUP CONFIG failed: AggregatedResult (BACKUP CONFIG): {'demo-cisco-xe': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"]], 'nxos-0': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"], MultiResult: [Result: "SAVE BACKUP CONFIGURATION TO FILE", MultiResult: [Result: "get_config", Result: "netmiko_send_command"]]]}
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,992: INFO/ForkPoolWorker-6] Running task 'close_connections_task' with args {} on 2 hosts
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,993: DEBUG/ForkPoolWorker-6] Host 'nxos-0': running task 'close_connections_task'
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,993: DEBUG/ForkPoolWorker-6] Host 'demo-cisco-xe': running task 'close_connections_task'
nautobot-golden-config-worker-1    | `E3001:` General Exception handler, original error message ```AggregatedResult (BACKUP CONFIG): {'demo-cisco-xe': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"]], 'nxos-0': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"], MultiResult: [Result: "SAVE BACKUP CONFIGURATION TO FILE", MultiResult: [Result: "get_config", Result: "netmiko_send_command"]]]}```
nautobot-golden-config-worker-1    | [2024-01-04 17:33:22,993: ERROR/ForkPoolWorker-6] `E3001:` General Exception handler, original error message ```AggregatedResult (BACKUP CONFIG): {'demo-cisco-xe': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"]], 'nxos-0': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"], MultiResult: [Result: "SAVE BACKUP CONFIGURATION TO FILE", MultiResult: [Result: "get_config", Result: "netmiko_send_command"]]]}```
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,004: ERROR/ForkPoolWorker-6] Task nautobot_golden_config.jobs.AllDevicesGoldenConfig[132f1a85-a32c-4a3f-b953-94187b3c6d0c] raised unexpected: NornirNautobotException('`E3001:` General Exception handler, original error message ```AggregatedResult (BACKUP CONFIG): {\'demo-cisco-xe\': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"]], \'nxos-0\': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"], MultiResult: [Result: "SAVE BACKUP CONFIGURATION TO FILE", MultiResult: [Result: "get_config", Result: "netmiko_send_command"]]]}```')
nautobot-golden-config-worker-1    | Traceback (most recent call last):
nautobot-golden-config-worker-1    |   File "/source/nautobot_golden_config/nornir_plays/config_backup.py", line 130, in config_backup
nautobot-golden-config-worker-1    |     nr_with_processors.run(
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/nornir/core/__init__.py", line 151, in run
nautobot-golden-config-worker-1    |     self.processors.task_completed(task, result)
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/nornir/core/processor.py", line 73, in task_completed
nautobot-golden-config-worker-1    |     p.task_completed(task, result)
nautobot-golden-config-worker-1    |   File "/source/nautobot_golden_config/nornir_plays/processor.py", line 19, in task_completed
nautobot-golden-config-worker-1    |     raise ValueError(result)
nautobot-golden-config-worker-1    | ValueError: AggregatedResult (BACKUP CONFIG): {'demo-cisco-xe': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"]], 'nxos-0': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"], MultiResult: [Result: "SAVE BACKUP CONFIGURATION TO FILE", MultiResult: [Result: "get_config", Result: "netmiko_send_command"]]]}
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | The above exception was the direct cause of the following exception:
nautobot-golden-config-worker-1    | 
nautobot-golden-config-worker-1    | Traceback (most recent call last):
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
nautobot-golden-config-worker-1    |     R = retval = fun(*args, **kwargs)
nautobot-golden-config-worker-1    |                  ^^^^^^^^^^^^^^^^^^^^
nautobot-golden-config-worker-1    |   File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 148, in __call__
nautobot-golden-config-worker-1    |     return self.run(*args, **deserialized_kwargs)
nautobot-golden-config-worker-1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-golden-config-worker-1    |   File "/source/nautobot_golden_config/jobs.py", line 244, in run
nautobot-golden-config-worker-1    |     BackupJob().run.__func__(self, **data)  # pylint: disable=too-many-function-args
nautobot-golden-config-worker-1    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-golden-config-worker-1    |   File "/source/nautobot_golden_config/jobs.py", line 181, in run
nautobot-golden-config-worker-1    |     config_backup(self.job_result, self.logger.getEffectiveLevel(), data)
nautobot-golden-config-worker-1    |   File "/source/nautobot_golden_config/nornir_plays/config_backup.py", line 143, in config_backup
nautobot-golden-config-worker-1    |     raise NornirNautobotException(error_msg) from error
nautobot-golden-config-worker-1    | nornir_nautobot.exceptions.NornirNautobotException: `E3001:` General Exception handler, original error message ```AggregatedResult (BACKUP CONFIG): {'demo-cisco-xe': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"]], 'nxos-0': MultiResult: [Result: "BACKUP CONFIG", MultiResult: [Result: "TEST CONNECTIVITY", Result: "check_connectivity"], MultiResult: [Result: "SAVE BACKUP CONFIGURATION TO FILE", MultiResult: [Result: "get_config", Result: "netmiko_send_command"]]]}```
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,005: DEBUG/ForkPoolWorker-6] nautobot_golden_config.jobs.AllDevicesGoldenConfig[132f1a85-a32c-4a3f-b953-94187b3c6d0c]: Finished the Execute All Golden Configuration Jobs - Multiple Device job execution.
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,008: DEBUG/ForkPoolWorker-6] nautobot_golden_config.jobs.AllDevicesGoldenConfig[132f1a85-a32c-4a3f-b953-94187b3c6d0c]: Pushing Execute All Golden Configuration Jobs - Multiple Device results to repo https://github.com/jeffkala/home-gc-generated.git.
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,010: DEBUG/ForkPoolWorker-6] Committing with message `EXECUTE ALL GOLDEN CONFIGURATION JOBS - MULTIPLE DEVICE JOB 2024-01-04 17:33:23.005701+00:00`
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,010: DEBUG/ForkPoolWorker-6] Popen(['git', 'status', '--porcelain', '--untracked-files'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,013: DEBUG/ForkPoolWorker-6] Popen(['git', 'add'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,014: DEBUG/ForkPoolWorker-6] Popen(['git', 'add', '--update'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,017: DEBUG/ForkPoolWorker-6] Popen(['git', 'cat-file', '--batch-check'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=False, shell=None, istream=<valid stream>)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,019: DEBUG/ForkPoolWorker-6] Popen(['git', 'cat-file', '--batch'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=False, shell=None, istream=<valid stream>)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,021: DEBUG/ForkPoolWorker-6] Commit completed
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,021: DEBUG/ForkPoolWorker-6] Push changes to repo
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,022: DEBUG/ForkPoolWorker-6] Popen(['git', 'push', '--porcelain', '--', 'origin'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=True, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,758: DEBUG/ForkPoolWorker-6] nautobot_golden_config.jobs.AllDevicesGoldenConfig[132f1a85-a32c-4a3f-b953-94187b3c6d0c]: Pushing Execute All Golden Configuration Jobs - Multiple Device results to repo https://github.com/jeffkala/home-gc-backups.git.
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,761: DEBUG/ForkPoolWorker-6] Committing with message `EXECUTE ALL GOLDEN CONFIGURATION JOBS - MULTIPLE DEVICE JOB 2024-01-04 17:33:23.005701+00:00`
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,761: DEBUG/ForkPoolWorker-6] Popen(['git', 'status', '--porcelain', '--untracked-files'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,763: DEBUG/ForkPoolWorker-6] Popen(['git', 'add'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,764: DEBUG/ForkPoolWorker-6] Popen(['git', 'add', '--update'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,768: DEBUG/ForkPoolWorker-6] Popen(['git', 'cat-file', '--batch-check'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=False, shell=None, istream=<valid stream>)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,771: DEBUG/ForkPoolWorker-6] Popen(['git', 'cat-file', '--batch'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=False, shell=None, istream=<valid stream>)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,773: DEBUG/ForkPoolWorker-6] Commit completed
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,773: DEBUG/ForkPoolWorker-6] Push changes to repo
nautobot-golden-config-worker-1    | [2024-01-04 17:33:23,774: DEBUG/ForkPoolWorker-6] Popen(['git', 'push', '--porcelain', '--', 'origin'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=True, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:24,513: DEBUG/ForkPoolWorker-6] nautobot_golden_config.jobs.AllDevicesGoldenConfig[132f1a85-a32c-4a3f-b953-94187b3c6d0c]: Pushing Execute All Golden Configuration Jobs - Multiple Device results to repo https://github.com/jeffkala/home-gc-generated.git.
nautobot-golden-config-worker-1    | [2024-01-04 17:33:24,516: DEBUG/ForkPoolWorker-6] Committing with message `EXECUTE ALL GOLDEN CONFIGURATION JOBS - MULTIPLE DEVICE JOB 2024-01-04 17:33:23.005701+00:00`
nautobot-golden-config-worker-1    | [2024-01-04 17:33:24,516: DEBUG/ForkPoolWorker-6] Popen(['git', 'status', '--porcelain', '--untracked-files'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:24,518: DEBUG/ForkPoolWorker-6] Popen(['git', 'add'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:24,520: DEBUG/ForkPoolWorker-6] Popen(['git', 'add', '--update'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:24,523: DEBUG/ForkPoolWorker-6] Popen(['git', 'cat-file', '--batch-check'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=False, shell=None, istream=<valid stream>)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:24,525: DEBUG/ForkPoolWorker-6] Popen(['git', 'cat-file', '--batch'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=False, shell=None, istream=<valid stream>)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:24,528: DEBUG/ForkPoolWorker-6] Commit completed
nautobot-golden-config-worker-1    | [2024-01-04 17:33:24,528: DEBUG/ForkPoolWorker-6] Push changes to repo
nautobot-golden-config-worker-1    | [2024-01-04 17:33:24,528: DEBUG/ForkPoolWorker-6] Popen(['git', 'push', '--porcelain', '--', 'origin'], cwd=/opt/nautobot/git/gen_cfgs, universal_newlines=True, shell=None, istream=None)
nautobot-golden-config-beat-1      | [2024-01-04 17:33:24,937: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
nautobot-golden-config-nautobot-1  | 17:33:24.997 INFO    django.server :
nautobot-golden-config-nautobot-1  |   "GET /api/extras/job-results/132f1a85-a32c-4a3f-b953-94187b3c6d0c/ HTTP/1.1" 200 4676
nautobot-golden-config-nautobot-1  | 17:33:25.199 INFO    django.server :
nautobot-golden-config-nautobot-1  |   "GET /extras/job-results/132f1a85-a32c-4a3f-b953-94187b3c6d0c/log-table/ HTTP/1.1" 200 19527
nautobot-golden-config-nautobot-1  | 17:33:25.242 INFO    django.server :
nautobot-golden-config-nautobot-1  |   "GET /extras/job-results/132f1a85-a32c-4a3f-b953-94187b3c6d0c/?tab=main HTTP/1.1" 200 237974
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,275: DEBUG/ForkPoolWorker-6] nautobot_golden_config.jobs.AllDevicesGoldenConfig[132f1a85-a32c-4a3f-b953-94187b3c6d0c]: Pushing Execute All Golden Configuration Jobs - Multiple Device results to repo https://github.com/jeffkala/home-gc-backups.git.
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,277: DEBUG/ForkPoolWorker-6] Committing with message `EXECUTE ALL GOLDEN CONFIGURATION JOBS - MULTIPLE DEVICE JOB 2024-01-04 17:33:23.005701+00:00`
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,278: DEBUG/ForkPoolWorker-6] Popen(['git', 'status', '--porcelain', '--untracked-files'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-nautobot-1  | 17:33:25.278 INFO    django.server :
nautobot-golden-config-nautobot-1  |   "GET /__debug__/history_sidebar/?store_id=a4552294bc344659a1f2896130577aa6 HTTP/1.1" 200 9502
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,280: DEBUG/ForkPoolWorker-6] Popen(['git', 'add'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,282: DEBUG/ForkPoolWorker-6] Popen(['git', 'add', '--update'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=False, shell=None, istream=None)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,284: DEBUG/ForkPoolWorker-6] Popen(['git', 'cat-file', '--batch-check'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=False, shell=None, istream=<valid stream>)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,287: DEBUG/ForkPoolWorker-6] Popen(['git', 'cat-file', '--batch'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=False, shell=None, istream=<valid stream>)
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,290: DEBUG/ForkPoolWorker-6] Commit completed
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,290: DEBUG/ForkPoolWorker-6] Push changes to repo
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,290: DEBUG/ForkPoolWorker-6] Popen(['git', 'push', '--porcelain', '--', 'origin'], cwd=/opt/nautobot/git/backup_cfgs, universal_newlines=True, shell=None, istream=None)
nautobot-golden-config-nautobot-1  | 17:33:25.408 INFO    django.server :
nautobot-golden-config-nautobot-1  |   "GET /extras/job-results/132f1a85-a32c-4a3f-b953-94187b3c6d0c/log-table/ HTTP/1.1" 200 19527
nautobot-golden-config-worker-1    | [2024-01-04 17:33:25,988: INFO/ForkPoolWorker-6] nautobot_golden_config.jobs.AllDevicesGoldenConfig[132f1a85-a32c-4a3f-b953-94187b3c6d0c]: Job completed
nautobot-golden-config-nautobot-1  | 17:33:29.669 INFO    django.server :
nautobot-golden-config-nautobot-1  |   "GET /extras/job-results/132f1a85-a32c-4a3f-b953-94187b3c6d0c/log-table/?page=2 HTTP/1.1" 200 16135
nautobot-golden-config-nautobot-1  | 17:33:29.913 INFO    django.server :
nautobot-golden-config-nautobot-1  |   "GET /__debug__/history_sidebar/?store_id=b4929b8602404d0cbda0316aadca938b HTTP/1.1" 200 9565
jeffkala commented 10 months ago

May need to either, move to celery primitives using job chaining, or write some try/except logic in the all-multiple job to still execute compliance even if a ValueError was raised from the processor in a previous job execution (backup or generation)

jeffkala commented 8 months ago

closed in #724