search

ibm-mas / ansible-devops

Ansible collection supporting devops for IBM Maximo Application Suite
https://ibm-mas.github.io/ansible-devops/
Eclipse Public License 2.0
49 stars 88 forks source link

Backup-Restore - DB2 recovery from local is not working. #1480

Closed guillemmayolramis1 closed 3 weeks ago

guillemmayolramis1 commented 1 month ago

Collection version

ibmmas/cli:10.9.2

Environment information

# Local working dir
cd /Users/gcmr/techzone

# download container
podman run -dit -v  /Users/gcmr/techzone:/mnt/home:z --pull always --name mascli quay.io/ibmmas/cli:latest bash

# execute
podman exec -it mascli bash

What happened?


# oc login cluster SRC

# Full backup all manage data for the MAS_INSTANCE_ID instance and MAS_WORKSPACE_ID workspace
export MASBR_ACTION=backup
export MASBR_STORAGE_TYPE=local
export MASBR_STORAGE_LOCAL_FOLDER=/mnt/home/backup
export MAS_INSTANCE_ID=masbrtest
export MAS_WORKSPACE_ID=prod
export DB2_INSTANCE_NAME=mas-masbrtest-prod-manage
ansible-playbook ibm.mas_devops.br_manage

# successful backup
[ibmmas/cli:10.9.2]backup-restore$ ls -lrth /mnt/home/backup/backups/
...
drwxr-xr-x. 6 root root 192 Sep 19 09:44 mongodb-masbrtest-full-20240919094347
drwxr-xr-x. 5 root root 160 Sep 19 09:58 core-masbrtest-full-20240919094347
drwxr-xr-x. 6 root root 192 Sep 19 10:02 db2-mas-masbrtest-prod-manage-full-20240919094347
drwxr-xr-x. 6 root root 192 Sep 19 10:02 manage-masbrtest-full-20240919094347
[ibmmas/cli:10.9.2]backup-restore$ 

# oc login cluster DST
# fresh mas install with same MAS_INSTANCE_ID and MAS_WORKSPACE_ID

# manage recovery 
export MASBR_ACTION=restore
export MASBR_STORAGE_TYPE=local
export MASBR_STORAGE_LOCAL_FOLDER=/mnt/home/backup
export MASBR_RESTORE_FROM_VERSION=20240919094347
export MAS_INSTANCE_ID=masbrtest
export MAS_WORKSPACE_ID=prod
export DB2_INSTANCE_NAME=mas-masbrtest-prod-manage
ansible-playbook ibm.mas_devops.br_manage

# output folder
[ibmmas/cli:10.9.2]backup-restore$ ls -lrth /mnt/home/backup/restores/ | grep 20240919094347
drwxr-xr-x. 4 root root 128 Sep 19 10:20 mongodb-masbrtest-full-20240919094347-20240919101945
drwxr-xr-x. 3 root root  96 Sep 19 10:21 db2-mas-masbrtest-prod-manage-full-20240919094347-20240919101945-Failed
drwxr-xr-x. 3 root root  96 Sep 19 10:21 db2-mas-masbrtest-prod-manage-full-20240919094347-20240919101945
drwxr-xr-x. 4 root root 128 Sep 19 10:22 core-masbrtest-full-20240919094347-20240919101945
drwxr-xr-x. 4 root root 128 Sep 19 10:22 manage-masbrtest-full-20240919094347-20240919101945
[ibmmas/cli:10.9.2]backup-restore$

So, mongo DB and Core get fully recovered (Users, api keys,core configs, etc...) but DB2 is not. The relevant logs have been attached.

Relevant log output

TASK [ibm.mas_devops.db2 : Debug: db2 database restore folder] *****************************************************************************************
ok: [localhost] => 
  msg: Db2 database restore folder ....... /mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240919101945/database

TASK [ibm.mas_devops.db2 : Copy backup file from specified storage location to pod] ********************************************************************
included: /opt/app-root/lib64/python3.9/site-packages/ansible_collections/ibm/mas_devops/roles/db2/tasks/restore/copy-db2-backup-file.yml for localhost

TASK [ibm.mas_devops.db2 : Copy backup file from specified storage location] ***************************************************************************
included: /opt/app-root/lib64/python3.9/site-packages/ansible_collections/ibm/mas_devops/common_tasks/backup_restore/copy_storage_files_to_pod.yml for localhost

TASK [ibm.mas_devops.db2 : Set fact: local storage job folder] *****************************************************************************************
ok: [localhost] => changed=false 
  ansible_facts:
    masbr_storage_job_folder: /mnt/home/backup/backups/db2-mas-masbrtest-prod-manage-full-20240919094347

TASK [ibm.mas_devops.db2 : Debug: local storage job folder] ********************************************************************************************
ok: [localhost] => 
  msg: Local storage job folder .......... /mnt/home/backup/backups/db2-mas-masbrtest-prod-manage-full-20240919094347

TASK [ibm.mas_devops.db2 : Copy files from local storage folder to pod folder] *************************************************************************
skipping: [localhost] => (item={'src_file': 'database/db2-mas-masbrtest-prod-manage-full-20240919094347.tar.gz', 'dest_folder': '/mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240919101945/database'})  => changed=false 
  ansible_loop_var: item
  false_condition: item.src_folder is defined and item.src_folder | length > 0
  item:
    dest_folder: /mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240919101945/database
    src_file: database/db2-mas-masbrtest-prod-manage-full-20240919094347.tar.gz
  skip_reason: Conditional result was False
skipping: [localhost] => changed=false 
  msg: All items skipped

TASK [ibm.mas_devops.db2 : Copy file from local storage folder to pod folder] **************************************************************************
failed: [localhost] (item={'src_file': 'database/db2-mas-masbrtest-prod-manage-full-20240919094347.tar.gz', 'dest_folder': '/mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240919101945/database'}) => changed=true 
  ansible_loop_var: item
  cmd: oc exec c-mas-masbrtest-prod-manage-db2u-0 -c db2u -n db2u -- bash -c 'mkdir -p /mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240919101945/database && oc cp --retries=50 -c db2u /mnt/home/backup/backups/db2-mas-masbrtest-prod-manage-full-20240919094347/database/db2-mas-masbrtest-prod-manage-full-20240919094347.tar.gz db2u/c-mas-masbrtest-prod-manage-db2u-0:/mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240919101945/database'
  delta: '0:00:01.265575'
  end: '2024-09-19 10:21:30.138181'
  item:
    dest_folder: /mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240919101945/database
    src_file: database/db2-mas-masbrtest-prod-manage-full-20240919094347.tar.gz
  msg: non-zero return code
  rc: 127
  start: '2024-09-19 10:21:28.872606'
  stderr: |-
    bash: oc: command not found
    command terminated with exit code 127
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

TASK [ibm.mas_devops.db2 : Update database restore status: Failed] *************************************************************************************
included: /opt/app-root/lib64/python3.9/site-packages/ansible_collections/ibm/mas_devops/common_tasks/backup_restore/update_job_status.yml for localhost
guillemmayolramis1 commented 1 month ago

I think I found the issue. It's attempting to run "oc cp" inside the db2 container which its wrong. The issue happens due to wrongly placed quote character (').

Error is at https://github.com/ibm-mas/ansible-devops/blob/12e101fd94527089f7cced73f2b638eb145ee86c/ibm/mas_devops/common_tasks/backup_restore/copy_storage_files_to_pod.yml#L40

Now shell: >- oc exec {{ masbr_cf_pod_name }} -c {{ masbr_cf_container_name }} -n {{ masbr_cf_namespace }} -- bash -c 'mkdir -p {{ item.dest_folder }} && oc cp --retries=50 -c {{ masbr_cf_container_name }} {{ [masbr_storage_job_folder, item.src_file] | path_join }} {{ masbr_cf_namespace }}/{{ masbr_cf_pod_name }}:{{ item.dest_folder }}' After shell: >- oc exec {{ masbr_cf_pod_name }} -c {{ masbr_cf_container_name }} -n {{ masbr_cf_namespace }} -- bash -c 'mkdir -p {{ item.dest_folder }}' && oc cp --retries=50 -c {{ masbr_cf_container_name }} {{ [masbr_storage_job_folder, item.src_file] | path_join }} {{ masbr_cf_namespace }}/{{ masbr_cf_pod_name }}:{{ item.dest_folder }} I have manually changed this locally and now playbook completes successfully. Note that the issue appears to be present in the other Conditions of the same yaml. This only fixes condition 2. Please review.

Task result after the change TASK [ibm.mas_devops.db2 : Copy file from local storage folder to pod folder] ************************************************* changed: [localhost] => (item={'src_file': 'database/db2-mas-masbrtest-prod-manage-full-20240919094347.tar.gz', 'dest_folder': '/mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240923083656/database'}) => changed=true ansible_loop_var: item cmd: oc exec c-mas-masbrtest-prod-manage-db2u-0 -c db2u -n db2u -- bash -c 'mkdir -p /mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240923083656/database' && oc cp --retries=50 -c db2u /mnt/home/backup/backups/db2-mas-masbrtest-prod-manage-full-20240919094347/database/db2-mas-masbrtest-prod-manage-full-20240919094347.tar.gz db2u/c-mas-masbrtest-prod-manage-db2u-0:/mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240923083656/database delta: '0:03:14.607793' end: '2024-09-23 08:40:41.874926' item: dest_folder: /mnt/backup/db2-mas-masbrtest-prod-manage-full-20240919094347-20240923083656/database src_file: database/db2-mas-masbrtest-prod-manage-full-20240919094347.tar.gz msg: '' rc: 0 start: '2024-09-23 08:37:27.267133' stderr: '' stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> Playbook result after the change ` TASK [ibm.mas_devops.db2 : Summary] *** ok: [localhost] => msg: