search

DIMSI-IS / BackROLL

BackROLL is a KVM guest backup software
https://cloud.dimsi.io
Apache License 2.0
34 stars 1 forks source link

Error when creating backup of a VM #24

Closed virsibl closed 1 month ago

virsibl commented 3 months ago

Hello, an error occurs when creating a backup copy of a virtual machine.

Version cloudstack packages:

ii cloudstack-common 4.19.0.1-shapeblue0 all A common package which contains files which are shared by several CloudStack packages ii cloudstack-management 4.19.0.1-shapeblue0 all CloudStack server library

Logs(docker container):

core           | INFO:     10.10.10.2:52938 - "GET /api/v1/tasks/restore HTTP/1.1" 200 OK

core           | INFO:     10.10.10.2:52954 - "GET /api/v1/tasks/backup HTTP/1.1" 200 OK

core           | INFO:     10.10.10.2:42296 - "GET /api/v1/tasks/restore HTTP/1.1" 200 OK
core           | INFO:     10.10.10.2:42300 - "GET /api/v1/tasks/backup HTTP/1.1" 200 OK
core           | INFO:     10.10.10.2:40772 - "GET /api/v1/tasks/restore HTTP/1.1" 200 OK
core           | INFO:     10.10.10.2:40780 - "GET /api/v1/tasks/backup HTTP/1.1" 200 OK
core           | INFO:     10.10.10.2:40780 - "POST /api/v1/tasks/singlebackup/43deaea7-10db-4df4-891b-52cb638ed853 HTTP/1.1" 202 Accepted
core           | INFO:     10.10.10.2:40772 - "GET /api/v1/tasks/restore HTTP/1.1" 200 OK
core           | INFO:     10.10.10.2:40780 - "GET /api/v1/tasks/backup HTTP/1.1" 200 OK
worker         | /usr/local/lib/python3.8/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
worker         |   "class": algorithms.Blowfish,
worker         | /usr/local/lib/python3.8/dist-packages/celery/platforms.py:840: SecurityWarning: You're running the worker with superuser privileges: this is
worker         | absolutely not recommended!
worker         | 
worker         | Please specify a different user using the --uid option.
worker         | 
worker         | User information: uid=0 euid=0 gid=0 egid=0
worker         | 
worker         |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
worker         | [2024-06-23 13:43:59,686: WARNING/ForkPoolWorker-17] 43deaea7-10db-4df4-891b-52cb638ed853
worker         | [2024-06-23 13:43:59,686: WARNING/ForkPoolWorker-17] 80c1d07c-a890-44a6-9dff-1b75b33f5515
worker         | [2024-06-23 13:43:59,686: WARNING/ForkPoolWorker-17] 616efeb5-9675-4082-8fc0-b14c0e7b431d
worker         | [2024-06-23 13:43:59,686: WARNING/ForkPoolWorker-17] 43deaea7-10db-4df4-891b-52cb638ed853
worker-lrt     | /usr/local/lib/python3.8/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
worker-lrt     |   "class": algorithms.Blowfish,
worker-lrt     | /usr/local/lib/python3.8/dist-packages/celery/platforms.py:840: SecurityWarning: You're running the worker with superuser privileges: this is
worker-lrt     | absolutely not recommended!
worker-lrt     | 
worker-lrt     | Please specify a different user using the --uid option.
worker-lrt     | 
worker-lrt     | User information: uid=0 euid=0 gid=0 egid=0
worker-lrt     | 
worker-lrt     |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
worker-lrt     | [2024-06-23 13:44:00,143: WARNING/ForkPoolWorker-1] {'id': 39, 'uuid': '43deaea7-10db-4df4-891b-52cb638ed853', 'name': 'i-2-32-VM', 'mem': 1048576, 'cpus': 1, 'host': 'acc91e74-0a23-460b-87a1-2effc730a8f1', 'host_tag': 'pbx', 'cloudstack_instance': True, 'state': 'Running'}
worker-lrt     | [2024-06-23 13:44:00,160: WARNING/ForkPoolWorker-1] [i-2-32-VM] Gathering data...
worker-lrt     | [2024-06-23 13:44:00,160: WARNING/ForkPoolWorker-1] [i-2-32-VM] Disk(s) found : 1
worker-lrt     | [2024-06-23 13:44:00,161: WARNING/ForkPoolWorker-1] [i-2-32-VM] Pre-Flight checks incoming.
worker-lrt     | [2024-06-23 13:44:00,346: WARNING/ForkPoolWorker-1] [i-2-32-VM] A snapshot has been detected !
worker-lrt     | [2024-06-23 13:44:00,346: WARNING/ForkPoolWorker-1] [i-2-32-VM] VM is currently under snapshot. Checking disk files...
worker-lrt     | [2024-06-23 13:44:00,531: WARNING/ForkPoolWorker-1] [i-2-32-VM] Snapshot i-2-32-VM.snap has been successfully deleted
worker-lrt     | [2024-06-23 13:44:00,532: WARNING/ForkPoolWorker-1] [i-2-32-VM] Snapshot deleted.
worker-lrt     | [2024-06-23 13:44:00,532: WARNING/ForkPoolWorker-1] [i-2-32-VM] Virtual Machine is now in clean condition.
worker-lrt     | [2024-06-23 13:44:00,532: WARNING/ForkPoolWorker-1] [i-2-32-VM] Pre-Flight checks done...
worker-lrt     | [2024-06-23 13:44:00,532: WARNING/ForkPoolWorker-1] [i-2-32-VM] Snapshotting virtual machines disks
worker-lrt     | [2024-06-23 13:44:00,787: WARNING/ForkPoolWorker-1] [i-2-32-VM] A Borg repository for this VM has been found
worker-lrt     | [2024-06-23 13:44:00,788: WARNING/ForkPoolWorker-1] [i-2-32-VM] Borg repository setup is OK
worker-lrt     | [2024-06-23 13:44:00,788: WARNING/ForkPoolWorker-1] [i-2-32-VM] Checking borg repository lock status
worker-lrt     | [2024-06-23 13:44:01,393: WARNING/ForkPoolWorker-1] [i-2-32-VM] Borg repository is unlocked, job will continue
worker-lrt     | [2024-06-23 13:44:01,810: WARNING/ForkPoolWorker-1] [i-2-32-VM] Disk vda has been successfully blockcommited
worker-lrt     | [2024-06-23 13:44:01,846: ERROR/ForkPoolWorker-1] Task Single_VM_Backup[e2c3d8ba-3a98-4839-93c4-1272cc102b1c] raised unexpected: JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
worker-lrt     | Traceback (most recent call last):
worker-lrt     |   File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 451, in trace_task
worker-lrt     |     R = retval = fun(*args, **kwargs)
worker-lrt     |   File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 734, in __protected_call__
worker-lrt     |     return self.run(*args, **kwargs)
worker-lrt     |   File "/usr/src/app/app/backup_tasks/single_backup.py", line 161, in single_vm_backup
worker-lrt     |     raise backup_error
worker-lrt     |   File "/usr/src/app/app/backup_tasks/single_backup.py", line 154, in single_vm_backup
worker-lrt     |     raise startbackup_error
worker-lrt     |   File "/usr/src/app/app/backup_tasks/single_backup.py", line 150, in single_vm_backup
worker-lrt     |     backup_result = backup_creation(virtual_machine_info)
worker-lrt     |   File "/usr/src/app/app/backup_tasks/single_backup.py", line 137, in backup_creation
worker-lrt     |     raise sequence_error
worker-lrt     |   File "/usr/src/app/app/backup_tasks/single_backup.py", line 133, in backup_creation
worker-lrt     |     return backup_sequence(info, host_info)
worker-lrt     |   File "/usr/src/app/app/backup_tasks/single_backup.py", line 126, in backup_sequence
worker-lrt     |     raise backup_error
worker-lrt     |   File "/usr/src/app/app/backup_tasks/single_backup.py", line 93, in backup_sequence
worker-lrt     |     backup_job.manage_backing_file(disk)
worker-lrt     |   File "/usr/src/app/app/borg/borg_core.py", line 155, in manage_backing_file
worker-lrt     |     qemu_img_info = json.loads(qemu_img_info)
worker-lrt     |   File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
worker-lrt     |     return _default_decoder.decode(s)
worker-lrt     |   File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
worker-lrt     |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
worker-lrt     |   File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
worker-lrt     |     raise JSONDecodeError("Expecting value", s, err.value) from None
worker-lrt     | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
worker         | /usr/local/lib/python3.8/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
worker         |   "class": algorithms.Blowfish,
worker         | /usr/local/lib/python3.8/dist-packages/celery/platforms.py:840: SecurityWarning: You're running the worker with superuser privileges: this is
worker         | absolutely not recommended!
worker         | 
worker         | Please specify a different user using the --uid option.
worker         | 
worker         | User information: uid=0 euid=0 gid=0 egid=0
worker         | 
worker         |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
worker         | [2024-06-23 13:44:04,810: WARNING/ForkPoolWorker-16] DEBUG TASK Failure: text: ({'id': 39, 'uuid': '43deaea7-10db-4df4-891b-52cb638ed853', 'name': 'i-2-32-VM', 'mem': 1048576, 'cpus': 1, 'host': 'acc91e74-0a23-460b-87a1-2effc730a8f1', 'host_tag': 'pbx', 'cloudstack_instance': True, 'state': 'Running'},)
worker         | [2024-06-23 13:44:04,894: ERROR/ForkPoolWorker-16] Task Handle task failure[e2e5a3d6-8bb4-4b69-b1a2-7db3403218ff] raised unexpected: ValueError(NoResultFound('No row was found when one was required'))
worker         | Traceback (most recent call last):
worker         |   File "/usr/src/app/app/routes/external_hooks.py", line 59, in filter_external_hook_by_id
worker         |     pool = results.one()
worker         |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/result.py", line 1408, in one
worker         |     return self._only_one_row(
worker         |   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/result.py", line 562, in _only_one_row
worker         |     raise exc.NoResultFound(
worker         | sqlalchemy.exc.NoResultFound: No row was found when one was required
worker         | 
worker         | The above exception was the direct cause of the following exception:
worker         | 
worker         | Traceback (most recent call last):
worker         |   File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 451, in trace_task
worker         |     R = retval = fun(*args, **kwargs)
worker         |   File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 734, in __protected_call__
worker         |     return self.run(*args, **kwargs)
worker         |   File "/usr/src/app/app/task_handler.py", line 199, in handle_task_failure
worker         |     hook = hook_route.filter_external_hook_by_id(policy.externalhook)
worker         |   File "/usr/local/lib/python3.8/dist-packages/celery/local.py", line 188, in __call__
worker         |     return self._get_current_object()(*a, **kw)
worker         |   File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 735, in __protected_call__
worker         |     return orig(self, *args, **kwargs)
worker         |   File "/usr/local/lib/python3.8/dist-packages/celery/app/task.py", line 392, in __call__
worker         |     return self.run(*args, **kwargs)
worker         |   File "/usr/src/app/app/routes/external_hooks.py", line 64, in filter_external_hook_by_id
worker         |     raise ValueError(exc) from exc
worker         | ValueError: No row was found when one was required

Task logs (i-2-32-VM):

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 451, in trace_task R = retval = fun(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 734, in __protected_call__ return self.run(*args, **kwargs) File "/usr/src/app/app/backup_tasks/single_backup.py", line 161, in single_vm_backup raise backup_error File "/usr/src/app/app/backup_tasks/single_backup.py", line 154, in single_vm_backup raise startbackup_error File "/usr/src/app/app/backup_tasks/single_backup.py", line 150, in single_vm_backup backup_result = backup_creation(virtual_machine_info) File "/usr/src/app/app/backup_tasks/single_backup.py", line 137, in backup_creation raise sequence_error File "/usr/src/app/app/backup_tasks/single_backup.py", line 133, in backup_creation return backup_sequence(info, host_info) File "/usr/src/app/app/backup_tasks/single_backup.py", line 126, in backup_sequence raise backup_error File "/usr/src/app/app/backup_tasks/single_backup.py", line 93, in backup_sequence backup_job.manage_backing_file(disk) File "/usr/src/app/app/borg/borg_core.py", line 155, in manage_backing_file qemu_img_info = json.loads(qemu_img_info) File "/usr/lib/python3.8/json/__init__.py", line 357, in loads return _default_decoder.decode(s) File "/usr/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

JoffreyLuang commented 3 months ago

Hi virsibl,

Could you try the following please? As you are using Cloudstack, the VMs storage mount must be mounted under /mnt/PR_STORAGE_ID to follow cloudstack naming.

I completed the documentation, I apologize if it was unclear previously. 

#If you are mounting a Cloudstack Primary storage please respect the Cloudstack format such as: /mnt/PR_STORAGE_ID
# Create directory under /mnt/ corresponding to your CS primary storage
mkdir /mnt/138338fb-xxxx-xxxx-b219-ff968ca3ed3d

# Mount using NFS, to make the mount persistent, edit fstab with corresponding values
mount -v -t nfs -o nolock NFS_server:/nfs_shareCS1 /mnt/138338fb-xxxx-xxxx-b219-ff968ca3ed3d
virsibl commented 3 months ago

Hello JoffreyLuang, Thanks for your support.

The backup copy was created successfully. But I have some questions, sorry off topic:

1) When turning off the VM, for some reason it is not possible to make a backup copy. It is not present in the vm lists.

2) As far as I understand, you store all the information about the state and recovery VM in a file, it is not entirely clear why this is done, given the presence of the mariadb database.

# file database.db
database.db: Berkeley DB (Hash, version 9, native byte-order)

3) If I deleted the VM in cloudstack, then I can no longer restore it from the web interface.

4) When restoring the VM, the time of the copy creation is not visible.

image

image

m-dhellin commented 1 month ago

Hello @virsibl,

I am closing the issue because you have managed to backup your VM.

For points 1 and 3 you can open new issues or start discussions here https://github.com/DIMSI-IS/BackROLL/discussions/categories/q-a as for point 2.

The point 4 has been done in a branch but we still need to review it before merging. Feel free to submit other ideas here https://github.com/DIMSI-IS/BackROLL/discussions/categories/ideas.