search

oVirt / ovirt-engine

The oVirt Engine virtualization manager
Other
513 stars 268 forks source link

incremental backup failed: Bitmap does not exist #896

Open meilihao opened 10 months ago

meilihao commented 10 months ago

Affected version

oVirt Engine: ovirt-engine-4.5.4-1.el8.noarch Host OS Variant: ovirt-node-ng-installer-4.5.4-2022120615.el8.iso VDSM version: vdsm-4.50.3.4-1.el8.x86_64

Describe the bug

op log:

# --- backup_vm.py is from python-ovirt-engine-sdk4-4.6.1/examples
# ./backup_vm.py -c engine1 --debug full --backup-dir bak_dir 95c839b9-04c6-4f9e-b113-83e0aefcddb1
[   0.0 ] Starting full backup for VM '95c839b9-04c6-4f9e-b113-83e0aefcddb1'
[   1.3 ] Waiting until backup '05d48f54-abed-4a7b-9273-4bc58902f671' is ready
[  15.6 ] Created checkpoint '79af917a-b113-45e7-bc10-ead951260373'
[  15.6 ] Downloading full backup for disk '198668cf-7f01-467a-badc-85396034c1de'
[  15.6 ] Creating backup file 'bak_dir/20231129174020.79af917a-b113-45e7-bc10-ead951260373.198668cf-7f01-467a-badc-85396034c1de.full.qcow2'
[  16.9 ] Image transfer 'd692b395-8110-4230-a6f3-d03966e558ec' is ready
[ 100% ] 10.00 GiB, 58.70 s, 174.46 MiB/s                                      
[  75.6 ] Finalizing image transfer
[  82.8 ] Download completed successfully
[  82.8 ] Finalizing backup
[ 159.2 ] Full backup '05d48f54-abed-4a7b-9273-4bc58902f671' completed successfully
# ./backup_vm.py -c engine1 --debug start --from-checkpoint-uuid 79af917a-b113-45e7-bc10-ead951260373 95c839b9-04c6-4f9e-b113-83e0aefcddb1
[   0.0 ] Starting incremental backup since checkpoint '79af917a-b113-45e7-bc10-ead951260373' for VM '95c839b9-04c6-4f9e-b113-83e0aefcddb1'
[   1.1 ] Waiting until backup '304dbd1b-1074-4d3d-8c9c-d09b1e117cdf' is ready
[  14.3 ] Created checkpoint 'f8453825-e0aa-484b-90c9-c759be8b222b'
[  14.4 ] Backup '304dbd1b-1074-4d3d-8c9c-d09b1e117cdf' is ready
05:43:32 chen@fyhdesktop29 examples ±|main ✗|→ ./backup_vm.py -c engine1 --debug download --backup-uuid 304dbd1b-1074-4d3d-8c9c-d09b1e117cdf --incremental 95c839b9-04c6-4f9e-b113-83e0aefcddb1
[   0.0 ] Downloading VM '95c839b9-04c6-4f9e-b113-83e0aefcddb1' disks
[   0.6 ] Downloading incremental backup for disk '198668cf-7f01-467a-badc-85396034c1de'
[   0.6 ] Creating backup file './20231129174405.f8453825-e0aa-484b-90c9-c759be8b222b.198668cf-7f01-467a-badc-85396034c1de.incremental.qcow2'
Traceback (most recent call last):
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/./backup_vm.py", line 542, in <module>
    main()
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/./backup_vm.py", line 145, in main
    args.command(args)
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/./backup_vm.py", line 227, in cmd_download
    download_backup(connection, backup, args, incremental=args.incremental)
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/./backup_vm.py", line 406, in download_backup
    download_disk(
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/./backup_vm.py", line 427, in download_disk
    transfer = imagetransfer.create_transfer(
  File "/opt/mark/test/ovirt/python-ovirt-engine-sdk4-4.6.1/examples/helpers/imagetransfer.py", line 200, in create_transfer
    transfer = transfers_service.add(transfer)
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/services.py", line 14109, in add
    return self._internal_add(image_transfer, headers, query, wait)
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/service.py", line 232, in _internal_add
    return future.wait() if wait else future
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/service.py", line 55, in wait
    return self._code(response)
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/service.py", line 229, in callback
    self._check_fault(response)
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/service.py", line 132, in _check_fault
    self._raise_error(response, body)
  File "/home/chen/.local/lib/python3.10/site-packages/ovirtsdk4/service.py", line 118, in _raise_error
    raise error
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400.

engine log:

2023-11-29 17:24:40,116+08 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48) [8dab67a2-29c3-455b-a63b-5a75fdc8199d] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_SUCCESS(68), Snapshot 'Auto-generated for Backup VM' creation for VM 'test' has been completed.
2023-11-29 17:24:40,150+08 INFO  [org.ovirt.engine.core.bll.storage.backup.HybridBackupCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-48) [8dab67a2-29c3-455b-a63b-5a75fdc8199d] Change VM '95c839b9-04c6-4f9e-b113-83e0aefcddb1' backup '304dbd1b-1074-4d3d-8c9c-d09b1e117cdf' phase from 'STARTING' to 'READY'
2023-11-29 17:24:40,511+08 INFO  [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-146) [63e5ef3b] User admin@ovirt@internalkeycloak-authz with profile [internalsso] successfully logged out
2023-11-29 17:24:40,530+08 INFO  [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-154) [307fc464] Running command: TerminateSessionsForTokenCommand internal: true.
2023-11-29 17:24:40,530+08 INFO  [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (default task-154) [307fc464] Not removing session 'e7AHKd/znvdsye0WvjEPKXix5z/qwkBVDaXo4XHsxno7x22lB3HVOYrWgCRAHa08k+LIeUNu1g1KoPpYBP/YCA==', session has running commands for user 'admin@ovirt@internalkeycloak-authz'.
2023-11-29 17:25:04,158+08 INFO  [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-42) [] Not removing session 'e7AHKd/znvdsye0WvjEPKXix5z/qwkBVDaXo4XHsxno7x22lB3HVOYrWgCRAHa08k+LIeUNu1g1KoPpYBP/YCA==', session has running commands for user 'admin@ovirt@internalkeycloak-authz'.
2023-11-29 17:25:07,370+08 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-49) [] Setting new tasks map. The map contains now 1 tasks
2023-11-29 17:25:12,431+08 INFO  [org.ovirt.engine.core.sso.service.ExternalOIDCService] (default task-145) [] User admin@ovirt@internalkeycloak-authz with profile [internalsso] successfully logged into external OP with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access
2023-11-29 17:25:12,518+08 INFO  [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-145) [56312054] Running command: CreateUserSessionCommand internal: false.
2023-11-29 17:25:12,567+08 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-145) [56312054] EVENT_ID: USER_VDC_LOGIN(30), User admin@ovirt@internalkeycloak-authz connecting from '192.168.88.236' using session 'jXndZzjdDxXOJNPeQbUBDrbcn1IID2sOs5baDTAWJcFX3tNyw03UFu+Ekc/SXDo85djAGO2yhjfjyW9biVRenA==' logged in.
2023-11-29 17:25:12,788+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-11-29 17:25:12,811+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Running command: TransferDiskImageCommand internal: false. Entities affected :  ID: 198668cf-7f01-467a-badc-85396034c1de Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER
2023-11-29 17:25:12,811+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Creating ImageTransfer entity for command 'b06937e2-da39-4685-86e5-634b0cdbdb3d', proxyEnabled: true
2023-11-29 17:25:12,814+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Starting image transfer: ImageTransfer:{id='b06937e2-da39-4685-86e5-634b0cdbdb3d', phase='Initializing', type='Download', active='false', lastUpdated='Wed Nov 29 17:25:12 CST 2023', message='null', vdsId='null', diskId='null', imagedTicketId='null', proxyUri='null', bytesSent='null', bytesTotal='10737418240', clientInactivityTimeout='60', timeoutPolicy='cancel', imageFormat='RAW', transferClientType='Transfer via API', shallow='false'}
2023-11-29 17:25:12,816+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Successfully added Download disk 'CentOS-8-GenericCloud-8.1.1911' (disk id: '198668cf-7f01-467a-badc-85396034c1de', image id: 'c0586bf2-e409-4e4e-98e1-f62ef967da26') for image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d'
2023-11-29 17:25:12,833+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] START, PrepareImageVDSCommand(HostName = node151, PrepareImageVDSCommandParameters:{hostId='609d05ed-1fad-4a32-95e3-2a6e6735a23d'}), log id: 6d2edf4f
2023-11-29 17:25:12,980+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] FINISH, PrepareImageVDSCommand, return: PrepareImageReturn:{status='Status [code=0, message=Done]'}, log id: 6d2edf4f
2023-11-29 17:25:12,996+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] START, StartNbdServerVDSCommand(HostName = node151, NbdServerVDSParameters:{hostId='609d05ed-1fad-4a32-95e3-2a6e6735a23d', serverId='b06937e2-da39-4685-86e5-634b0cdbdb3d', storageDomainId='3eab4802-18dd-4a4d-a140-ac35edca735a', imageId='198668cf-7f01-467a-badc-85396034c1de', volumeId='c0586bf2-e409-4e4e-98e1-f62ef967da26', readonly='true', discard='true', detectZeroes='true', backingChain='true', bitmap='79af917a-b113-45e7-bc10-ead951260373'}), log id: 7a95154c
2023-11-29 17:25:13,028+08 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Unexpected return value: Status [code=947, message=Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}"]
2023-11-29 17:25:13,028+08 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Failed in 'StartNbdServerVDS' method
2023-11-29 17:25:13,028+08 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Unexpected return value: Status [code=947, message=Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}"]
2023-11-29 17:25:13,032+08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM node151 command StartNbdServerVDS failed: Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}"
2023-11-29 17:25:13,032+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand' return value 'NbdServerURLReturn:{status='Status [code=947, message=Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}"]'}'
2023-11-29 17:25:13,032+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] HostName = node151
2023-11-29 17:25:13,033+08 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Command 'StartNbdServerVDSCommand(HostName = node151, NbdServerVDSParameters:{hostId='609d05ed-1fad-4a32-95e3-2a6e6735a23d', serverId='b06937e2-da39-4685-86e5-634b0cdbdb3d', storageDomainId='3eab4802-18dd-4a4d-a140-ac35edca735a', imageId='198668cf-7f01-467a-badc-85396034c1de', volumeId='c0586bf2-e409-4e4e-98e1-f62ef967da26', readonly='true', discard='true', detectZeroes='true', backingChain='true', bitmap='79af917a-b113-45e7-bc10-ead951260373'})' execution failed: VDSGenericException: VDSErrorException: Failed to StartNbdServerVDS, error = Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}", code = 947
2023-11-29 17:25:13,033+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] FINISH, StartNbdServerVDSCommand, return: , log id: 7a95154c
2023-11-29 17:25:13,033+08 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Failed to start NBD server for image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d': {}: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to StartNbdServerVDS, error = Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}", code = 947 (Failed with error unexpected and code 16)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2121)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.startImageTransferSession(TransferDiskImageCommand.java:1084)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.handleImageIsReadyForTransfer(TransferDiskImageCommand.java:681)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.executeCommand(TransferDiskImageCommand.java:518)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1174)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1332)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2010)
    at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:140)
    at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:79)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:424)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:387)
    at jdk.internal.reflect.GeneratedMethodAccessor860.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
    at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126)
    at org.ovirt.engine.core.common//org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12)
    at jdk.internal.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105)
    at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82)
    at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60)
    at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77)
    at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89)
    at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102)
    at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13)
    at jdk.internal.reflect.GeneratedMethodAccessor557.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
    at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:232)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:446)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:164)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
    at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
    at org.wildfly.security.elytron-base@1.16.1.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:633)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
    at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
    at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
    at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:191)
    at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
    at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:251)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:279)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:171)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:160)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:187)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:180)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.performCreate(BackendImageTransfersResource.java:82)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.addForDisk(BackendImageTransfersResource.java:49)
    at jdk.internal.reflect.GeneratedMethodAccessor2260.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.ovirt.engine.api.restapi-definition//org.ovirt.engine.api.resource.ImageTransfersResource.add(ImageTransfersResource.java:91)
    at jdk.internal.reflect.GeneratedMethodAccessor2254.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:138)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:546)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:435)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:396)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:398)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:150)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:104)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:440)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:245)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:61)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.1.Final//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
    at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
    at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
    at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:251)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:186)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:227)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:167)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:87)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:106)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:85)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:46)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at org.ovirt.engine.core.aaa//org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:100)
    at org.ovirt.engine.api.restapi-jaxrs//org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:91)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:272)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
    at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
    at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
    at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)
    at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
    at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
    at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
    at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
    at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
    at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
    at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
    at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
    at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535)
    at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535)
    at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
    at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
    at io.undertow.core@2.2.8.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
    at io.undertow.core@2.2.8.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841)
    at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
    at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
    at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
    at org.jboss.xnio@3.8.4.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to StartNbdServerVDS, error = Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/_images/3eab4802-18dd-4a4d-a140-ac35edca735a/images/198668cf-7f01-467a-badc-85396034c1de/c0586bf2-e409-4e4e-98e1-f62ef967da26', 'bitmap': '79af917a-b113-45e7-bc10-ead951260373'}", code = 947
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81)
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:224)
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:194)
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand.executeVdsBrokerCommand(StartNbdServerVDSCommand.java:21)
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123)
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111)
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
    at org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14)
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:410)
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source)
    at jdk.internal.reflect.GeneratedMethodAccessor625.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78)
    at org.ovirt.engine.core.common//org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12)
    at jdk.internal.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
    at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
    at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source)
    ... 216 more

2023-11-29 17:25:13,039+08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] EVENT_ID: TRANSFER_IMAGE_STOPPED_BY_SYSTEM_FAILED_TO_CREATE_TICKET(1,068), Transfer was stopped by system. Reason: failed to create a signed image ticket.
2023-11-29 17:25:13,040+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Updating image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' phase from 'Initializing' to 'Cancelled'
2023-11-29 17:25:13,098+08 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] EVENT_ID: TRANSFER_IMAGE_INITIATED(1,031), Image Download with disk CentOS-8-GenericCloud-8.1.1911 was initiated by admin@ovirt@internalkeycloak-authz.
2023-11-29 17:25:13,100+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-145) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-11-29 17:25:13,101+08 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-145) [] Operation Failed: []
2023-11-29 17:25:13,158+08 INFO  [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-146) [63e5ef3b] User admin@ovirt@internalkeycloak-authz with profile [internalsso] successfully logged out
2023-11-29 17:25:13,171+08 INFO  [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-154) [22ab80fd] Running command: TerminateSessionsForTokenCommand internal: true.
2023-11-29 17:25:13,291+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-61) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' cancelled by system for Download disk 'CentOS-8-GenericCloud-8.1.1911' (disk id: '198668cf-7f01-467a-badc-85396034c1de', image id: 'c0586bf2-e409-4e4e-98e1-f62ef967da26')
2023-11-29 17:25:13,292+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-61) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Updating image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' phase from 'Cancelled' to 'Finalizing Failure'
2023-11-29 17:25:15,336+08 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Finalizing failed image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' for Download disk 'CentOS-8-GenericCloud-8.1.1911' (disk id: '198668cf-7f01-467a-badc-85396034c1de', image id: 'c0586bf2-e409-4e4e-98e1-f62ef967da26')
2023-11-29 17:25:15,336+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.RemoveImageTicketVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] START, RemoveImageTicketVDSCommand(HostName = node151, RemoveImageTicketVDSCommandParameters:{hostId='609d05ed-1fad-4a32-95e3-2a6e6735a23d', ticketId='317f9abf-212d-4714-923a-e6d8f1ab8854', timeout='null'}), log id: 600ce72f
2023-11-29 17:25:15,342+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.RemoveImageTicketVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] FINISH, RemoveImageTicketVDSCommand, return: StatusOnlyReturn [status=Status [code=0, message=Done]], log id: 600ce72f
2023-11-29 17:25:15,342+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Successfully stopped image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' for ticket '317f9abf-212d-4714-923a-e6d8f1ab8854'
2023-11-29 17:25:15,342+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Removing proxy image ticket '317f9abf-212d-4714-923a-e6d8f1ab8854' for image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d'
2023-11-29 17:25:15,344+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StopNbdServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] START, StopNbdServerVDSCommand(HostName = node151, NbdServerVDSParameters:{hostId='609d05ed-1fad-4a32-95e3-2a6e6735a23d', serverId='b06937e2-da39-4685-86e5-634b0cdbdb3d', storageDomainId='null', imageId='null', volumeId='null', readonly='false', discard='false', detectZeroes='false', backingChain='true', bitmap='null'}), log id: 509d00cf
2023-11-29 17:25:15,371+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.StopNbdServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] FINISH, StopNbdServerVDSCommand, return: , log id: 509d00cf
2023-11-29 17:25:15,376+08 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' failed for Download disk 'CentOS-8-GenericCloud-8.1.1911' (disk id: '198668cf-7f01-467a-badc-85396034c1de', image id: 'c0586bf2-e409-4e4e-98e1-f62ef967da26')
2023-11-29 17:25:16,379+08 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand' with failure.
2023-11-29 17:25:16,379+08 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Failed to transfer disk '00000000-0000-0000-0000-000000000000' for image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d'
2023-11-29 17:25:16,379+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-11-29 17:25:16,381+08 INFO  [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] Updating image transfer 'b06937e2-da39-4685-86e5-634b0cdbdb3d' phase from 'Finalizing Failure' to 'Finished Failure'
2023-11-29 17:25:16,414+08 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [50ebde6e-cff4-452b-9ecc-7693af9b0ccf] EVENT_ID: TRANSFER_IMAGE_CANCELLED(1,033), Image Download with disk CentOS-8-GenericCloud-8.1.1911 was cancelled.
^C

full log: full.log

To reproduce

see op log

Expected behavior

incremental backup is ok

meilihao commented 10 months ago

my disk is qcow2 compat 0.10, upgrade compat to 1.1, incremental backup works.

meilihao commented 10 months ago

The document only mentions that qcow2 is required. Does incremental backup require qcow2 v3?