Restore VM from Delta Backups stops at exactly 5mins

eddiecheng commented 1 year ago

Xen Orchestra, commit bf51b

Tested on Fully updated XCP-NG Host and patches Fresh installation.

SR restore to Local NVME SR -> Stops at 5mins. Tried on 5 similar setup blade servers.

Tried on SR restore to NFS SR -> works okay

eddiecheng commented 1 year ago

Additional Informations

XOA From sources installed on OS : Debian 11

Latest Commit XCP 8.2.1 with all patches done as at 5th Feb 2023

julien-f commented 1 year ago

It looks like the bug with the default value change for requestTimeout in Node 18.

But it has been fixed already in f6fd1db1ef12633cc5bb8ec8ab5bc84682dd3fe7

Check that your xo-server is up-to-date and has restarted since this fix.

To be sure your configuration is correct run the following command in xo-server's directory:

> npx app-conf xo-server . -p http.listenOptions.requestTimeout
86400000

This message is outdate, please see this one instead.

eddiecheng commented 1 year ago

app-conf /opt/xen-orchestra/packages/xo-server/config.toml +0ms
app-conf /root/.config/xo-server/config.z-auto.json +3ms
app-conf /opt/xen-orchestra/packages/xo-server/.xo-server.toml +3ms
86400000

seems to be correct.

This issue only happens if we restore from backup from delta.

Stops exactly at 5mins.

Full error

```js { "code": "VDI_IO_ERROR", "params": [ "Device I/O errors" ], "url": "https://192.168.0.234/import_raw_vdi/?format=vhd&vdi=OpaqueRef%3A332f7deb-911c-40e2-9afc-dfc682d8bcb5&session_id=OpaqueRef%3A535b10e8-f854-452b-90f5-827e03a864c3&task_id=OpaqueRef%3A3f1af711-85e6-4101-8402-3b2894d8ba94", "task": { "uuid": "ea158f39-375c-f2b5-4e2a-3f1e6aabf73c", "name_label": "[XO] Importing content into VDI xxx.net 0", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20230205T20:14:44Z", "finished": "20230205T20:19:45Z", "status": "failure", "resident_on": "OpaqueRef:2550d0ea-f7e0-49f3-861e-11f22d8e36df", "progress": 1, "type": "", "result": "", "error_info": [ "VDI_IO_ERROR", "Device I/O errors" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/vhd_tool_wrapper.ml)(line 77))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/import_raw_vdi.ml)(line 170)))" }, "pool_master": { "uuid": "a5ed16e1-6221-402f-881e-d93ecbfd6358", "name_label": "blade4.quape.com", "name_description": "Default install", "memory_overhead": 3805732864, "allowed_operations": [ "vm_migrate", "provision", "vm_resume", "evacuate", "vm_start" ], "current_operations": {}, "API_version_major": 2, "API_version_minor": 16, "API_version_vendor": "XenSource", "API_version_vendor_implementation": {}, "enabled": true, "software_version": { "product_version": "8.2.1", "product_version_text": "8.2", "product_version_text_short": "8.2", "platform_name": "XCP", "platform_version": "3.2.1", "product_brand": "XCP-ng", "build_number": "release/yangtze/master/58", "hostname": "localhost", "date": "2022-10-12", "dbv": "0.0.1", "xapi": "1.20", "xen": "4.13.4-9.19.1", "linux": "4.19.0+1", "xencenter_min": "2.16", "xencenter_max": "2.16", "network_backend": "openvswitch", "db_schema": "5.602" }, "other_config": { "agent_start_time": "1675387608.", "boot_time": "1669959397.", "rpm_patch_installation_time": "1675387592.774", "iscsi_iqn": "iqn.2022-09.com.quape:2586113e" }, "capabilities": [ "xen-3.0-x86_64", "hvm-3.0-x86_32", "hvm-3.0-x86_32p", "hvm-3.0-x86_64", "" ], "cpu_configuration": {}, "sched_policy": "credit", "supported_bootloaders": [ "pygrub", "eliloader" ], "resident_VMs": [ "OpaqueRef:8676efa0-6e9d-4e64-82a6-ba27263dc0f3", "OpaqueRef:f9514645-6180-4c73-aeb8-84a97ed4a887", "OpaqueRef:51720f04-cff9-422e-ad64-ca9cd189e63f", "OpaqueRef:19143736-8a11-4da3-9fc0-4310926b65e6", "OpaqueRef:04c12412-3c61-4860-b703-07854d931ed5" ], "logging": {}, "PIFs": [ "OpaqueRef:a3eb0cd4-139e-4dfd-8034-6b9a28768127", "OpaqueRef:682e9357-803e-4d37-92df-94b3108813c3", "OpaqueRef:357edb0d-1058-4dcc-ac50-c52cfeea6437", "OpaqueRef:066e1d7f-97fc-4170-a095-17f155559283" ], "suspend_image_sr": "OpaqueRef:f60b963a-a3ad-4b36-ad60-abda02597930", "crash_dump_sr": "OpaqueRef:f60b963a-a3ad-4b36-ad60-abda02597930", "crashdumps": [], "patches": [], "updates": [], "PBDs": [ "OpaqueRef:ea2d588b-779e-44b8-843a-bd3b024d1eb8", "OpaqueRef:e71f1fbb-0090-4194-b878-92ff65b3d44e", "OpaqueRef:dba88fa5-a694-4aed-b75e-caa2be20a9df", "OpaqueRef:ace7e2ee-dc7e-429b-84c8-848c382d9736", "OpaqueRef:a53f387e-4107-4f81-85d7-79a93097d8bc", "OpaqueRef:793e6779-6c9e-44f8-ad39-3c2fc5c35e45", "OpaqueRef:6605d755-260e-4d38-bcf5-a29538427aea", "OpaqueRef:570825c6-05a9-4d9e-9203-78d02366466e" ], "host_CPUs": [ "OpaqueRef:f7d5bb33-6e42-4814-a913-6ad08abf99be", "OpaqueRef:22607bdb-acf0-4cb5-a3f5-eacdfa7bda13", "OpaqueRef:66c6ec50-df12-405a-99d1-f9710b7a06d9", "OpaqueRef:1d3fc4e8-57d0-469b-88af-386f908a3bd0", "OpaqueRef:dcfe7fc6-5332-4fa2-bdd3-60ceb0f7c9c6", "OpaqueRef:be30b3eb-5958-42f1-bc72-a1bae93a74bf", "OpaqueRef:6b93cf49-667f-48aa-912a-ebe357e7844e", "OpaqueRef:4dee4c51-ddcc-49aa-82fc-485651e7748d", "OpaqueRef:30fceba9-7ee6-4feb-951e-a9914fc530d2", "OpaqueRef:2a134b0e-f31f-404b-84f1-4c7ed8a39244", "OpaqueRef:9e97d163-19bd-465a-adb9-66ea91c6416d", "OpaqueRef:db25ecfc-525f-4aa1-b0f6-98fa91cd70c8", "OpaqueRef:16edef73-9ea9-4d95-a33c-d3cd3087c0ae", "OpaqueRef:a6cf670d-f6a3-42d2-a719-3528a41d9f24", "OpaqueRef:422d2888-a6a5-4541-9ef6-84202ed4a011", "OpaqueRef:e7ea9f28-e7d7-40ad-91b0-98e0d02280dd", "OpaqueRef:d0a65e9e-d7c6-4e7b-b89b-c0fffe470d17", "OpaqueRef:65406754-3ba9-439c-9831-9bb113f726da", "OpaqueRef:5967fa51-ee21-43b8-a399-36b1d4bd9683", "OpaqueRef:f7ab2fd7-46e1-4e45-9ba6-12c608396aa4", "OpaqueRef:1f945ce4-e1fb-434b-a79f-997f46d58493", "OpaqueRef:eeb2296e-148c-4f0e-aa74-7dc4352344c0", "OpaqueRef:9a7e344d-d6e2-4bee-b4e9-b4cae06ccc9d", "OpaqueRef:dc63d539-ee48-46a0-9ed5-1dccbd77d9a6", "OpaqueRef:7e976320-a64c-4192-bf16-d70ab47dfecc", "OpaqueRef:96a86c82-2e0d-43ba-9e13-8fd4ac997789", "OpaqueRef:5df816a0-26bd-4dec-ba86-7ef2b1413ba8", "OpaqueRef:921ff087-1308-4c9b-bd8c-32bd880585ef", "OpaqueRef:648f3bca-60fb-4fb0-a0d1-7869c30cc722", "OpaqueRef:9c69ca39-4c11-464a-b808-82b8920f8f2e", "OpaqueRef:58f5a55e-1fe5-475e-b1a2-b94563dc91f6", "OpaqueRef:c1179278-2632-4736-88bb-1edfc156d89e", "OpaqueRef:a1f88177-d690-4e5d-9147-d43e1c7dd0e4", "OpaqueRef:e709542c-7aaf-486d-9b1c-d6201a4ab415", "OpaqueRef:f0725ec0-41bf-4ce3-aff1-6d57fc8b484b", "OpaqueRef:e990ec0a-2d68-45f2-8ee4-6c31bafdfd5d", "OpaqueRef:2668e02d-257c-41fa-ab69-2a0730daddbd", "OpaqueRef:0f561ab1-dfa1-4508-bae7-ddce77ce47ca", "OpaqueRef:b79be6b8-894e-41d5-bf57-bc15be417845", "OpaqueRef:83afaa9d-4319-4fb1-a1f7-72c28e8ef390" ], "cpu_info": { "cpu_count": "40", "socket_count": "2", "vendor": "GenuineIntel", "speed": "2200.052", "modelname": "Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz", "family": "6", "model": "85", "stepping": "7", "flags": "fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault intel_ppin ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase bmi1 avx2 bmi2 erms rdseed adx clflushopt clwb xsaveopt xsavec xgetbv1 arch_capabilities", "features_pv": "1fc9cbf5-f6f83203-2991cbf5-00000123-00000007-018c0b39-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000", "features_hvm": "1fcbfbff-f7fa3223-2d93fbff-00000523-0000000f-019c4fbb-00000008-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000", "features_hvm_host": "1fcbfbff-f7fa3223-2c100800-00000121-0000000f-019c47ab-00000008-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000", "features_pv_host": "1fc9cbf5-f6f83203-28100800-00000121-00000007-018c0329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000" }, "hostname": "blade4.quape.com", "address": "103.69.130.234", "metrics": "OpaqueRef:55ffea32-b83c-44e6-9a64-ac996d64cdbd", "license_params": { "restrict_vswitch_controller": "false", "restrict_lab": "false", "restrict_stage": "false", "restrict_storagelink": "false", "restrict_storagelink_site_recovery": "false", "restrict_web_selfservice": "false", "restrict_web_selfservice_manager": "false", "restrict_hotfix_apply": "false", "restrict_export_resource_data": "false", "restrict_read_caching": "false", "restrict_cifs": "false", "restrict_health_check": "false", "restrict_xcm": "false", "restrict_vm_memory_introspection": "false", "restrict_batch_hotfix_apply": "false", "restrict_management_on_vlan": "false", "restrict_ws_proxy": "false", "restrict_vlan": "false", "restrict_qos": "false", "restrict_pool_attached_storage": "false", "restrict_netapp": "false", "restrict_equalogic": "false", "restrict_pooling": "false", "enable_xha": "true", "restrict_marathon": "false", "restrict_email_alerting": "false", "restrict_historical_performance": "false", "restrict_wlb": "false", "restrict_rbac": "false", "restrict_dmc": "false", "restrict_checkpoint": "false", "restrict_cpu_masking": "false", "restrict_connection": "false", "platform_filter": "false", "regular_nag_dialog": "false", "restrict_vmpr": "false", "restrict_vmss": "false", "restrict_intellicache": "false", "restrict_gpu": "false", "restrict_dr": "false", "restrict_vif_locking": "false", "restrict_storage_xen_motion": "false", "restrict_vgpu": "false", "restrict_integrated_gpu_passthrough": "false", "restrict_vss": "false", "restrict_guest_agent_auto_update": "false", "restrict_pci_device_for_auto_update": "false", "restrict_xen_motion": "false", "restrict_guest_ip_setting": "false", "restrict_ad": "false", "restrict_nested_virt": "false", "restrict_live_patching": "false", "restrict_set_vcpus_number_live": "false", "restrict_pvs_proxy": "false", "restrict_igmp_snooping": "false", "restrict_rpu": "false", "restrict_pool_size": "false", "restrict_cbt": "false", "restrict_usb_passthrough": "false", "restrict_network_sriov": "false", "restrict_corosync": "true", "restrict_zstd_export": "false", "restrict_pool_secret_rotation": "false" }, "ha_statefiles": [], "ha_network_peers": [], "blobs": {}, "tags": [], "external_auth_type": "", "external_auth_service_name": "", "external_auth_configuration": {}, "edition": "xcp-ng", "license_server": { "address": "localhost", "port": "27000" }, "bios_strings": { "bios-vendor": "Dell Inc.", "bios-version": "2.11.2", "system-manufacturer": "Dell Inc.", "system-product-name": "PowerEdge FC640", "system-version": "", "system-serial-number": "FDKXJF3", "baseboard-manufacturer": "Dell Inc.", "baseboard-product-name": "0CTHW9", "baseboard-version": "A01", "baseboard-serial-number": ".FDKXJF3.CNWS3000CG00DP.04.", "oem-1": "Xen", "oem-2": "MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d", "oem-3": "Dell System", "oem-4": "5[0000]", "oem-5": "14[1]", "oem-6": "26[0]", "oem-7": "17[119011A83995114C]", "oem-8": "17[0C7BDEA7BD862A09]", "oem-9": "18[0]", "oem-10": "19[1]", "oem-11": "19[1]", "hp-rombios": "" }, "power_on_mode": "", "power_on_config": {}, "local_cache_sr": "OpaqueRef:f60b963a-a3ad-4b36-ad60-abda02597930", "chipset_info": { "iommu": "true" }, "PCIs": [ "OpaqueRef:ddb430d4-0e6d-466e-8dc6-0e0c930c297c", "OpaqueRef:c5f1ec4a-0128-4f60-ac88-ebcd26b3b6db", "OpaqueRef:ac35b638-ebe0-4bcb-97d3-1f4666555ae0", "OpaqueRef:70ad0496-9f9f-48b8-9b0c-c72bbab6b635", "OpaqueRef:4ec93675-434a-463c-9fd2-2d90fe25a99c", "OpaqueRef:3aa37113-5d27-4388-a763-f297d799aa58", "OpaqueRef:31444a61-67f4-496d-87c8-58986dc3b7d9", "OpaqueRef:26f0bbe1-9fdc-4ebf-af97-6559a6f47ec0", "OpaqueRef:1d485190-cf34-42d8-abaa-429902a02d56", "OpaqueRef:1949a48b-ac59-4c51-bc0e-1d8719c9234b" ], "PGPUs": [ "OpaqueRef:8b39b81e-0852-4aff-9742-5cd30c57cb3d" ], "PUSBs": [ "OpaqueRef:b5c5d9d4-f44a-470c-855f-05418330404b" ], "ssl_legacy": false, "guest_VCPUs_params": {}, "display": "enabled", "virtual_hardware_platform_versions": [ 0, 1, 2 ], "control_domain": "OpaqueRef:04c12412-3c61-4860-b703-07854d931ed5", "updates_requiring_reboot": [], "features": [], "iscsi_iqn": "iqn.2022-09.com.quape:2586113e", "multipathing": false, "uefi_certificates": "", "certificates": [], "editions": [ "xcp-ng" ] }, "SR": { "uuid": "67537a38-9582-0854-4298-5c0f9ae48eb1", "name_label": "NVME", "name_description": "", "allowed_operations": [ "vdi_enable_cbt", "vdi_list_changed_blocks", "unplug", "plug", "pbd_create", "vdi_disable_cbt", "update", "pbd_destroy", "vdi_resize", "vdi_clone", "vdi_data_destroy", "scan", "vdi_snapshot", "vdi_mirror", "vdi_create", "vdi_destroy", "vdi_set_on_boot" ], "current_operations": {}, "VDIs": [ "OpaqueRef:332f7deb-911c-40e2-9afc-dfc682d8bcb5", "OpaqueRef:e4c556d5-118d-4609-84ee-2cef0e8e249b", "OpaqueRef:175b9dd3-c663-4129-b90f-f15ceb2c2835" ], "PBDs": [ "OpaqueRef:570825c6-05a9-4d9e-9203-78d02366466e" ], "virtual_allocation": 644245094400, "physical_utilisation": 14298042368, "physical_size": 1007984996352, "type": "ext", "content_type": "user", "shared": false, "other_config": {}, "tags": [], "sm_config": { "devserial": "" }, "blobs": {}, "local_cache_enabled": false, "introduced_by": "OpaqueRef:NULL", "clustered": false, "is_tools_sr": false }, "VDI": { "uuid": "3dfecd2b-161d-4887-81b2-7edd6655af89", "name_label": "xxx.net 0", "name_description": "Created by template provisioner", "allowed_operations": [ "generate_config", "update", "forget", "destroy", "snapshot", "resize", "copy", "clone" ], "current_operations": {}, "SR": "OpaqueRef:9a9e4971-6b70-4839-a87e-14c2837d1842", "VBDs": [ "OpaqueRef:0f3e651c-7540-4fe5-a265-4a880aed7227" ], "crash_dumps": [], "virtual_size": 322122547200, "physical_utilisation": 636416, "type": "system", "sharable": false, "read_only": false, "other_config": { "content_id": "12e5ea5a-b819-1616-ce0a-7a5838da32a8", "xo:copy_of": "8b7dc75c-7bbb-4ca1-b78a-8f8fb2721d05" }, "storage_lock": false, "location": "3dfecd2b-161d-4887-81b2-7edd6655af89", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": {}, "is_a_snapshot": false, "snapshot_of": "OpaqueRef:NULL", "snapshots": [], "snapshot_time": "19700101T00:00:00Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": false }, "message": "VDI_IO_ERROR(Device I/O errors)", "name": "XapiError", "stack": "XapiError: VDI_IO_ERROR(Device I/O errors)\n at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)\n at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)\n at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:954:37)\n at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:988:14)\n at Array.forEach ()\n at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:978:12)\n at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1144:14)" } ```

julien-f commented 1 year ago

Can you test with latest master?

eddiecheng commented 1 year ago

Hi Julien Thanks. It turn out to be the same. using the latest commit https://github.com/vatesfr/xen-orchestra/commit/083db67df9e1645a2f8fe2fac564b3aecf30d55e

Full error

```json { "code": "ETIMEOUT", "pool_master": { "uuid": "cfed9b93-b559-4833-8185-cf5d0385af05", "name_label": "blade2.quape.com", "name_description": "Default install", "memory_overhead": 1620668416, "allowed_operations": [ "vm_migrate", "provision", "vm_resume", "evacuate", "vm_start" ], "current_operations": {}, "API_version_major": 2, "API_version_minor": 16, "API_version_vendor": "XenSource", "API_version_vendor_implementation": {}, "enabled": true, "software_version": { "product_version": "8.2.1", "product_version_text": "8.2", "product_version_text_short": "8.2", "platform_name": "XCP", "platform_version": "3.2.1", "product_brand": "XCP-ng", "build_number": "release/yangtze/master/58", "hostname": "localhost", "date": "2022-10-12", "dbv": "0.0.1", "xapi": "1.20", "xen": "4.13.4-9.28.1", "linux": "4.19.0+1", "xencenter_min": "2.16", "xencenter_max": "2.16", "network_backend": "openvswitch", "db_schema": "5.602" }, "other_config": { "agent_start_time": "1675968765.", "boot_time": "1675968707.", "rpm_patch_installation_time": "1675343505.428", "iscsi_iqn": "iqn.2022-09.com.quape:6a4e1d75" }, "capabilities": [ "xen-3.0-x86_64", "hvm-3.0-x86_32", "hvm-3.0-x86_32p", "hvm-3.0-x86_64", "" ], "cpu_configuration": {}, "sched_policy": "credit", "supported_bootloaders": [ "pygrub", "eliloader" ], "resident_VMs": [ "OpaqueRef:d84882fe-d49d-42ac-ae4a-15a2e62e6039", "OpaqueRef:0e6dfcfb-f4a3-4a6f-af24-7c3929432e87", "OpaqueRef:11328a23-8432-48a0-8239-33f79de6db20", "OpaqueRef:d9b020a6-4684-43a3-a477-7eecca0fc025", "OpaqueRef:b0264f39-15fe-40eb-b7c4-10555e930112", "OpaqueRef:2bcb5f5b-7325-4f62-8238-e086cf5811da" ], "logging": {}, "PIFs": [ "OpaqueRef:dc7e9669-4dff-4f17-92eb-59f2ed8e082c", "OpaqueRef:9f0049cf-7451-431f-b3e8-61f41579ac3f", "OpaqueRef:8c247822-d16b-4d78-8ad8-16cd210abe1c", "OpaqueRef:717778c0-5208-4b32-9216-6067c0785ce0" ], "suspend_image_sr": "OpaqueRef:00386e62-5ae1-44f5-8c06-3a2fa0acc87e", "crash_dump_sr": "OpaqueRef:00386e62-5ae1-44f5-8c06-3a2fa0acc87e", "crashdumps": [], "patches": [], "updates": [], "PBDs": [ "OpaqueRef:9da23a5a-eb69-4572-bbd8-20be28f1c555", "OpaqueRef:4140cecc-0235-4139-ba3a-8a2daa32b27a", "OpaqueRef:3f71f131-2090-4410-a61c-d483cafaccc9", "OpaqueRef:3cf47bc9-9dfb-4804-a711-1d83f986ec30", "OpaqueRef:3c93cb5c-42dd-46d4-b0aa-ca11cad04bc0", "OpaqueRef:2ed865e3-145e-415c-ba6f-56c8f13a42df", "OpaqueRef:10606c1e-7e38-4f57-bfc7-a276d24f4548" ], "host_CPUs": [ "OpaqueRef:7d811e42-a764-4eba-a415-aff131d200f1", "OpaqueRef:e95a5429-8814-4346-b9e1-219a7434e881", "OpaqueRef:fd23536a-1077-4014-92ca-eb8f741d6455", "OpaqueRef:d47ab879-5307-49de-8577-1e702e320604", "OpaqueRef:4ffeb9ed-42cc-49f9-ac8c-81d80d4ff091", "OpaqueRef:0f100e41-4459-48e3-88c6-dc1e105a5b77", "OpaqueRef:b2cf1b3f-65e3-47f7-afd8-0db8a48a37ed", "OpaqueRef:340f97b6-410d-4016-bfe4-3994ecd7b7bc", "OpaqueRef:299087fe-7f7c-47f7-94f5-fb1f3e2658f5", "OpaqueRef:95348090-b343-4158-8423-878f45404e91", "OpaqueRef:bcc2d95a-9ff3-4a88-90f8-601a04bc568a", "OpaqueRef:2b77faa1-25be-4c82-803e-bc62a7783a6d", "OpaqueRef:8f98ace9-d03b-4180-ac6a-207e9c3e524a", "OpaqueRef:3df44d87-4f4b-42de-95d1-c0b066e03f2b", "OpaqueRef:64bc3841-5975-46e9-b4e1-d7ed2f5aeedd", "OpaqueRef:a68de1d1-5d7f-474f-b6cc-4012e5ce30b1", "OpaqueRef:aad23092-cb6c-402d-9d3e-def149208139", "OpaqueRef:42794787-b409-416a-900a-330c2fb897cc", "OpaqueRef:7a9c3e3c-0a34-4fb8-b799-a00d4b615286", "OpaqueRef:f81b626b-8681-4e24-8067-b8541a237e88", "OpaqueRef:bbf52d57-a6bd-4303-ab0f-f158ce7eb824", "OpaqueRef:4aecc522-f4ed-4f23-992c-fbcf2ba723fb", "OpaqueRef:b05d040e-1233-4241-a905-190898d2ba17", "OpaqueRef:b1790804-9795-4064-9c4b-60230ef1ea19", "OpaqueRef:229f4834-591e-4bca-a756-49a070a573fe", "OpaqueRef:2a3d9ac1-ccb3-4e8a-ae6e-f294c1b1e88e", "OpaqueRef:d2ad0f09-4456-4124-9da4-9ed9a57b3747", "OpaqueRef:d5948ebe-912c-450c-ad8f-dfc28de21d0a", "OpaqueRef:fbcb128f-2332-4c52-8737-c022a86b277b", "OpaqueRef:3973a0f7-210f-4796-9450-0d1a4c897c85", "OpaqueRef:bfa27888-e183-4c09-8a80-ec91dca4e4c6", "OpaqueRef:a315839a-3dd8-40b3-908e-d5847b1491ac", "OpaqueRef:0835d80b-0c1a-438f-ab0e-4de70f41f6a8", "OpaqueRef:939a40e7-0b3b-4abb-9d37-c499de1fc0cb", "OpaqueRef:21467708-ac48-417a-8336-54cde7c2b51e", "OpaqueRef:277dbf30-1603-452e-98d6-6810346afb62", "OpaqueRef:5eaa8fed-2f6c-47e5-ac3c-11ba900946cd", "OpaqueRef:2eda602f-8bba-4a3d-9d95-58cc107b8ebd", "OpaqueRef:d0e020cb-4b8c-4aab-9466-186f9944a788", "OpaqueRef:541393c0-6d11-4641-a002-87dc3609b6ba" ], "cpu_info": { "cpu_count": "40", "socket_count": "2", "vendor": "GenuineIntel", "speed": "2200.080", "modelname": "Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz", "family": "6", "model": "85", "stepping": "7", "flags": "fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault intel_ppin ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase bmi1 avx2 bmi2 erms rdseed adx clflushopt clwb xsaveopt xsavec xgetbv1 arch_capabilities", "features_pv": "1fc9cbf5-f6f83203-2991cbf5-00000123-00000007-018c0b39-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000", "features_hvm": "1fcbfbff-f7fa3223-2d93fbff-00000523-0000000f-019c4fbb-00000008-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000", "features_hvm_host": "1fcbfbff-f7fa3223-2c100800-00000121-0000000f-019c47ab-00000008-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000", "features_pv_host": "1fc9cbf5-f6f83203-28100800-00000121-00000007-018c0329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000" }, "hostname": "blade2.quape.com", "address": "103.69.130.232", "metrics": "OpaqueRef:3fc1b6c6-fd0c-42d0-b233-22287c22a6a1", "license_params": { "restrict_vswitch_controller": "false", "restrict_lab": "false", "restrict_stage": "false", "restrict_storagelink": "false", "restrict_storagelink_site_recovery": "false", "restrict_web_selfservice": "false", "restrict_web_selfservice_manager": "false", "restrict_hotfix_apply": "false", "restrict_export_resource_data": "false", "restrict_read_caching": "false", "restrict_cifs": "false", "restrict_health_check": "false", "restrict_xcm": "false", "restrict_vm_memory_introspection": "false", "restrict_batch_hotfix_apply": "false", "restrict_management_on_vlan": "false", "restrict_ws_proxy": "false", "restrict_vlan": "false", "restrict_qos": "false", "restrict_pool_attached_storage": "false", "restrict_netapp": "false", "restrict_equalogic": "false", "restrict_pooling": "false", "enable_xha": "true", "restrict_marathon": "false", "restrict_email_alerting": "false", "restrict_historical_performance": "false", "restrict_wlb": "false", "restrict_rbac": "false", "restrict_dmc": "false", "restrict_checkpoint": "false", "restrict_cpu_masking": "false", "restrict_connection": "false", "platform_filter": "false", "regular_nag_dialog": "false", "restrict_vmpr": "false", "restrict_vmss": "false", "restrict_intellicache": "false", "restrict_gpu": "false", "restrict_dr": "false", "restrict_vif_locking": "false", "restrict_storage_xen_motion": "false", "restrict_vgpu": "false", "restrict_integrated_gpu_passthrough": "false", "restrict_vss": "false", "restrict_guest_agent_auto_update": "false", "restrict_pci_device_for_auto_update": "false", "restrict_xen_motion": "false", "restrict_guest_ip_setting": "false", "restrict_ad": "false", "restrict_nested_virt": "false", "restrict_live_patching": "false", "restrict_set_vcpus_number_live": "false", "restrict_pvs_proxy": "false", "restrict_igmp_snooping": "false", "restrict_rpu": "false", "restrict_pool_size": "false", "restrict_cbt": "false", "restrict_usb_passthrough": "false", "restrict_network_sriov": "false", "restrict_corosync": "true", "restrict_zstd_export": "false", "restrict_pool_secret_rotation": "false" }, "ha_statefiles": [], "ha_network_peers": [], "blobs": {}, "tags": [], "external_auth_type": "", "external_auth_service_name": "", "external_auth_configuration": {}, "edition": "xcp-ng", "license_server": { "address": "localhost", "port": "27000" }, "bios_strings": { "bios-vendor": "Dell Inc.", "bios-version": "2.11.2", "system-manufacturer": "Dell Inc.", "system-product-name": "PowerEdge FC640", "system-version": "", "system-serial-number": "BDKXJF3", "baseboard-manufacturer": "Dell Inc.", "baseboard-product-name": "0CTHW9", "baseboard-version": "A01", "baseboard-serial-number": ".BDKXJF3.CNWS3000CG00B6.02.", "oem-1": "Xen", "oem-2": "MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d", "oem-3": "Dell System", "oem-4": "5[0000]", "oem-5": "14[1]", "oem-6": "26[0]", "oem-7": "17[0BE81FA865312E57]", "oem-8": "17[11F4E2A7E6AA0035]", "oem-9": "18[0]", "oem-10": "19[1]", "oem-11": "19[1]", "hp-rombios": "" }, "power_on_mode": "", "power_on_config": {}, "local_cache_sr": "OpaqueRef:00386e62-5ae1-44f5-8c06-3a2fa0acc87e", "chipset_info": { "iommu": "true" }, "PCIs": [ "OpaqueRef:eb5fc608-7670-4e08-b030-8d1fa54f203f", "OpaqueRef:db09c222-8244-4ae6-880f-57789f388e11", "OpaqueRef:d84f7ce0-5a59-42cd-8e71-5b2853b178bd", "OpaqueRef:a4b6aead-cb87-4b0b-b937-d2ddf08b1e65", "OpaqueRef:8e212eef-1edf-4445-a3e1-8db1fa80ac20", "OpaqueRef:7cefbb75-df9a-47f3-863c-82ed9d71a5e3", "OpaqueRef:727f9849-3530-45fa-8b4d-efb8168651f9", "OpaqueRef:42f12867-069e-4dec-a693-4def926fb43f", "OpaqueRef:3d74fa16-088d-4924-af34-42b899df93dd", "OpaqueRef:1cdf80f5-8790-4b7a-8019-5ce29705b60f" ], "PGPUs": [ "OpaqueRef:989ea17b-9b69-4f30-b312-7b05a59c8556" ], "PUSBs": [ "OpaqueRef:a89c1cf3-306a-4c76-8aa3-b6e98a3e58a2" ], "ssl_legacy": false, "guest_VCPUs_params": {}, "display": "enabled", "virtual_hardware_platform_versions": [ 0, 1, 2 ], "control_domain": "OpaqueRef:2bcb5f5b-7325-4f62-8238-e086cf5811da", "updates_requiring_reboot": [], "features": [], "iscsi_iqn": "iqn.2022-09.com.quape:6a4e1d75", "multipathing": false, "uefi_certificates": "", "certificates": [], "editions": [ "xcp-ng" ] }, "SR": { "uuid": "43720163-18b5-686f-f387-bc9ac470dd47", "name_label": "SSD2", "name_description": "", "allowed_operations": [ "vdi_enable_cbt", "vdi_list_changed_blocks", "unplug", "plug", "pbd_create", "vdi_disable_cbt", "update", "pbd_destroy", "vdi_resize", "vdi_clone", "vdi_data_destroy", "scan", "vdi_snapshot", "vdi_mirror", "vdi_create", "vdi_destroy", "vdi_set_on_boot" ], "current_operations": {}, "VDIs": [ "OpaqueRef:d70df70d-0fe1-4dae-8e51-0ec92cde7e21", "OpaqueRef:f9b85a70-1075-44ce-bc33-2e58bfa26a03", "OpaqueRef:956bdb8c-5fb9-476c-907a-a9c63e970c5e", "OpaqueRef:eb7ce2ad-d17d-4fb0-94c1-c1c0ec8f4e38", "OpaqueRef:1a2f9bb1-5e4e-4fe8-a56b-349ab4efc06e", "OpaqueRef:5ef0e05b-dd0c-4411-b811-bd6198ccb19e", "OpaqueRef:b70f7aed-08a9-44e6-a8c1-baf36e3f196b", "OpaqueRef:a0200a46-3020-4af9-a877-0780fb387bac", "OpaqueRef:42db52f1-1a6a-47a6-aaec-08ffafba6735", "OpaqueRef:5304001b-c383-40f1-b749-a71aa69d701d", "OpaqueRef:805ff107-c4d8-420f-8e96-9624032ac03c" ], "PBDs": [ "OpaqueRef:2ed865e3-145e-415c-ba6f-56c8f13a42df" ], "virtual_allocation": 1331439861760, "physical_utilisation": 260164993024, "physical_size": 2016108560384, "type": "ext", "content_type": "user", "shared": false, "other_config": {}, "tags": [], "sm_config": { "devserial": "scsi-357c35481c5f11bff" }, "blobs": {}, "local_cache_enabled": false, "introduced_by": "OpaqueRef:NULL", "clustered": false, "is_tools_sr": false }, "VDI": { "uuid": "bb7338ae-9d36-458c-bcdc-92905a1a3af7", "name_label": "quape.net 0", "name_description": "Created by template provisioner", "allowed_operations": [ "snapshot", "clone" ], "current_operations": {}, "SR": "OpaqueRef:423d702d-90f9-463d-b08f-9c2809f3e201", "VBDs": [ "OpaqueRef:0bdded01-1c08-464a-aa83-ebf5bde613a2", "OpaqueRef:232b6998-9ed8-4318-b64b-a2afa91f2041" ], "crash_dumps": [], "virtual_size": 322122547200, "physical_utilisation": 636416, "type": "system", "sharable": false, "read_only": false, "other_config": { "xo:copy_of": "2931b727-696a-467c-9cf3-4259b7421a11" }, "storage_lock": false, "location": "bb7338ae-9d36-458c-bcdc-92905a1a3af7", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "host_OpaqueRef:16724981-fa4d-44e5-9ac9-cf0e76ed07bf": "RW", "read-caching-reason-cfed9b93-b559-4833-8185-cf5d0385af05": "NO_RO_IMAGE", "read-caching-enabled-on-cfed9b93-b559-4833-8185-cf5d0385af05": "false" }, "is_a_snapshot": false, "snapshot_of": "OpaqueRef:NULL", "snapshots": [], "snapshot_time": "19700101T00:00:00Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": false }, "message": "A \"socket\" was not created for HTTP request before 300000ms", "name": "Error", "stack": "Error: A \"socket\" was not created for HTTP request before 300000ms\n at Timeout.ontimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202302181724/node_modules/agent-base/src/index.ts:245:40)\n at listOnTimeout (node:internal/timers:569:17)\n at processTimers (node:internal/timers:512:7)" } ```

julien-f commented 1 year ago

Are you using an HTTP proxy anywhere in your XO?

eddiecheng commented 1 year ago

No proxy setup. Let me know if you need me to perform anything to find this issue. I see that 4 other users have similar issue on fresh installation.

cirosantos0 commented 1 year ago

Hello, my return on the command npx app-conf xo-server . -p http.listenOptions.requestTimeout was 0.

I altered requestTimeout to 86400000 in /opt/xen-orchestra/packages/xo-server/config.toml file, same result. I tried a SMB/CIFS share and a NFS share. No proxy on the network between the share, server and xen-orchestra

Xen Orchestra, commit ee837 xo-server 5.109.3 xo-web 5.111.1 Fully updated XCP-ng

Xen-Orchestra on Ubuntu 22.04.01

eddiecheng commented 1 year ago

I have difficulty doing backups now after using latest commit . ee837 . Previously backups do not have socket timeout error. Now our daily Delta backups have this error as well. Restore backup is also not working. Hope this can be fixed. I have stop XOCE for now, let me know if you need help replicating the issue.

julien-f commented 1 year ago

Hello, due to a Node.js issue, requestTimeout is now set to 0 by default in XO and y ou should not change it, see a3e37ec.

eddiecheng commented 1 year ago

Hi @julien-f the settings on requestTimeout = 0

# This breaks a number of XO use cases, for instance uploading a VDI via the
# REST API, therefore it's changed to 1 day.
#
# Completely disabled for now because it appears to be broken:
# https://github.com/nodejs/node/issues/46574
requestTimeout = 0

[http.mounts]
'/' = '../xo-web/dist'

cirosantos0 commented 1 year ago

Hi Julian, I had to rebuild XOA from sources on a new VM for it to work. Even after setting requestTimeout backup to 0, restore tasks stopped at 5 minutes. It's working now, thank you Untitled

julien-f commented 1 year ago

Hey everybody,

Thanks for your reports, unfortunately, without reproducing no my side it's difficult to investigate further and to find the root cause of this issue.

If you can reproduce quickly and easily the problem, you can use git bisect to pinpoint the problematic commit, please follow my explanation there: https://xcp-ng.org/forum/post/58981

eddiecheng commented 1 year ago

@cirosantos0 you should try again few times. It might happen again. I have similar issue with rebuild totally okay the first time and it happens again.

eddiecheng commented 1 year ago

okay julien will do that.

cirosantos0 commented 1 year ago

Hi @julien-f , using you explanation about git bisect; I reached this commit:

root@xen-orchestra:/opt/xen-orchestra# git bisect bad ab96c549aee17c9ae3c4e78c43962700d11cbf8d is the first bad commit commit ab96c549aee17c9ae3c4e78c43962700d11cbf8d

bisect log:

git bisect start

# good: [103b22ebb21945e347c20b1992d473b8f45590c6] fix(backups/importDeltaVm): resize cloned VDI if necessary
git bisect good 103b22ebb21945e347c20b1992d473b8f45590c6
# bad: [c8b29da67718f8d8b891a7df6474d83534d65978] feat(xo-cli): better output for returned values
git bisect bad c8b29da67718f8d8b891a7df6474d83534d65978
# good: [a73a24c1dfab189075d8ccd7b85a534d8da32b08] chore(xo-cli): don't use exec-promise
git bisect good a73a24c1dfab189075d8ccd7b85a534d8da32b08
# bad: [083db67df9e1645a2f8fe2fac564b3aecf30d55e] feat: release 5.79.1
git bisect bad 083db67df9e1645a2f8fe2fac564b3aecf30d55e
# skip: [b42127f083611435feddd63b31329fa594567ff9] feat: technical release (#6674)
git bisect skip b42127f083611435feddd63b31329fa594567ff9
# good: [bc0afb589e96694fb30410cc5843ada6a866d995] feat(lite/stories): needed components for incoming Component Stories (#6611)
git bisect good bc0afb589e96694fb30410cc5843ada6a866d995
# bad: [45b07f46f10d6df2c425423c0b81d5bf54983b77] feat(xen-api): 1.2.4
git bisect bad 45b07f46f10d6df2c425423c0b81d5bf54983b77
# bad: [4023127c8737241f0ee2f9395628eacd3bd66751] feat(xen-api/putResource): can ignore connection premature close
git bisect bad 4023127c8737241f0ee2f9395628eacd3bd66751
# bad: [ab96c549aee17c9ae3c4e78c43962700d11cbf8d] chore: use http-request-plus@1
git bisect bad ab96c549aee17c9ae3c4e78c43962700d11cbf8d
# first bad commit: [ab96c549aee17c9ae3c4e78c43962700d11cbf8d] chore: use http-request-plus@1

eddiecheng commented 1 year ago

any possible fix on this ? I think more users are facing this issue and this is critical bug. Replication, backup and restore that takes more than 5mins.

To replicate it you can easily do this.

1.Debian 11. Fresh install minimal. 2.Follow the instructions to install from scratch.

Do any replication, backup and restore.
Any backup with > 5m will fail with socket 3000ms.

arsenieciprian commented 1 year ago

yes i m still wathing for this fix,

https://xcp-ng.org/forum/topic/7007/a-socket-was-not-created-for-http-request-before-300000ms/36

the bus is related or is the same

julien-f commented 1 year ago

I have not been able to reproduce.

If anyone can reproduce in an official XO appliance (feel free to request a trial extension if necessary) and open a support tunnel I'll be able to investigate 🙂

arsenieciprian commented 1 year ago

extend pls for my email i will try to test @.***

On 28 Feb 2023, at 21:41, Julien Fontanet @.***> wrote:

I have not been able to reproduce.

If anyone can reproduce in an official XO appliance (feel free to request a trial extension if necessary) and open a support tunnel I'll be able to investigate 🙂

— Reply to this email directly, view it on GitHub https://github.com/vatesfr/xen-orchestra/issues/6656#issuecomment-1448756730, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3ENNKZCPOYELIAZCQRTGE3WZZIGLANCNFSM6AAAAAAURG7RME. You are receiving this because you commented.

arsenieciprian commented 1 year ago

my username @gmail.com

olivierlambert commented 1 year ago

Your trial is extended, please open a support ticket with a tunnel so Julien can access it remotely :)

arsenieciprian commented 1 year ago

ok 10 minutes pls

arsenieciprian commented 1 year ago

i m unable to open ticcket , but i whas able to reproduce the error, the backups are not working on server 2

Give this id to the support: 46431

olivierlambert commented 1 year ago

Keep the tunnel open while @julien-f will access your XOA and work on it.

eddiecheng commented 1 year ago

on xoa , I've used the latest branch and do not have such issues.

arsenieciprian commented 1 year ago

ok tunnel is up and running an will remain open as long as u need id,

arsenieciprian commented 1 year ago

@julien-f @olivierlambert Are u able login ? is everything good with connection to this xoa ?

julien-f commented 1 year ago

@arsenieciprian Thanks, I'm able to connect, I'll investigate ASAP and I'll keep you posted :slightly_smiling_face:

arsenieciprian commented 1 year ago

any news ?

julien-f commented 1 year ago

Not yet, I have a lot on my plate these days, but I'll investigate today :slightly_smiling_face:

julien-f commented 1 year ago

It should be fixed, please keep me posted if that's not the case :slightly_smiling_face:

@arsenieciprian Thanks a lot for the appliance, that helped!

vatesfr / xen-orchestra

Restore VM from Delta Backups stops at exactly 5mins #6656