search

vatesfr / xen-orchestra

The global orchestration solution to manage and backup XCP-ng and XenServer.
https://xen-orchestra.com
Other
775 stars 262 forks source link

Creation of iSCSI SR from XenOrchestra fails (Synology NAS iSCSI target) #5661

Closed itnok closed 3 months ago

itnok commented 3 years ago

Context

Expected behavior

Providing an IP address for the target and the correct CHAP credentials XO should be able to connect to the Synology NAS iSCSI target, retrieve the available IQN(s), figure out the LUN and allow the creation of a shared SR.

Current behavior

Despite the correct CHAP credentials were provided, XO refuses to connect to the Synology iSCSI target pretending the credentials are wrong and it logs SR_BACKEND_FAILURE_68(, ISCSI login failed - check access settings for the initiator on the Synology NAS, if CHAP is used verify CHAP credentials, ).

Logging into XCP-ng console and adding the SR from the CLI providing the same credentials leads to immediate and correct creation of the SR. To achieve the desired result from the CLI the following sequence of commands was used...

    $ xe sr-probe     type=lvmoiscsi     device-config:target=10.10.10.124     device-config:targetIQN=iqn.2000-01.com.synology:drogonscuzy.2546e6a256    device-config:chapuser='<redacted-username>'     device-config:chappassword='<redacted-password>'

It returns something like:

<iscsi-target>
    <LUN>
        <vendor>SYNOLOGY</vendor>
        <serial>fea34a26-504d-4794-8564-b2a802ce7f97</serial>
        <LUNid>1</LUNid>
        <size>4398046511104</size>
        <SCSIid>36001405fea34a26d504dd4794d8564db</SCSIid>
    </LUN>
</iscsi-target>

Then with the now known SCSIid:

    $ xe sr-create     name-label='SR Custom Name'     type=lvmoiscsi     content-type=user     device-config:target=10.10.10.124     device-config:targetIQN=iqn.2000-01.com.synology:drogonscuzy.2546e6a256     device-config:chapuser='<redacted-username>'     device-config:chappassword='<redacted-password>'     device-config:SCSIid=36001405fea34a26d504dd4794d8564db
Adures commented 1 year ago

Hello,

I encountered the same issue: Here are the logs 

sr.probeIscsiIqns
{
  "host": "33e883a8-450e-4e77-a435-7fde7dc81fc6",
  "target": "", #comment IP removed by me
  "chapUser": "", #comment username removed by me
  "chapPassword": "* obfuscated *" #comment this was obfuscated in logs
}
{
  "code": "SR_BACKEND_FAILURE_68",
  "params": [
    "",
    "ISCSI login failed - check access settings for the initiator on the storage, if CHAP is used verify CHAP credentials",
    ""
  ],
  "call": {
    "method": "SR.probe",
    "params": [
      "OpaqueRef:022bb45a-294c-4197-8ade-2d38bdf25482",
      {
        "target": "", #comment IP removed by me
        "chapuser": "", #comment username removed by me
        "chappassword": "" #comment removed password that was NOT obfuscated in logs
      },
      "lvmoiscsi",
      {}
    ]
  },
  "message": "SR_BACKEND_FAILURE_68(, ISCSI login failed - check access settings for the initiator on the storage, if CHAP is used verify CHAP credentials, )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_68(, ISCSI login failed - check access settings for the initiator on the storage, if CHAP is used verify CHAP credentials, )
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202302011837/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xo/xo-builds/xen-orchestra-202302011837/packages/xen-api/src/transports/json-rpc.js:37:27
    at AsyncResource.runInAsyncScope (node:async_hooks:204:9)
    at cb (/opt/xo/xo-builds/xen-orchestra-202302011837/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202302011837/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202302011837/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202302011837/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202302011837/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202302011837/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202302011837/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202302011837/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202302011837/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202302011837/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (node:internal/timers:471:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

I was able to connect with CHAP authentication when:

  1. I removed CHAP on Synology
  2. Searched again in XenOrchestra
  3. It found LUN, as below

image

  1. Enabled CHAP authentication on Synology
  2. Added credentials in XenOrchestra to the above image
  3. Clicked create, (before I clicked create I tried searching again and it gave me the same error with login failed, but I was still able to create SR after this)
  4. ISCSI SR was successfully created

Hope this helps.

EDIT:

Adding Context

XO origin: Built from source (67540)
Versions:
    xo-web: 5.111.0
    xo-server: 5.109.0
julien-f commented 1 year ago

Hi,

Are you sure your NAS is up-to-date?

We don't have this NAS to test with, can you deploy an official XO Appliance (let us know if you need a trial) and open a support tunnel so we can investigate on with your NAS directly?

Adures commented 1 year ago

Hi,

When this was happening my Synology was on DSM 7.1.1-42962 Update 1. Since then it was updated to DSM 7.1.1-42962 Update 4. I will check again this week if this is still happening after the update. If yes, I will deploy official XO Appliance and let you know.

Adures commented 1 year ago

As it always is, you declare something and turns out you have no time.

I sat down today and wanted to update xcp-ng to newest version before proceeding. I now know for sure that the bug still exists, as after restarting host I can't reconnect my to my Synology nas.

Trying to reconnect gives me this error:

sr.connectAllPbds
{
  "id": "7fac3eab-edd2-98ff-3591-543195c4b1c1"
}
{
  "code": "SR_BACKEND_FAILURE_47",
  "params": [
    "",
    "The SR is not available [opterr=ISCSI login failed - check access settings for the initiator on the storage, if CHAP is used verify CHAP credentials]",
    ""
  ],
  "task": {
    "uuid": "edb94086-0d0c-3d01-bf5c-c66462ef2537",
    "name_label": "Async.PBD.plug",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20230313T20:19:50Z",
    "finished": "20230313T20:19:51Z",
    "status": "failure",
    "resident_on": "OpaqueRef:022bb45a-294c-4197-8ade-2d38bdf25482",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "SR_BACKEND_FAILURE_47",
      "",
      "The SR is not available [opterr=ISCSI login failed - check access settings for the initiator on the storage, if CHAP is used verify CHAP credentials]",
      ""
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_access.ml)(line 32))((process xapi)(filename ocaml/xapi/xapi_pbd.ml)(line 194))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 233))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 104)))"
  },
  "message": "SR_BACKEND_FAILURE_47(, The SR is not available [opterr=ISCSI login failed - check access settings for the initiator on the storage, if CHAP is used verify CHAP credentials], )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_47(, The SR is not available [opterr=ISCSI login failed - check access settings for the initiator on the storage, if CHAP is used verify CHAP credentials], )
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202302011837/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xo/xo-builds/xen-orchestra-202302011837/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xo/xo-builds/xen-orchestra-202302011837/packages/xen-api/src/index.js:954:37)
    at forEach (/opt/xo/xo-builds/xen-orchestra-202302011837/packages/xen-api/src/index.js:988:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xo/xo-builds/xen-orchestra-202302011837/packages/xen-api/src/index.js:978:12)
    at Xapi._watchEvents (/opt/xo/xo-builds/xen-orchestra-202302011837/packages/xen-api/src/index.js:1144:14)"
}

Which happens, even when I removed chap authentication from Synology side. I probably have to do it from cli as original poster, as I see no option to change authentication from xen orchestra.

I deployed official instance, opened support tunnel and created test LUN. Please let me know whom should I give support id number.

EDIT: I tried on test LUN and the issue I described earlier still exists for sure.

olivierlambert commented 1 year ago

Okay good: you have a test system where we could probably add some debug and see what's going on :) You can give the ID here, it's perfectly secure since it's only connected to our private bastion (and all access are logged).

Also, we'll need to have the info to connect on your LUN: you can write a lun.txt file in your XO home if you want with the details so it's not public.

If you are more comfortable to keep everything private, you can create a ticket with your XO account.

Adures commented 1 year ago

I created lun.txt file in /home with details. Support tunnel ID: 36863

julien-f commented 1 year ago

@Adures Looking at it today, please keep the tunnel open :slightly_smiling_face:

Adures commented 1 year ago

@julien-f Great! Please let me know when you are done and I can close it.

julien-f commented 1 year ago

@Adures I took a first look but I still need to investigate more and unfortunately the tunnel is not working anymore, can you please re-open it?

Adures commented 1 year ago

@julien-f I checked and unfortunately there seems to be some bigger, unexpected network connectivity problem, which prevents me from connecting and resolving the issue remotely. I will be able to check it locally Thursday / Friday so this has to wait till then.

I will let you know when it's fixed and you can connect again.

Adures commented 1 year ago

@julien-f you should be able to connect again. Not sure what happened, but the host crashed and had to be restarted, so all vms there were unresponsive. Support tunnel id is the same.

One thing to keep in mind is that 27.03 my trial license will end.

julien-f commented 1 year ago

@Adures Hello, the tunnel is yet closed again, let me know when it's open and I'll get on it ASAP :slightly_smiling_face:

Don't worry about your trial, I'll extend it.

Adures commented 1 year ago

@julien-f I recreated the tunnel again, same id.

julien-f commented 1 year ago

Connected!

julien-f commented 1 year ago

@Adures I'm unable to add your ISCSI SR to XO, and therefore, unable to investigate further.

I get this error message SR_BACKEND_FAILURE_68(, ISCSI login failed - check access settings for the initiator on the storage, if CHAP is used verify CHAP credentials, ) which indicates that the CHAP credentials are incorrect.

If you can come chat on the forum, it might help speed up the investigation.

Adures commented 1 year ago

@julien-f yes, that's the problem, this error shows up, even though the CHAP credentials are correct. See my post from Feb 1. Sure, I will join the chat on the forum, so we can speed this up.

olivierlambert commented 1 year ago

Have we managed to reproduce the issue? Is it still a bug?

Adures commented 1 year ago

Hi Oliver,

I don't think so. For some reason the first set of credentials were not working as expected. I recreated the connection with completely new set of credentials and shared it with Julien, confirmed it's working and shared some details from my tests as well.

For reconnecting existing iSCSI connections between synology nas and xcp-ng host, I found workaround that doing the first 4 steps I described here https://github.com/vatesfr/xen-orchestra/issues/5661#issuecomment-1412599970 and clicking reconnect in existing iSCSI connection, allows to make it available again.

As iSCSI connection will fail every time xcp-ng host is restarted and require the workaround I stopped using it.

julien-f commented 1 year ago

@Adures Thanks a lot for all your tests :pray:

It may be worth it for the XCP-ng team to take a look at this.

Adures commented 1 year ago

If this is any help I am willing to set it up again for testing purposes

olivierlambert commented 1 year ago

Can you try again to see if it works as expected now?

olivierlambert commented 3 months ago

No news, closing. Feel free to comment if you still have the issue, thanks!