Problems about iOS Conformance Tool, cost a lots of time each test and always failed

FrankTsaiTPI commented 7 months ago

By submitting this issue you are acknowledging that any information regarding this issue will be publicly available.

If you have privacy concerns, please email conformance-tools@fidoalliance.org

FIRST PRE CHECK

[ ] I SOLEMNLY SWEAR THAT I HAVE SEARCHED DOCUMENTATION AND WAS NOT ABLE TO RESOLVE MY ISSUE

What protocol are you implementing?

[ ] FIDO2 Server
[ ] CTAP2.0
[ ] CTAP2.1
[x] UAF 1.1
[ ] U2F 1.1
[ ] U2F 1.2

NOTE: UAF 1.0 certification have been officially sunset. U2F 1.2 only supported version of U2F.

What is your implementation class?

[ ] Security Key / FIDO2 / U2F authenticators
[ ] Server
[x] UAF Client-ASM-Authenticator combo
[ ] UAF Client
[ ] UAF ASM-Authenticator

If you are platform authenticator vendor, please email conformance-tools@fidoalliance.org

What is the version of the tool are you using?

v1.6.2

What is the OS and the version are you running?

iOS 14.8 For desktop tools

[x] OSX
[ ] Windows
[ ] Linux

For UAF mobile tools

[x] iOS
[ ] Android

Issue description

When I passed my payload to the conformance tool, each test cost lots of time to finish, it takes about 30 minutes to reach Test 5/167 and every cases are failed, is this normal? Here is one of my payload sample:

Received from test tool: FidoUAFClient1://x-callback-url/UAF_OPERATION?x-success=FIDOConformaceToolsIonic://x-callback-url/UAF_OPERATION_RESULT&key=ZIBTJRDRW9LRWrmLuHyuQcP-4P-6mxXPLeOIPidmWIU&json=ewogICJtZXNzYWdlIiA6ICJ7XG4gIFwidWFmUHJvdG9jb2xNZXNzYWdlXCIgOiBcIlt7XFxcImhlYWRlclxcXCI6e1xcXCJ1cHZcXFwiOntcXFwibWFqb3JcXFwiOjEsXFxcIm1pbm9yXFxcIjoxfSxcXFwib3BcXFwiOlxcXCJSZWdcXFwiLFxcXCJhcHBJRFxcXCI6XFxcIlxcXCJ9LFxcXCJjaGFsbGVuZ2VcXFwiOlxcXCJIOWlXOXlBOWFBWEZfbGVsUW9pX0RoVWs1MTRBZDhUcXYwekNuQ3FLRHBvXFxcIixcXFwidXNlcm5hbWVcXFwiOlxcXCJoZWxsb0B0ZXN0LmNvbVxcXCIsXFxcInBvbGljeVxcXCI6e1xcXCJhY2NlcHRlZFxcXCI6W1t7XFxcImFhaWRcXFwiOltudWxsXX1dXX19XVwiXG59IiwKICAiY2hhbm5lbEJpbmRpbmdzIiA6ICJ7XG4gIFwic2VydmVyRW5kUG9pbnRcIiA6IG51bGwsXG4gIFwiY2lkX3B1YmtleVwiIDogbnVsbCxcbiAgXCJ0bHNTZXJ2ZXJDZXJ0aWZpY2F0ZVwiIDogbnVsbCxcbiAgXCJ0bHNVbmlxdWVcIiA6IG51bGxcbn0iCn0&state=3FD8ECAB4F03438889E91DCAF2619A5C

UAFMessage: {\"additionalData\":\"\",\"uafProtocolMessage\":\"{\\\"exts\\\":[],\\\"statusCode\\\":0,\\\"responseData\\\":{\\\"assertionScheme\\\":\\\"UAFV1TLV\\\",\\\"assertion\\\":\\\"AT5rAgM-sQALLgkARkZGRiNGQzAxDi4HAAABAQEAAAEKLiAAHrsZ6slQrlFL0S6oLThGEM4_Q51yaxYZymcfr98F07UJLiAArjVz4OvjfjN3AS0w2PkHqAaM3KpIxif-1t2pwWssNsgNLggAAAAAAAEAAAAMLkEABOJ9odoc3hrDmkUMP3xiKOTa5fil3BLy8pgt5uJPel5YZY1qyI-41HNHbx0a3Vt1oIhjWjYTe8IFiEuRAi2NtocHPrIBBi5AAOZMU7AgPvkR5vkUwHk8rpjCchNCcAloEO7EtKX2renTj73ErbypV9nhhyyvdEnOVXAVWuvxvR7aPjukfL0-WnIFLmoBMIIBZjCCAQugAwIBAgIUItr5oi4HDdUZZXm9ez4oJ6ayUYswCgYIKoZIzj0EAwIwFTETMBEGA1UEAwwKVFBJRmlkb1NESzAeFw0yNDAxMjYxMDM0NTVaFw0yNTAxMjUxMDM0NTVaMBUxEzARBgNVBAMMClRQSUZpZG9TREswWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATifaHaHN4aw5pFDD98Yijk2uX4pdwS8vKYLebiT3peWGWNasiPuNRzR28dGt1bdaCIY1o2E3vCBYhLkQItjbaHozkwNzAPBgNVHRMBAf8EBTADAQH_MA4GA1UdDwEB_wQEAwICBDAUBgNVHREEDTALgglsb2NhbGhvc3QwCgYIKoZIzj0EAwIDSQAwRgIhANTW6tsed8w5MN_I7mt6UiAiTY-DV5xLWRAGn4A-GZhlAiEAtFLufvknZBFkNFnfd1wZNia6yHdNkZBVsdFKffFvzdI=\\\"}}\"}

URL call back for test tool FIDOConformaceToolsIonic://x-callback-url/UAF_OPERATION_RESULT?state=3FD8ECAB4F03438889E91DCAF2619A5C&json=ZXlKbGJtTWlPaUpCTVRJNFEwSkRMVWhUTWpVMklpd2lZV3huSWpvaVpHbHlJbjAuLm9HeFVySjBhd2NrSm0xbFFnRzAteGcuSTJnVU1GcDNSUWtaMmNZV1Q0NFRWRlFWVG1mOUNLalZFeE51LW16OUhXWkctYlFGb3hoUTdYLUh6QmhhRHktS3BDYmhZUW4wMS1EYzE0ZEMzTHZSZ0RValRmLWF2VjRWQVVwbkZYMU9oRmJJRzJaTHhDaXYxZ0RxZ1Vvc0U0VlJYMEpxbUdCZ2V3OFA4WVdoUnF2VUF2VXhTb3Y1OFN1dXNyVlM5X2tjM1cyNGZGNnNhbkQ1azl0REg2c2RXRkUxQ2lfSVJRUkdZMGZfa0ZhNWdtVE1XMXpUYmF6VWd1MlBFdS1oTkJJMVZOZGMyQkw0ei1yTWVzdkdxcmFlSHJfb2ZzLVRjcVV0bDJUcU1pNi12ZGloQ0dMV1F0UnJiRU1oSzFnb0VSVDkzNnFYbGhJUVhRb2pLLWdPdUhqYVl1UUtDZWp4QWdNcG1zOF9KN3NtSThuR2lIMklGZUxsVlZYUnd6N2VtLV9RUnlUZ0V6bDUybFZscHoxZmwxMk1obFVlcWhxSnZyR0lRV1F3NDFtcFRveTlIcHh2OHpWV2lqVUszemJmd0xWdFFoTTFUc2FVWWVLUzd1cThRZ21qMmdnUTVudkxqU3RxNXVRb3VEeVN1NWE5MFdraWpFcDhDODgyMkNDOHBfRExPZmNRNy1qQ2d2dm84bWx1b05OLVJzUDNnNTRlYk1xVExabzdoSy1ZQmVTY2pQdS04RF9ZX0c5bWJ3TDA4VDFfTnJDQ1JETVFCVVJjYUk0Z2ZoalFRcGJucTRwRnA4eFpnYTZnS21uTXMxS0xHaHdGU2d6a1BJSGQxOXhNMTV0dTJ6YlNWejBLZVM5eUg1eGt5eHhsNWNVM1UzbElYY2QwTGFfRF9vT0lnTnE5RndrOVNORXV6Z18xczlpUkl0ekkwbHljQmVOdWJCZVliWVotdTA3N1NuUmdCMzhJZnU4WWd1TlJROE13VGxRc1JwQTkzLTRjYzBSMXZtbHBmWHIzX19GUE4xTHF6QXpLWTJRR1VjZmQ0dVI0QUhCYTV2RGlUalZDME95YWFGbXpPLUowZk0ycmJoSC1UUzlabm1SdVZRMmZTUExyWVRZTXB5QU9KRUFvZkVmSERlYmpkYWdYUE05d1F5UFVUTWM2TGlpM0NBZnRtVnczeXJXelQ0T0Fqc2hDSVhyT1EzMXNBbFUwVWJraGxTUUxYZjc0dUJfRTBPNXhrVHVRcFJ3bFFMbW5ZTkpHVDZORnlIYWlKcER4bXZvdWVuX3VCWURuQmNNRTdKRnp1a0RNZ1hrVk1CaTdDbmNpZktVbHpwYkRPYXVLcGtaVmo0MDBfUU4zRkUxNlFLd0FGczNJWnV6LTF3SmlIbUJPQ2d5Y3NHMFZOZG1QZkRZWkZKNDF5bHZkV1RVcnhfRGptNzc5YmZRVlR6YnhRQ2k3bVZ0c0VkSHk5OGc0ZW1hX0ZuM1RKbU9tZzk4aEJWeVJFbTFINl9WS29QUUFnV2s1Nm9telhIb3dTS3RWWkxncjhQaUVUTGVyWHE0bnVZRDFvOFVlcENNQjdpb2hvdFd0eEM4eEdaTm5HMG5VYlgzTEwtbWYwNVRtc1lGU0VfWXNScC0yVjh0Q2NfTUhKTXN6eGk5bzBnZzdKM2MzUXVLZ3pIRDBQTUF4RXpyRm92MXZ1Nl9DcWhpQ0N1XzdkOXVyNFJSS1N4UkUwaHVoQ1pFU29OenRZYXVjYy1sUzIxeEZlT1VSdFpJcXNBUUF6endXbzI4RlN0dDRvWEFIZ1lxOThxWXd4YzJhV2VXVkhZSnBqWlpleFVUZlVocFZFYjRva0dhaDBETVZXaFp2aHhJZURUVFJ2bnkxUldxOW10aFNFNzJlTmRRQ0hiWWQ2Y3ZCaTczVC4zMnFqeHlDSEktc2lkWFhTUlNXc2hn

iirachek commented 7 months ago

No, 30 minutes to reach the 5th test isn't normal. Can you describe, what is happening during this time? Does the tooling report any specific error?

FrankTsaiTPI commented 7 months ago

No, 30 minutes to reach the 5th test isn't normal. Can you describe, what is happening during this time? Does the tooling report any specific error?

No, I don't have received any error from tool, it looks like running normally, except it takes a long time

I start a new test, and first test takes about 5 minutes and failed

iirachek commented 6 months ago

I've looked into it and think this is likely the issue with the formatting of callback URL. Here is a short explanation of what can be done and why it works that way.

FrankTsaiTPI commented 6 months ago

I've looked into it and think this is likely the issue with the formatting of callback URL. Here is a short explanation of what can be done and why it works that way.

It's a bit strange, before I fired this issue, I inquired about another issue with the FIDO Alliance via email. The response I received was, according to the documents, my URL was incorrect, and I needed to change '&' to '?'. If I did that, I would encounter another issue

Here is the question I ask before: According to the document here: fido-uaf-client-api-transport-v1.2, I have to use x-callback-url to communicate with the testing tools.

I got some information from the testing tool, for example: FidoUAFClient1://x-callback-url/UAF_OPERATION?x-success=FIDOConformaceToolsIonic://x-callback-url/UAF_OPERATION_RESULT&state=DAAC327185034F27B0F2E9BDC8DC4F74&key=WcT8PY8A2XumZ9HmIfTGojPUES_3Io6YKi5OGvgX3TY&json=ewogICJtZXNzYWdlIiA6ICJ7XG4gIFwidWFmUHJvdG9jb2xNZXNzYWdlXCIgOiBcIlt7XFxcImNoYWxsZW5nZVxcXCI6XFxcIkg5aVc5eUE5YUFYRl9sZWxRb2lfRGhVazUxNEFkOFRxdjB6Q25DcUtEcG9cXFwiLFxcXCJ1c2VybmFtZVxcXCI6XFxcImhlbGxvQHRlc3QuY29tXFxcIixcXFwicG9saWN5XFxcIjp7XFxcImFjY2VwdGVkXFxcIjpbW3tcXFwiYWFpZFxcXCI6W251bGxdfV1dfX1dXCJcbn0iLAogICJjaGFubmVsQmluZGluZ3MiIDogIntcbiAgXCJzZXJ2ZXJFbmRQb2ludFwiIDogbnVsbCxcbiAgXCJjaWRfcHVia2V5XCIgOiBudWxsLFxuICBcInRsc1NlcnZlckNlcnRpZmljYXRlXCIgOiBudWxsLFxuICBcInRsc1VuaXF1ZVwiIDogbnVsbFxufSIKfQ

And I replied: FIDOConformaceToolsIonic://x-callback-url/UAF_OPERATION_RESULT&state=DAAC327185034F27B0F2E9BDC8DC4F74&json=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..dDKdx84VykrabLWVqJtv_A.G2E_Z5IyZ0UG3nCUw926kg.vI7z-bparONgjnxna7wr9w I replied the same state, and the json value was a JWE encrypted with the key I got from the test tool. But the weird thing was the original json was just a ramdom string encrypted with the key using JOSE framework (JWE; alg set direct, enc set A128CBCHS256), and the testing tool still said that I sent it {errorCode: 6}.

I passed 122 test cases because the cases were expecting errorCode 6, but I just sent a ramdon string (ex. an JWE encryped "aaaa" with the key I got from the tool).

================================== Here is the replied I received: The 0x06 error code is the PROTOCOL_ERROR. Certain tests are intentionally performed with incomplete or invalid data to ensure that the other party detects such requests and handles them according to the protocol. The likely reason behind tools displaying this error is that they failed to process the decoded response due to json being a random string.

Do note that the delimiter in the response between [UAFxResponseType] and 'state=' should be '?' instead of '&' (see Example 11)

iirachek commented 6 months ago

In this case I'd suggest following what's outlined by the comment, since it was left by the original developer of the tooling.

The email response references an example from specification, so it's understandable why it was mentioned. Unfortunately, in practice this particular part of specification differs from what's actually expected from the implementation.

litaoyu commented 6 months ago

Hello, is there a solution to this problem? I had the same problem

FrankTsaiTPI commented 6 months ago

Hello, is there a solution to this problem? I had the same problem

No, still looking for solution, I already change ? to & and got another problem

fido-alliance / conformance-test-tools-resources