[TT-1417] Custom key registration failing with sha256 enabled when OrgID is empty

[mcandre]

Branch/Environment/Version

• Branch/Version: 2.9.4.3
• Environment: community

Describe the bug

When "sha256" is configured for hash key function, then the Tyk admin REST interface mangles custom API keys on registration, failing to preserve the key that the administrator supplies.

Reproduction steps

1. Enable sha256 hash function configuration.
2. Submit a user-defined API key for registration.
3. Lookup the API key.

Actual behavior

Redis registers a completely different text key.

Tyk admin key lookup returns HTTP status code 404.

Expected behavior

Tyk and should continue operating largely as it has been with murmur32 configured. Only the hash flavor in Redis should be affected, not the plaintext API keys.

Followup

Has the Tyk REST interface changed in some documented way within the v2.x series, on a hash function relative basis?

[tedbjorklund]

To provide more detail, we're following the steps outlined here https://tyk.io/docs/frequently-asked-questions/import-existing-keys-tyk/ and are seeing these results instead.

➜  ~ curl --request POST \
  --url http://localhost:8080/tyk/keys/tykdirecthitsha256key \
  --header 'content-type: application/json' \
  --header 'x-tyk-authorization: asdfasdfasdfasdfasdfasdfasdf' \ 
  --data '{
        "org_id": "",
        "alias": "am-validation",
        "apply_policies": [
                "first-api-key"
        ],
        "meta_data": {
                "nm-name": "am-validation"
        }
}'
{"key":"eyJvcmciOiIiLCJpZCI6InR5a2RpcmVjdGhpdHNoYTI1NmtleSIsImgiOiJzaGEyNTYifQ==","status":"ok","action":"added","key_hash":"99fff1e1ec6db98739fed788820dc32f5be0a73c075a9fdb3b5f6aa429b9c36c"}

I would expect the "key" in the response to be tykdirecthitsha256key, as it is when using murmur. When trying to make an api call to get the key just created results in a 404.

➜  ~ curl --request GET \   
  --url http://localhost:8080/tyk/keys/tykdirecthitsha256key \
  --header 'x-tyk-authorization: asdfasdfasdfasdfasdfasdfasdf'
{"status":"error","message":"Key not found"}

API call with that key

➜  ~ curl -k -H "Authorization: tykdirecthitsha256key" https://tyk.api-management/api/v1/first/stuff
{
    "error": "Access to this API has been disallowed"
}

Doing the same set of steps with the defaults, results in different behavior.

➜  ~ curl --request POST \   
  --url http://localhost:8080/tyk/keys/tykdirecthitmurmurkey \
  --header 'content-type: application/json' \                 
  --header 'x-tyk-authorization: asdfasdfasdfasdfasdfasdfasdf' \
  --data '{
        "org_id": "",
        "alias": "am-validation",
        "apply_policies": [
                "first-api-key"
        ],
        "meta_data": {
                "nm-name": "am-validation"
        }
}'
{"key":"tykdirecthitmurmurkey","status":"ok","action":"added","key_hash":"5b51c6df"}

And a GET for the key results in success.

➜  ~ curl --request GET \    
  --url http://localhost:8080/tyk/keys/tykdirecthitmurmurkey \
  --header 'x-tyk-authorization: asdfasdfasdfasdfasdfasdfasdf'
{"last_check":0,"allowance":0,"rate":0,"per":0,"throttle_interval":0,"throttle_retry_limit":0,"date_created":"2020-08-26T18:27:15.783737365Z","expires":0,"quota_max":0,"quota_renews":1598466435,"quota_remaining":0,"quota_renewal_rate":0,"access_rights":{"first-api-proxy.sample-first":{"api_name":"first-api-proxy","api_id":"first-api-proxy.sample-first","versions":["stable","pre-release"],"allowed_urls":[{"url":"/","methods":["GET","HEAD","POST","PUT","DELETE","PATCH","OPTIONS","CONNECT","TRACE"]}],"limit":{"rate":0,"per":0,"throttle_interval":0,"throttle_retry_limit":0,"quota_max":0,"quota_renews":0,"quota_remaining":0,"quota_renewal_rate":0},"allowance_scope":""},"redis-commander-proxy.db":{"api_name":"redis-commander-proxy","api_id":"redis-commander-proxy.db","versions":["stable"],"allowed_urls":[{"url":"/","methods":["GET","HEAD","POST","PUT","DELETE","PATCH","OPTIONS","CONNECT","TRACE"]}],"limit":{"rate":0,"per":0,"throttle_interval":0,"throttle_retry_limit":0,"quota_max":0,"quota_renews":0,"quota_remaining":0,"quota_renewal_rate":0},"allowance_scope":""},"tyk-http-echo-ms.api-management":{"api_name":"tyk-http-echo-ms","api_id":"tyk-http-echo-ms.api-management","versions":["stable"],"allowed_urls":[{"url":"/","methods":["GET","HEAD","POST","PUT","DELETE","PATCH","OPTIONS","CONNECT","TRACE"]}],"limit":{"rate":0,"per":0,"throttle_interval":0,"throttle_retry_limit":0,"quota_max":0,"quota_renews":0,"quota_remaining":0,"quota_renewal_rate":0},"allowance_scope":""}},"org_id":"","oauth_client_id":"","oauth_keys":null,"certificate":"","basic_auth_data":{"password":"","hash_type":""},"jwt_data":{"secret":""},"hmac_enabled":false,"enable_http_signature_validation":false,"hmac_string":"","rsa_certificate_id":"","is_inactive":false,"apply_policy_id":"","apply_policies":["first-api-key"],"data_expires":0,"monitor":{"trigger_limits":null},"enable_detail_recording":false,"meta_data":{"nm-name":"am-validation"},"tags":[],"alias":"am-validation","last_updated":"1598466435","id_extractor_deadline":0,"session_lifetime":0}

And a successful call with that api key

➜  ~ curl -k -H "Authorization: tykdirecthitmurmurkey" https://tyk.api-management/api/v1/first/stuff
{"method":"GET","response":"first-stable"}

[tedbjorklund]

Also of note, if I use the key that comes back from the creation request with sha256 enabled, it does work. But this does not meet our use case as we need to import existing keys.

a GET for that key

➜  ~ curl --request GET \                                                                                                                            
  --url http://localhost:8080/tyk/keys/eyJvcmciOiIiLCJpZCI6InR5a2RpcmVjdGhpdHNoYTI1NmtleSIsImgiOiJzaGEyNTYifQ\=\= \                   
  --header 'x-tyk-authorization: asdfasdfasdfasdfasdfasdfasdf'
{"last_check":0,"allowance":0,"rate":0,"per":0,"throttle_interval":0,"throttle_retry_limit":0,"date_created":"2020-08-26T18:22:16.872042201Z","expires":0,"quota_max":0,"quota_renews":1598466136,"quota_remaining":0,"quota_renewal_rate":0,"access_rights":{"first-api-proxy.sample-first":{"api_name":"first-api-proxy","api_id":"first-api-proxy.sample-first","versions":["stable","pre-release"],"allowed_urls":[{"url":"/","methods":["GET","HEAD","POST","PUT","DELETE","PATCH","OPTIONS","CONNECT","TRACE"]}],"limit":{"rate":0,"per":0,"throttle_interval":0,"throttle_retry_limit":0,"quota_max":0,"quota_renews":0,"quota_remaining":0,"quota_renewal_rate":0},"allowance_scope":""},"redis-commander-proxy.db":{"api_name":"redis-commander-proxy","api_id":"redis-commander-proxy.db","versions":["stable"],"allowed_urls":[{"url":"/","methods":["GET","HEAD","POST","PUT","DELETE","PATCH","OPTIONS","CONNECT","TRACE"]}],"limit":{"rate":0,"per":0,"throttle_interval":0,"throttle_retry_limit":0,"quota_max":0,"quota_renews":0,"quota_remaining":0,"quota_renewal_rate":0},"allowance_scope":""},"tyk-http-echo-ms.api-management":{"api_name":"tyk-http-echo-ms","api_id":"tyk-http-echo-ms.api-management","versions":["stable"],"allowed_urls":[{"url":"/","methods":["GET","HEAD","POST","PUT","DELETE","PATCH","OPTIONS","CONNECT","TRACE"]}],"limit":{"rate":0,"per":0,"throttle_interval":0,"throttle_retry_limit":0,"quota_max":0,"quota_renews":0,"quota_remaining":0,"quota_renewal_rate":0},"allowance_scope":""}},"org_id":"","oauth_client_id":"","oauth_keys":null,"certificate":"","basic_auth_data":{"password":"","hash_type":""},"jwt_data":{"secret":""},"hmac_enabled":false,"enable_http_signature_validation":false,"hmac_string":"","rsa_certificate_id":"","is_inactive":false,"apply_policy_id":"","apply_policies":["first-api-key"],"data_expires":0,"monitor":{"trigger_limits":null},"enable_detail_recording":false,"meta_data":{"nm-name":"am-validation"},"tags":[],"alias":"am-validation","last_updated":"1598466136","id_extractor_deadline":0,"session_lifetime":0}

A successful api call

➜  ~ curl -k -H "Authorization: eyJvcmciOiIiLCJpZCI6InR5a2RpcmVjdGhpdHNoYTI1NmtleSIsImgiOiJzaGEyNTYifQ==" https://tyk.api-management/api/v1/first/stuff
{"method":"GET","response":"first-stable"}

[joshblakeley]

Thanks a lot for the detail @tedbjorklund i'll make sure the product team get eyes on this.

[furkansenharputlu]

If I am not wrong, I think in default one, it uses murmur32 but it thinks it is a legacy token and fallbacks to murmur32. It means that hash_key_function "" and murmur32 behave differently. When you set murmur32 to hash_key_function you will see the same behavior with sha256. So, murmur32 and sha256 don't behave differently. When you import your key to Tyk by hashing, while accessing to that key, you somehow should tell the hashing algorithm so that Tyk makes hashing and gets the hashed value. For that Tyk uses a base64 encoded JSON format.

In your case: base64Decode(eyJvcmciOiIiLCJpZCI6InR5a2RpcmVjdGhpdHNoYTI1NmtleSIsImgiOiJzaGEyNTYifQ==) gives us {"org":"","id":"tykdirecthitsha256key","h":"sha256"}

So I think there is no bug here. However, there may still be a way to handle this if we answer how we will differentiate with a legacy token and a token without JSON in purpose question. A possible solution is to define a config field: hash_key_function_for_non_json_tokens, it will be set sha256. Note that if legacy token users set this field any value except murmur32, their legacy tokens will not work.

@buger What do you think?

[buger]

const defaultHashAlgorithm = "murmur64" Default is murmur64

So here we're talking about case when hashing function set non default value, and if the same issue with "murmur32" maybe we hit a bug here.

[furkansenharputlu]

const defaultHashAlgorithm = "murmur64" when you set hash_key_function and you made a mistake .i.e. mrmur128, it is used there. It is not related to here.

[buger]

@furkansenharputlu see hashFunction. If hashing function is not detected, e.g. it is not json format, it always use murmur32. Same if you had a typo.

[buger]

Hi!

We have tested all this flows, and so far it is all working with a few nuances. First I think there is still bug with Open source version (e.g. without Dashboard), when you out empty OrgID. For the custom keys in current implementation it is important to have non empty org_id. If you are using OSS gateway, then you need to send some org id, like "test" when you create a key, then you also need to have "org_id" attribute for your API definition with same value.

But in Pro on-prem installation, when orgs IDs are required it works out of the box.

Additionally, the fact that when you create a key, it returns Key in "Tyk" format is expected (one of the limitations of implementation), but you can decode it to understand how it works since it is base64 encoded json (like JWT).

When you need to get information about the key you can do it without knowing "internal" Tyk value, but you will need to pass "username=true" and "org_id" arguments like this:

curl -k --request GET \
  --url 'https://localhost:8181/tyk/keys/tykfoobar123?username=true&org_id=5d2def07058f8579bea6df27' \
  --header 'x-tyk-authorization: 352d20ee67be67f6340b4c0605b044b7'

Hope it works for you!

[buger]

Updated ticket title and added "when OrgID is empty"