[TT-7130] Transform Response configuration doesn't work when rewrite rules are configured

LowCostCustoms commented 2 years ago

Branch/Environment/Version

Branch/Version: v3.2.2
Environment: On-prem

Describe the bug When transform_response and url_rewrites are used together, transform rules are applied improperly.

Reproduction steps

Apply the following Tyk configuration:

{
  "name": "my-api",
  "api_id": "my-api",
  "org_id": "my-org",
  "proxy": {
    "listen_path": "/",
    "target_url": "",
    "strip_listen_path": true,
    "disable_strip_slash": true
  },
  "version_data": {
    "not_versioned": true,
    "versions": {
      "Default": {
        "name": "Default",
        "use_extended_paths": true,
        "extended_paths": {
          "cache": [".*"],
          "url_rewrites": [
            {
              "path": "^/$",
              "method": "GET",
              "match_pattern": "^/$",
              "rewrite_to": "http://upstream:8080/api"
            },
            {
              "path": "^/([^/]+)$",
              "method": "GET",
              "match_pattern": "^/([^/]+)$",
              "rewrite_to": "http://upstream:8080/api/$1"
            },
            {
              "path": "^/([^/]+)/([^/]+)$",
              "method": "GET",
              "match_pattern": "^/([^/]+)/([^/]+)$",
              "rewrite_to": "http://upstream:8080/api/$1/records/$2"
            }
          ],
          "transform_response": [
            {
              "path": "^/$",
              "method": "GET",
              "template_data": {
                "input_type": "json",
                "template_mode": "file",
                "template_source": "./templates/template-0.tmpl"
              }
            },
            {
              "path": "^/([^/]+)$",
              "method": "GET",
              "template_data": {
                "input_type": "json",
                "template_mode": "file",
                "template_source": "./templates/template-1.tmpl"
              }
            },
            {
              "path": "^/([^/]+)/([^/]+)$",
              "method": "GET",
              "template_data": {
                "input_type": "json",
                "template_mode": "file",
                "template_source": "./templates/template-2.tmpl"
              }
            }
          ]
        }
      }
    }
  },
  "response_processors": [
    {
      "name": "response_body_transform",
      "options": {}
    }
  ],
  "use_keyless": true,
}

Make a GET request against http://localhost:8080/first/second and ensure the response wasn't updated.

Actual behavior Request path is not matched and response body transform rules aren't applied.

Expected behavior Request path is matched and body transform rules are applied.

Configuration (tyk config file): See above.

Additional context Perhaps, the bug is that in case rewrite rules and body transform rules are used together, matching is performed not against the request path, but against the url rewrite path instead. When debugging I found the following:

gateway/api_definition.go:1191: ctxGetUrlRewritePath(r) returns the path from the request rule configuration, i.e. ^/([^/]+)/([^/]+)$;
gateway/api_definition.go:1206: an extra leading slash is added to the path, so it becomes ^/([^/]+)/([^/]+)$.

UPD: the following patch seems to solve the problem

diff --git a/gateway/mw_url_rewrite.go b/gateway/mw_url_rewrite.go
index c6fdb941..70fc0704 100644
--- a/gateway/mw_url_rewrite.go
+++ b/gateway/mw_url_rewrite.go
@@ -191,7 +191,7 @@ func (gw *Gateway) urlRewrite(meta *apidef.URLRewriteMeta, r *http.Request) (str
                log.Debug("URL Re-written to: ", newpath)

                // put url_rewrite path to context to be used in ResponseTransformMiddleware
-               ctxSetUrlRewritePath(r, meta.Path)
+               ctxSetUrlRewritePath(r, path)
        }

        newpath = gw.replaceTykVariables(r, newpath, true)

besasch88 commented 1 year ago

Hi, I know this is an open issue, but has anyone found a possible workaround? I'm stuck on transforming the response body once the URL rewrite is done and the API call is executed... I can't find an alternative. Thanks!

buger commented 1 year ago

@lorenzocastelli have you tried it with latest Tyk versions?

besasch88 commented 1 year ago

Hi @buger! Yes, I'm on the latest one: Tyk 5.0.0. I'm running it into docker: docker.tyk.io/tyk-gateway/tyk-gateway:v5.0.0

besasch88 commented 1 year ago

@buger Just to give you more context. This is my API definition:

{
  ...
  "version_data": {
    "not_versioned": true,
    "versions": {
      "v1": {
        "name": "v1",
        "use_extended_paths": true,
        "extended_paths": {
          "url_rewrites": [
            {
              "path": "/$",
              "method": "POST",
              "match_pattern": "\/$",
              "rewrite_to": "https://stage.cloudacademy.com/api/v4/organizations/accounts/$tyk_context.headers_Claim_account_id/members/invite/",
              "triggers": []
            }
          ],
          "transform": [
            {
              "path": "/$",
              "method": "POST",
              "template_data": {
                "input_type": "json",
                "template_mode": "file",
                "template_source": "/stargate/templates/transformations/internal/v1/members/requests/post_transformation.tmpl"
              }
            }
          ],
          "transform_response": [
            {
              "path": "invite/$",
              "method": "POST",
              "template_data": {
                "input_type": "json",
                "template_mode": "file",
                "template_source": "/stargate/templates/transformations/internal/v1/members/responses/post_transformation.tmpl"
              }
            }
          ]
        }
      }
    }
  },
  "proxy": {
    "listen_path": "/api/internal/v1/members",
    "target_url": "https://stage.cloudacademy.com/api/v1/not-found/",
    "strip_listen_path": true,
    "disable_strip_slash": true
  },
  "active": true,
  "enable_context_vars": true,
  "response_processors": [
    {
      "name": "response_body_transform",
      "options": {}
    }
  ]
}

Basically, everything is working well: I send the request POST /api/internal/v1/members/ to Tyk, the input body transformation is done correctly and the rewrite URL is performed. I receive back the response with a payload but before sending back to me the response, the body transformation is not performed.

If you look at the path property in the transform_response, I tried different approaches:

matching the rewrited URL
matching the inbound URL
applying a more flexible path with regex like .*

None of them seems work.

Note: Doing the same thing in a GET method API defined, the response is correctly transformed. I don't know why it happens, if it is related to the used method (only GET is supported) or other stuff.

Thanks!

besasch88 commented 1 year ago

@buger From the code shared by @LowCostCustoms , it seems that the transform_response path property needs to match the Path defined in the url_rewrites. So, as a workaround, we need to define a "regex" on the regex of the path property itself. In the example above, I have this API configuration:

"url_rewrites": [
            {
              "path": "/$",
              "method": "POST",

So, the transform_response path property needs to match the /$ instead of checking the initial path. IMHO, the fix defined above by @LowCostCustoms needs to be released, so I can apply the regex on the inbound URL, instead of on the regex

nvta-sbiyyala commented 1 year ago

@lorenzocastelli @buger, to add some context: @LowCostCustoms is part of my team, and his tyk patches have been running quite well in our production envs for over a year.

besasch88 commented 1 year ago

Hi @nvta-sbiyyala ! Thanks for the context. BTW I found a "workaround" (I honestly don't know if it was meant to work that way). But I leave the solution here for other people who have the same troube:

Having this in my API definition:

"url_rewrites": [
   {
      "path": "/$",
      "method": "POST",
      ...
   },
   {
      "path": "/([A-Za-z0-9-]*)/$",
      "method": "GET",
      "match_pattern": "\/([A-Za-z0-9-]*)\/$",
      ...
   },
]

the transform response needs to be:

"transform_response": [
   {
      "path": "/\\$",
      "method": "POST",
      ...
   }
   {
      "path": "/\(\[A-Za-z0-9-\]\*\)/\$",
      "method": "GET",
      ...
   }
]

So the path in the transform_response needs to match the path of the url_rewrites, instead of being evaluated on the inbound URL. Different case for the transform in the input request, in this case, the path is evaluated on the inbound URL.

My suggestion, as an improvement, is to follow the same approach both for rewrite_url, transform, and transform_response evaluating the Path regex always on the Inbound URL.

BTW Thanks for your time and patience!

buger commented 1 year ago

The workaround sound quite logical! But the proper fix should be that response transform mw, should depend on original url, and we probably already store it somehwere in request context.

However I quickly checked code: CheckSpecMatchesStatus https://github.com/TykTechnologies/tyk/blob/3bd48773e779fba67c5a40eac7c3c915039bba85/gateway/res_handler_transform.go#L93

https://github.com/TykTechnologies/tyk/blob/3bd48773e779fba67c5a40eac7c3c915039bba85/gateway/api_definition.go#L1443 Even code has the comment:

//If url-rewrite middleware was used, call response middleware of original path and not of rewritten path
    // context variable UrlRewritePath is set by rewrite middleware

So seems like this function has logic to work with rewriting 🤔

TykTechnologies / tyk

[TT-7130] Transform Response configuration doesn't work when rewrite rules are configured #3858