external classifier doesn't encode all characters in URI

[elico]

I am testing the external classifier feature and it works great. I am able to use reloaded config files, multiple external DNS categorization systems and DB's of many vendors.

My service received then next URLs from redwood:

https://app.frase.io/bot/%7B%7Binteraction.answers[0].product_image%7D%7D
https://app.frase.io/bot/{{interaction.answers[0].product_image}}

Which contains square brackets and UriParser library (https://github.com/uriparser/uriparser) doesn't like it. I believe these should be encoded before passing it to external systems that use RFC. Currently what I am doing is replacing every: "[" with "%5B" and "]" with "%5D"

[andybalholm]

Here is what the POST body would be for the first URL:

method=GET&url=https%3A%2F%2Fapp.frase.io%2Fbot%2F%257B%257Binteraction.answers%5B0%5D.product_image%257D%257D

But when your service decodes the values from the form, it gets the same URL that Redwood received in the original request from the client. I think it's best if Redwood doesn't try to "clean up" the URL.

By the way, that URL looks like it's the result of an HTML template that didn't get filled in properly.

[elico]

@andybalholm I assumed that there would be some RFC compatibility however I understand why keeping the url as it is would be fine for most use cases. I would like to verify if possible if there are limitations in RedWood to what is acceptable in a URL or not. There is probably a number of tests that can be run against RedWood and the GoLang HTTP library while I assume these already exist.

I do not intend to cleanup the url in the proxy but to be able to pass this url to the URI parser I need to either:

• write my own parser
• make the url readable by standard parsers
• cut the url from the character which violates the RFC or parsers algorithms and regexps

In any case it's better be written rather then being a mystery later on the RedWood journey.

I believe we would be able to close once we will have the answer and reference to the related GoLang libraries.

[andybalholm]

The Go url package accepts square brackets in URLs since Go 1.5:

https://go.googlesource.com/go/+/fced03a5c6dd22dd486106e3dd116510c28c6e4a https://github.com/golang/go/issues/5684

Before that, it was impossible to proxy a URL containing square brackets, and they do exist, even though the RFC doesn't allow them.

[elico]

@andybalholm Thanks, I was thinking it's only allowed for IPv6 hosts. Seems like GoLang took the relaxed approach to the subject.