URL characters encoding

[rkiyanchuk]

Hi!

I am using awesome httpie to talk to the REST API of YiDB tool which has unusual request URLs for querying that contain [, @ and ]. Examples follow:

GET repositories/cmsdb/branches/main/query/ApplicationService[@label="srp-app"]
GET repositories/cmsdb/branches/main/query/ServiceInstance[@domain =~ "ebay.*" and @activeManifestDiff = false]

The problem is: when I use http to query one of such URLs, I get 500 server error in response because these symbols ([, @, ]) are passed to the server unencoded. The request URL to server ends up being:

http://localhost:8080/cms/repositories/Asset/branches/main/query/Server[@name="test"]

Dirty solution: when I brutally use urllib to encode that part of the URL string inside httpie, it works great with the request URL ending up to be:

http://localhost:8080/cms/repositories/Asset/branches/main/query/Server%5B%40name%3D%22test%22%5D

Question is: What would be your recommendation to overcome this issue?

I have big doubts that this is necessary http's problem, but I would like to sort this out and be able to use the API with stock httpie. So is there any way I can form the desired URL with CLI arguments, or is it undoable without code patching?

If it's only through code change, I can prettify my changes and make a pull request to further discuss.

What are your thoughts on this?

Thank you!

[sigmavirus24]

[, ], and @ are all safe characters to have in the URI and according to my tests, we (requests) properly quote the URL. If you read RFC 3986, you'll see that a normalized (aka, quoted) URL has the path

/cms/repositories/Asset/branches/main/query/Server[@name=%22test%22]

If you're server is not accepting [, ], or @ in the path, that sounds like a problem with the server, not with httpie. I'm going to see what curl does but that should only inform httpie's decision on this bug. It won't influence requests's behaviour

[rkiyanchuk]

@sigmavirus24, thanks for the fast reply! Yeah, such behavior seemed strange to me as well, but I just wanted to make sure I'm using httpie correctly and so it's actually server's side problem.

Thanks for clarifying this.

[sigmavirus24]

@zoresvit further, you should verify but curl never even attempts to make a request over the network for me:

$ curl -i -v 'http://localhost:8080/cms/repositories/Asset/branches/main/query/Server[@name="test"]'
curl: (3) [globbing] bad range specification in column 73

curl thinks you're specifying some kind of range with [@ but n (i.e., name="test") isn't valid. So YiDB really shouldn't be using this syntax even if it is valid in a URI. If I percent encode the [ and ] it does actually attempt to talk to the network.

But yeah, httpie seems to happily send the request (as it should be) and the server then has an internal (500) error. So the server doesn't like the syntax it expects because it expects users to use clients that are overly eager in their percent-encoding of URIs.

It's up to httpie if it wants to go the extra mile here before handing the URL to requests, but as far as I'm concerned, we're doing it right. Sorry @zoresvit

[rkiyanchuk]

Yep, I've tried curl with the same results as well, you are right.

I'll look into their Python bindings and see if they do any additional magic there to make the server respond, but anyway it now seems clear it's not http's problem in forming the request, but in server interpreting it.

Thanks!

[rkiyanchuk]

@sigmavirus24 Just a follow-up regarding curl: it has some of its own URL globbing parser which can be turned off with -g parameter. So when used with -g it successfully forms the request URL just like httpie does — without encoding [, and @ — so it just additionally supports our conclusion that httpie does the right thing and YiDB is processing the incoming requests wrong.

[sigmavirus24]

@zoresvit to be clear (and hopefully not condescending) I speak only for the library on top of which @jakubroztocil built httpie. If the project/@jakubroztocil wants to add -G and -d that is their decision (as I said earlier). Regardless, they should not expect requests to provide any options, keyword arguments, or anything else to do this for them. This functionality would have to entirely live within httpie.