search

Bungie-net / api

Resources for the Bungie.net API
Other
1.22k stars 92 forks source link

Special characters in GET requests #266

Closed jgayfer closed 6 years ago

jgayfer commented 6 years ago

I'm running into problems looking for accounts with special characters in their names.

Example Username: Grëëm#1623

https://www.bungie.net/Platform/Destiny2/SearchDestinyPlayer/4/Gr%C3%AB%C3%ABm%231623

I'm pretty sure I'm encoding this properly, but I may not be. Wondering if anyone here has any suggestions on how I can continue.

Note that this username is from one of my users, and haven't explicitly confirmed that he has played the game on PC yet, but I am 99.5% sure that he has.

vthornheart-bng commented 6 years ago

Hmm, interesting! Yeah, URL Encoding should have done the trick... very interesting.

Could you have them send you a link to their profile page on bungie.net, and then link it here? That will give us some info.

jgayfer commented 6 years ago

Here is the link: https://www.bungie.net/en/Profile/254/15187153/Greem

vthornheart-bng commented 6 years ago

Welp, this is going to be an interesting mystery, I'll have to chase down the root cause but for some reason his name has been encoded according to our search as:

Grëëm#1623

... ugh.

I don't know why yet, so I don't have a workaround I can provide aside from the very specific situation that for some reason umlats are being converted to ë... I'll have to get back to you on this once I talk to some folks over here.

(Of course it's an encoding issue... I just don't know where in our pipeline it's falling over. I'll have to do some digging, this may take a bit)

jgayfer commented 6 years ago

Interesting. Thanks for getting back to me promptly! Don't work too hard though, it's getting late here on the west coast ;)

vthornheart-bng commented 6 years ago

^_^ No prob! I had to hit up some folks to get more info about potential upstream causes, and I'm not sure when they'll reply... so I might not have more info for a bit unfortunately. But I'll update here once I get more info. In the meantime, the hacky thing that comes to my mind is to try and convert the incoming UTF-8 to Windows-1252, this is what I heavily suspect is happening somewhere in our pipeline, so doing that should result in you passing characters that it recognizes for the string matching. Yuck. If the problem turns out to be upstream of us, we may end up having to do the same thing to at least start returning responses for these folks. Hopefully it's not upstream... I need some time to investigate further on our side. Fun times... glad you reported this!

floatingatoll commented 6 years ago

That looks like UTF-8 bytes were not decoded prior to search encoding, such that the search is receiving raw byte strings of un-decoded UTF-8, rather than native UTF-8 character strings decoded from the raw bytes in the URI.

On Wed, Oct 25, 2017 at 5:54 PM, Vendal Thornheart <notifications@github.com

wrote:

^_^ No prob! I had to hit up some folks to get more info about potential upstream causes, and I'm not sure when they'll reply... so I might not have more info for a bit unfortunately. But I'll update here once I get more info. In the meantime, the hacky thing that comes to my mind is to try and convert the incoming UTF-8 to Windows-1252, this is what I heavily suspect is happening somewhere in our pipeline, so doing that should result in you passing characters that it recognizes for the string matching. Yuck. If the problem turns out to be upstream of us, we may end up having to do the same thing to at least start returning responses for these folks. Fun times... glad you reported this!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Bungie-net/api/issues/266#issuecomment-339517319, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFqDA4SxOGwgc3pJ9pTLygz438AxYEzks5sv9hRgaJpZM4QG1yJ .

bladefist commented 6 years ago

I'm seeing Chinese/Japanese and other languages that use unicode not working as well. I'm assuming this is the same issue?

Tetron-bng commented 6 years ago

The display names for Battle.net accounts getting stored in a corrupted state in our database and so search will not be able to find them. A fix is in the works but no ETA.

Tetron-bng commented 6 years ago

This issue has been resolved. The user's display name will continue to be garbled until they sign into the game.