Doesn't work with non-latin letters

megabait1212 commented 5 years ago

Hi, the script doesn't work when an image URL contains non-latin letters. In my case it Polish letters ł,ś,ń and so on. Could give advice about how to fix such a problem? I won't be able to rename all images

willnorris commented 5 years ago

could you define "doesn't work"? Does it give an explicit error? Do you have a simple reproduction case? (I can easily set one up myself, but thought you might have something readily available)

megabait1212 commented 5 years ago

It doesn't resize the image. You can try by yourself just rename the image and name should contain any of these letters ł,ś,ń,ó. I found example https://zw.lt/assets/ostatni_dzwonek_szkoła_wilia_uczniowie-4059-1024x682.jpg script returns error 403. Or this one https://zw.lt/assets/Wielkanoc_Święconka_jajka_2019-9941-1024x683.jpg

willnorris commented 5 years ago

Are you using signatures? I've just tested these on my local machine, and with no signatures they are proxied (and resized) just fine. On my production instance (which requires signatures), it appears that the signature generation is failing, most likely choking on the non-latin characters. I'll dig into that more, but first wanted to confirm if you were seeing issues with or without signatures.

megabait1212 commented 5 years ago

I don't use signatures

willnorris commented 5 years ago

hmm, okay. Let's get a little more information then.

What version of imageproxy are you running? A specific version from a docker container, latest version of imageproxy from HEAD?
What flags are you passing to imageproxy on startup? You mentioned you got a 403, so I'm mostly interested in if you pass anything like allowHosts, denyHosts, or referrers
if you are passing flags, what happens if you pass no flags (aside from maybe addr if you need to)?
If you run with -verbose, can you provide the exact logs when you try to proxy one of the above images?

megabait1212 commented 5 years ago

I runned "./imageproxy -verbose" This with filename containing non-latin leters: 2019/05/11 20:22:35 fetching remote URL: https://zw.lt/assets/Wielkanoc_%C5%9Awi%C4%99conka_jajka_2019-9941-1024x683-1-1024x683.jpg 2019/05/11 20:22:36 error transforming image https://zw.lt/assets/Wielkanoc_%C5%9Awi%C4%99conka_jajka_2019-9941-1024x683-1-1024x683.jpg#280x140: image: unknown format 2019/05/11 20:22:36 request: {Method:GET URL:https://zw.lt/assets/Wielkanoc_%C5%9Awi%C4%99conka_jajka_2019-9941-1024x683-1-1024x683.jpg#280x140 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[willnorris/imageproxy] Accept:[image/]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:zw.lt Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr: RequestURI: TLS: Cancel: Response: ctx:} (served from cache: false) 2019/05/11 20:22:36 content-type not allowed: "text/html" This with normal filename: 2019/05/11 20:24:38 fetching remote URL: https://zw.lt/assets/compensa_cugowski-9999-1024x683.jpg 2019/05/11 20:24:38 request: {Method:GET URL:https://zw.lt/assets/compensa_cugowski-9999-1024x683.jpg#280x140 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[User-Agent:[willnorris/imageproxy] Accept:[image/]] Body: GetBody: ContentLength:0 TransferEncoding:[] Close:false Host:zw.lt Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr: RequestURI: TLS: Cancel: Response: ctx:} (served from cache: false)

megabait1212 commented 5 years ago

I installed by manual "go get willnorris.com/go/imageproxy/cmd/imageproxy"

megabait1212 commented 5 years ago

address default localhost:8080 but it behind nginx? I just re-read your questions and found that when I run only with a flag "-verbose" then I got error 502. And yes, before I used with flags allowHosts and referrers

willnorris commented 5 years ago

Okay, I can definitely reproduce this with running behind nginx. It appears to have to do with nginx encoding the non-latin characters. I'll play with it a bit more and see if I can figure out what to do.

willnorris commented 5 years ago

There's instruction at https://stackoverflow.com/a/37584637 for using proxy_pass without changing the encoding of the requested URL. Try that and see if it works for you. I'm having success with that in some cases, but not consistently, and then I'm still having various signature issues.

I'll keep plugging away, but wanted to point you at that and see if it solves your immediate issue.

willnorris commented 5 years ago

Actually, I suspect you were/are using the config mentioned in the README? That's long since no longer necessary for doing URL canonicalization, and that seesm to be what's causing the encoding issues. You could go with the much simpler:

location /api/imageproxy/ {
    proxy_pass http://localhost:4593/;
}

(You might need to use location ^~ /api/imageproxy/ depending on what other location directives you have and the precedence ordering)

Does that simpler directive work for you with these non-latin URLs? If so, I'll update the readme accordingly.

willnorris commented 5 years ago

not that you were having issues with signatures, but I'll record this here anyway... my signature issues during testing turned out to be entirely user error, so nothing to do with non-latin characters.

I think the only fix needed here is to simplify your nginx config as noted above.

megabait1212 commented 5 years ago

I have updated nginx config as you advised and it resolves all problems.

willnorris / imageproxy

Doesn't work with non-latin letters #178