No picture displayed (due to http accesses after login with https)

[guiyou65]

Each of my tablett and smartphone with Android 9 Pie are unable to show pictures with the latest v1.0.2. Folders are there. But no picture. I tried a standard user and a privileged one. No result.

Edit, ramack, Status: Some servers return strange URLs for the images depending on settings like original_url_protection and also if piwigo server is behind an reversproxy like nginx or haproxy

Solution: see comment from @EddyLB we need to:

• [ ] extract a port number if given by the user during login and store it separately in the account
• [ ] from URLs returned by the server remove protocol and port and replace it by what is stored in the account
• [ ] fix '&' (and maybe ' ') characters in URL received from Server by %.. URLencoding (happens e. g. if $conf['original_url_protection'] = 'images' or 'all' is set in the config of the server)

[bhlevca]

Same here, pictures do not show even for public albums. I do not think that permission is the issue. I tried with admin and guest. Same thing, no pictures show. It is not timing either, because I waited for days. Only albums, subalbum and number of images are shown

[ramack]

Thanks for reporting. I have heard now several times of that, but so far I cannot reproduce the issue. Would anyone of you mind sharing an URL to a piwigo setup showing the effect? - Feel free to send it by mail only to android@ pigigo org if you don't want to share it publicly.

[guiyou65]

Here are 2 sites. The first one https://www.guiyou.fr/piwigo doesn't show any picture. I own this site, on my own server. And the second one https://tugdualgaro.piwigo.com works well. It belongs to a friend of mine, on a piwigo server.

[ramack]

Thanks, that should be very good start for a first investigation. I'll come back in case I have further questions.

[ramack]

What is at least interesting is that on https://www.guiyou.fr - even though access via https, the thumbnail images have all a http (unencrypted) URL. But from another report is seems not affecting it.

[bhlevca]

I don't think that the https vs http is the issue behind not showing pictures. My site piwigo.hlxxxx.com has https everywhere and the android client does not show the pictures

[bhlevca]

From what I have seen so far all albums hosted on piwigo.com work and almost none of the private ones. Is this a pattern?

[ramack]

I am also not yet 100% convinced that http/https mix is the reason, as I could not yet reproduce it. But when I look what @bhlevca's site returns I see strange thumbnail URLS: https://piwigo.hlxxx.com:80/_data/i/upload/....jpg So yes, the protocol is giving https, but the port 80 doesn't fit to that and not even loading in the browser. (If I open the tn-URL delivered by the REST interface). So I believe that there is something in the piwigo-config that triggers the thumbnail URLs to be different. @plegall or @EddyLB did one of you see this before and can explain why it happens? Does the iOS app handle it in a specific way?

[jeff-amn]

I host my site on a private server and have never seen any issue with either http or https in displaying the images. You can use the app called Packet Capture to look at the ethernet traffic from the piwigo app to the host site. I also noticed that if I use my phone to look at my site in mobile mode some of the images do not load.

[bhlevca]

if it is of any help on "I see strange thumbnail URLS: https://piwigo.hlxxx.com:80"

This site is behind haproxy and the relevant config is below:

frontend http_proxy
         bind *:80
         bind *:443 ssl crt /etc/haproxy/certs/www.hlxxx.com.pem
         mode http
         option http-keep-alive
         http-request set-header X-SSL %[ssl_fc]
         reqadd X-Forwarded-Proto:\ https
         rspadd Strict-Transport-Security:\ max-age=15768000
         rspadd Content-Security-Policy:\ upgrade-insecure-requests

[ramack]

@guiyou65 do you also have a proxy in your piwigo setup?

[guiyou65]

No, actually there is no proxy, confirmed by my host provider www.online.net

[lergolas]

Hi. I have the same problem. When I set up my account over the lan, the thumbnails load in the app. But, when i setup it over my reverse proxy with Nginx (https), the thumbnails don´t load. With the other app piwigo-client from the Play Store, the thumbnails load over my proxy without problem.

The config of Nginx for this host is this:

` server { server_name xxxxxxx.duckdns.org;

The internal IP of the VM that hosts your Apache config

    set $upstream 192.168.10.10;

    location / {
            proxy_pass_header Authorization;
            proxy_pass http://$upstream;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_http_version 1.1;
            proxy_set_header Connection “”;
            proxy_buffering off;
            client_max_body_size 0;
            proxy_read_timeout 36000s;
            proxy_redirect off;
    }

} `

[mrufflesmcghie]

I'll have a play, seemed to be working fine for me... I've switched back to just local lan at the moment. When I'm home I'll pop my port forward back on and see what happens.

[ramack]

@plegall do you have any idea why the piwigo installation delivers thumbnail URLs with an explicit port setting or with http even though it is accessed via https? Is there any "base_url" setting?

[tvass]

Same error here.

From what i can see on my setup, when you configure a Piwigo account with HTTPS, the apps get a reply with a 301 (Moved Permanently) but does not follow the redirect.

[ramack]

Thanks for investigation. Do you happen to know which URL is accessed and to which it gets redirected? This would give us a more complete picture.

[tvass]

Hi @ramack Please find the details from NGINX (configured as a reverse proxy) with Let's Encrypt TLS.

It is OK when you configure an account with the full HTTPS URL. Requests are sent to the HTTPS vhost (server directive in NGINX).

==> /var/log/nginx/ssl.access.log <==
x.x.x.x - - [19/Nov/2019:13:39:41 +0000] "GET / HTTP/1.1" 200 2203 "-" "Dalvik/2.1.0 (Linux; U; Android 10; Pixel 2 Build/QP1A.191105.004)"
x.x.x.x - - [19/Nov/2019:13:39:41 +0000] "POST /ws.php?method=pwg.session.login&format=json HTTP/1.1" 200 68 "-" "Piwigo-Android 1.0.0"
x.x.x.x - - [19/Nov/2019:13:39:41 +0000] "GET /ws.php?method=pwg.session.getStatus&format=json HTTP/1.1" 200 346 "-" "Piwigo-Android 1.0.0"
x.x.x.x - - [19/Nov/2019:13:40:06 +0000] "GET /ws.php?method=pwg.categories.getImages&cat_id=0&format=json HTTP/1.1" 200 119 "-" "Piwigo-Android 1.0.0"
x.x.x.x - - [19/Nov/2019:13:40:06 +0000] "GET /ws.php?method=pwg.categories.getList&cat_id=0&thumbnail_size=medium&format=json HTTP/1.1" 200 492 "-" "Piwigo-Android 1.0.0"
x.x.x.x - - [19/Nov/2019:13:40:06 +0000] "POST /ws.php?method=pwg.session.login&format=json HTTP/1.1" 200 68 "-" "Piwigo-Android 1.0.0"
x.x.x.x - - [19/Nov/2019:13:40:06 +0000] "GET /ws.php?method=pwg.session.getStatus&format=json HTTP/1.1" 200 346 "-" "Piwigo-Android 1.0.0"

Then, when you start to browse albums, Thumbnails do not show because the Piwigo Android application requests HTTP URL and not HTTPS.

This is surprising as we configured an HTTPS base URL in account:

• It does NOT follow redirect sent back by NGINX reverse proxy (HTTP 301).
• It leaks covers direct JPG link over HTTP (with no S) which are supposed to be private.

==> /var/log/nginx/no_ssl.access.log <==
x.x.x.x - - [19/Nov/2019:13:44:20 +0000] "GET /_data/i/upload/2019/11/18/20191118132721-2dfa2902-me.jpg HTTP/1.1" 301 185 "-" "Dalvik/2.1.0 (Linux; U; Android 10; Pixel 2 Build/QP1A.191105.004)"
x.x.x.x - - [19/Nov/2019:13:44:20 +0000] "GET /_data/i/upload/2019/11/16/20191116144652-bf42f26a-me.jpg HTTP/1.1" 301 185 "-" "Dalvik/2.1.0 (Linux; U; Android 10; Pixel 2 Build/QP1A.191105.004)"
x.x.x.x - - [19/Nov/2019:13:44:20 +0000] "GET /_data/i/upload/2019/11/16/20191116143320-f25abf50-me.jpg HTTP/1.1" 301 185 "-" "Dalvik/2.1.0 (Linux; U; Android 10; Pixel 2 Build/QP1A.191105.004)"
x.x.x.x - - [19/Nov/2019:13:44:22 +0000] "GET /_data/i/upload/2019/11/16/20191116143320-f25abf50-me.jpg HTTP/1.1" 301 185 "-" "Dalvik/2.1.0 (Linux; U; Android 10; Pixel 2 Build/QP1A.191105.004)"
x.x.x.x - - [19/Nov/2019:13:44:22 +0000] "GET /_data/i/upload/2019/11/18/20191118132721-2dfa2902-me.jpg HTTP/1.1" 301 185 "-" "Dalvik/2.1.0 (Linux; U; Android 10; Pixel 2 Build/QP1A.191105.004)"
x.x.x.x - - [19/Nov/2019:13:44:22 +0000] "GET /_data/i/upload/2019/11/16/20191116144652-bf42f26a-me.jpg HTTP/1.1" 301 185 "-" "Dalvik/2.1.0 (Linux; U; Android 10; Pixel 2 Build/QP1A.191105.004)"

Thanks

[ramack]

Thanks for investigation. Actually the App tries to load the photos directly from the URL that is returned by the piwigo server. And there just takes whatever is in the URL, including the protocol or a port specification. @plegall can you tell us why piwigo server is returning a full URL and not just relative URLs in the WS interface? @eddyLB how does the iOS app handle this? Does it modify the protocol from what was returned by the piwigo web service? Because I agree with @tvass that we normally should not do http accesses in case the piwigo URL is using https.

[EddyLB]

Hi @ramack

The iOS app checks the validity of the URL provided manually by the user, removes any extra "/" or " " and stores separately the protocol "http(s)://" and the "host, port and path".

Then, as some servers return strange image URLs (other protocol, wrong port, wrong encoding, etc.), the app checks that the URL is RFC compliant and tries to guess it if not. Then, it keeps only the path and the query (protocol, host, port and path are replaced with those stored), checks/correct the encoding and finally concatenates the URL that will be used.

When $conf['original_url_protection'] = 'images' or 'all' is set in the config of the server, the URLs returned by the servers contain "&" instead of & (UTF8). So the app also replaces "&" with the proper char to fix that too.

[ramack]

ok thanks @EddyLB, then we have to do the same on Android.

[tvass]

@ramack Thanks for the help + high prio label. Let me know if i can help you with something.

[ramack]

I think the problem is quite clear and for the fix I have updated the issue text. If you have some programming skills you could help out implementing, otherwise I don't see any blocking point, so as soon as we find the time we'll implement it. And I think we'll do so for the next release.

[coolo]

At least in the case of https://www.guiyou.fr/piwigo you don't need any of this magic. The returned URLs are http but redirect to proper https and load fine.

[coolo]

IMO this is a regression of 972461ca42545e30ae3132bebd620bf9944638c0 - when readding the downloader handling redirects it works (on almost all images - for some strange reason the first image referenced uses a different URL scheme leading to 404).

picasso 2.5.2 is just old and buggy :(

[ramack]

but with 2.71828 - which is what we have currently in master it also works to follow the redirects without the downloader handling, cause it is included in picasso, isn't it?

But still I'd like to see this handling implemented as using http for the requests exposes information that would not be visible in clear text via https. In a sense I consider the server side setup broken and maybe the URLs in the REST output should be only relative to overcome that problem. - But today we have what we have and it would be good to support that setup in a secure manner.

[coolo]

No, master and 1.0.2 have picasso 2.5.2 without downloader: https://github.com/Piwigo/Piwigo-Android/blob/master/app/build.gradle#L97

Your local-repo branch uses 2.71 (which btw is also 378 commits behind master, picasso is just very bad in releasing :) and it works on this setup as well.

[ramack]

oops, as I was quite lonely the last days my branch gave me the impression of being master :grin: For picasso, yes. Last release is quite old and there have been rumors that e would be an abbreviation of end, so I am happy that they continued committing. Obviously there was some trouble with 2.7xx that I recently managed to solve or work around, so we'll come to the latest released version soon.

[GRBurst]

Hey, I don't think that the problem is in the app. I had the same issue - and fixed it by letting the app know that I am using https. For example, you can fix it by applying the mentioned changes in piwigo: https://github.com/Piwigo/Piwigo/issues/982

[bhlevca]

I don't think that the problem is in the app. I had the same issue - and fixed it by letting the app know that I am using https.

This is not a proof. The fact that it works in a browser it means that the app has an issue.

În addition, you did not say if you are behind a proxy.

I think that the app has a problem with communicating with the piwigo server behind a reverse proxy. The browser does not have this problem.

Thank you for your input.

[coolo]

there definitely is a problem in the used picasso version.

[GRBurst]

I am using a reverse proxy and I could resolve the issue as mentioned.

In my case, the problem with the app is that it does not follow redirects (301, permanently moved).

[GRBurst]

I am currently in holidays, so I did not spent to much information in debugging it.

In my opinion there are two issues, but only the redirect is related to the app.

If piwigo is running behind an reverse proxy as SSL termination point, it does not send the correct links and uses http instead of https. This should be fixed on the server side, not the app. This often produces mixed content, but I did not check whether this is the case on the website or not. Anyhow, the server should be able to produce correct links.

Happy to hear from you :-)

[coolo]

the problem is that the http url redirects to https - and then you're back to problem 1. Fixing your setup is just avoiding that problem - the setup would work with any client following redirects though, so it's not 'broken'.

[GRBurst]

@coolo yes correct, so both issues should be addressed because there are legitimate use-cases for redirects as well.

My point is that I just would not replace the URLs as mentioned in the issue description on the client side "from URLs returned by the server remove protocol and port and replace it by what is stored in the account".

So more accurately I am saying that this issue is not solely a problem of the client / app.

[ramack]

I do not fully agree. Even if the server (or proxy) returns a redirect, we should try to access the URL with https (if account configured to use https) even if a http-url was received, as http access would already expose secure data (session cookie but also already the information which image is viewed exposes personal data unencrypted)

Piwigo server should already return https URLs - but as piwigo doesn't know about the reverseproxy I am not sure whether it can do it. Omitting the protocol (and returning only relative paths) could be an option.

Additionally the app should follow redirects and potentially even update the URL (after asking?)

[coolo]

We could offer an option to rewrite http urls - but at this point I'd say there are more important issues at hand.

[p-baum]

nextcloud has an option that solves this exact problem:

'overwriteprotocol' => 'https'

I have traefik terminating ssl. I am still looking for a solution.

I'm also interested in what is more important than having a working android app for photos. Incompatibilty with reverse proxying should be clearly labeled. I just spent all day setting this up.

[coolo]

The bug is fixed in git

[tvass]

Is there an APK somewhere we could try ?

[coolo]

I can only offer an unsigned one apk as I don't have any keys. But I can look if there are some CI ways to create them, so you don't have to trust me

[coolo]

Well, travis-ci.org offers an upload to github releases, but it requires maintainer rights. So unless @ramack finds time to set this up, I can only offer a self built apk:

https://stephan.kulow.org/Piwigo-Android-df0b17604c90f1d3f8a669db0f949f9313e3251e.apk (you will get a warning that the key is not recognized and you will have to install the official playstore app before you can install [edit uninstall!] this, but you've been warned: it's just me and you don't know me)

And additionally notice that master has some regressions, but feel free to report what you see - but please note the version used.

[ramack]

Personally I am not a big fan of nightly builds but prefer clearly tagged versions. But it is also relatively easy to publish a beta version, so if we want we can release the current HEAD into the wild...

To the topic: for me the issue is not really solved by just following redirects and making the app show all pictures, as I consider it a security flaw to access the gallery via http - in case the server config delivers non-https URLs...

[coolo]

That's clearly a misconfig to be fixed on the server side and a web browser will get the same http URLs. Taking that the login happens through https, all we're leaking is the session cookie - which (by default) is bound to the IP address of the client.

[bhlevca]

https://stephan.kulow.org/Piwigo-Android-df0b17604c90f1d3f8a669db0f949f9313e3251e.apk (you will get a warning that the key is not recognized and you will have to install the official playstore app before you can install [edit uninstall!] this, but you've been warned: it's just me and you don't know me)

I tried, unfortunately does not work, I am getting javax.net.ssl.SSLHandshakeException and no pictures are shown

[coolo]

this seems to have little to do with the android app

[calmor15014]

I have Piwigo behind HAProxy using a Let's Encrypt certificate. The iOS app works normally, as does the website on both mobile (Android Chrome and iOS Safari/Chrome) and desktop (Firefox). HAProxy is set to force SSL to the visitor, but passes unencrypted between HAProxy and Piwigo, so Piwigo sees non-SSL calls to port 80. I have Nextcloud set up the same way, and both are hosted at home so I have full control over the configurations. I wouldn't want to break security though, so I haven't looked at the http header change options.

HAProxy is set to 301 any http:// to the https:// site.

The links visible by hovering over a photo in the desktop browser are https:// links without any port number indicated. However, I grabbed some of the GET calls to ws.php from apache.log and accessed them in Firefox directly, and see that all of the image references are all http:// links, including the main photo and every sub-version of size and thumbnail.

"thumb":{"url":"http:\/\/**server.url**\/_data\/i\/upload\/2019\/09\/02\/20190902215519-cd3be668-th.jpg","width":108,"height":144}

If I use any of those http:// links in Firefox directly, I can access the photos (even in private mode with no session login... which shouldn't be allowed but that's a different problem). But it does automatically redirect as HAProxy is requesting it to, switching the address to https:// but displaying the photo.

It seems that perhaps the Android app gets the 301 and fails out instead of handling the redirect? It does handle it correctly on the server setup page, where if I enter http://server.url and attempt to log in it automatically sets to https://server.url

[tvass]

Please read some previous comments. We already discussed this before.

If the HTTP/Piwigo server is configured to serve files over TLS and the app configured with an account https://, it should NOT try to reach unsecured ports. Of course, the app could follow redirect and it will work, but it introduces severe privacy/security flaws.

Btw, some people don't enable redirects to avoid security risks. In this case, the issue will remain.

I believe this issue requires some improvement on the Piwigo server code and this is why we are still stuck at the same point. On my side, I am now looking for a different solution :(

[bhlevca]

My setup is very similar to that of @calmor15014. I have no issues with HA proxy and Nextcloud and honestly I would have much more to fear for a privacy/security issues on that sided, where I store important files rather than in Piwigo where I mostly have public pictures. Therefore, I am not buying into that argument about privacy. Especially that the web solution works just fine.

[calmor15014]

@tvass - sorry, I did read through a bunch of stuff both here and on Piwigo's github issues but must have missed those specifics. I apologize for not RTFM in this case.

However, it seems odd that the Piwigo server code correctly gives out https:// links but sends http:// links via the API. Or, at least, that's what it seems, because the link actually leaves out the transport method and just passes relative URLs to /_data/... on the website, but absolute URLs to http://server.info/_data/... on the API.

HAProxy blog recommends allowing HAProxy to terminate the SSL connection and pass the request to the server unencrypted. This page explains how to configure it, and its benefits. From a network administration perspective, it's better for me to have pfSense manage the network stuff and let the containers manage the data side of things. The less exposure to the internet for the containers, the better, and fewer chances to screw up Apache configuration or whatever webserver the next "thing" I try to install uses that I'm unfamiliar with.

I can somewhat see why the app wouldn't want to acknowledge redirects from a potential hijacking perspective, but a redirect to just the https:// version of the same site should be permissible and plenty secure, and in fact occurs during the setup of the server access as I mentioned above.

If Piwigo served relative URLs in the API, and the app had to prepend the server URL from its config, that would probably fix the problem - if the app could handle relative URLs in the JSON response. But, on the app side, if it allowed redirecting to the same host as is in the config, or the same host that is in the API links only SSL instead, that should solve the security problem and also the proxy config issue at the same time.

Could solve both problems by detecting the server info in the URL, stripping it if it exists, and prepending what's in the app's config.