A crawler gets a valid session value if switching from secured to unsecured pages

ADDISON74 commented 8 years ago

Here is my server configuration in a VM test machine (Debian 7.9 x64, Magento CE 1.9.2.2 + Default sample pack, Turpentine the latest devel version):

Pound (IP: 192.168.159.102, Ports: 80 and 443 SSL-Offloading) => Varnish (IP: 192.168.159.102, Port: 8090) => Apache (IP: 192.168.159.102, Port 8080)

I am not using full secured frontend, just only for My Account, Checkout pages. Here is my configuration in backend (please note there are no other changes except in disabling Magento default cache in Cache Management and basic Turpentine options):

System -> Configuration -> Web -> Unsecure -> Base URL = http://www.mydomain.com/
System -> Configuration -> Web -> Secure -> Base URL = https://www.mydomain.com/
System -> Configuration -> Web -> Use Secure URLs in Frontend = Yes

(see difference between unsecure/secure values).

Let's set up in Turpentine Crawler IP value to 192.168.159.102 in order to consider this IP address a crawler and bypassing cookies for all visitors.

crawler_ip

Here is first time visiting the Home Page (all cookies and browser cache cleared before). There are no cookies after loading the page. It is a crawler-session. In this image you don't see crawler_session cookie because I cleared this cookie after visiting the page.

home_page

Next we visit any category, in my case Accessories. Also there are no cookies after loading this page.

category_accessories

Now let's visit "My Account" which is a secured page. Now we get 3 cookies: adminhtml, frontend, frontend_cid. Look how frontend cookie gets an alphanumeric value, changing it from crawler_session.

secured_link

Let's go back to a previously visited category, click on Accessories link. This page is unsecured. 2 cookies were deleted: adminhtml and frontend_cid. We still have frontend cookie with an alphanumeric value, not "crawler_session" and this is a big issue. From now on any page visit of this crawler it is considered a usual visitor.

category_accesories_after

In conclusion: You should change your code for this scenario to avoid cookie creation/change in secured pages if it is a "crawler_session". Switching from secured to unsecured pages should not change initial value of frontend cookie!.

aricwatson commented 8 years ago

I've just tested a bit and was unable to replicate this on a website that's not using Pound/SSL - so it seems like this issue is related to that scenario. I don't have a test site set up with Pound/SSL at the moment to test with. I will look into getting this set up, since we're only going to see more and more problems related to all SSL sites.

ADDISON74 commented 8 years ago

You are right this issue is not present in Varnish/Apache configuration. Once you need a secured website with a SSL certificate you have to install Pound in front of Varnish.

Let's speed up the things. I will post for you all configurations needed for that set up (Pound, Apache, Magento). Hope this helps you.

ADDISON74 commented 8 years ago

Here is the configuration we will set up bellow:

Pound (IP: YOUR_SERVER_IP, Ports: 80 and 443 with SSL-Offloading) => Varnish (IP: YOUR_SERVER_IP, Port: 8090) => Apache (IP: YOUR_SERVER_IP, Port 8080)

ADDISON74 commented 8 years ago

POUND CONFIGURATION

1) Install Pound "apt-get install pound" for Debian. Edit the configuration file /etc/pound/pound.cfg.

Replace bellow YOUR_SERVER_IP, YOUR_SSL_CERTIFICATE. For creating PEM file issue your self-signed certificate and concatenate this certificate with the private key in one file with pem extension.

## Minimal sample pound.cfg
##
## see pound(8) for details
######################################################################
## global options:

User            "www-data"
Group           "www-data"
#RootJail       "/chroot/pound"

## Logging: (goes to syslog by default)
##      0       no logging
##      1       normal
##      2       extended
##      3       Apache-style (common log format)
LogLevel        1

## check backend every X secs:
Alive           30

## use hardware-accelleration card supported by openssl(1):
#SSLEngine      "<hw>"

# poundctl control socket
Control "/var/run/pound/poundctl.socket"
######################################################################

TimeOut 3600

ListenHTTP
    Address YOUR_SERVER_IP
    Port 80

    ##  If  1  force Pound to change the Location: and Content-location:
    ##  headers in responses. If they point to the back-end itself or to
    ##  the  listener (but with the wrong protocol) the response will be
    ##  changed to show the virtual host  in  the  request.  Default:  1
    ##  (active).  If the value is set to 2 only the back-end address is
    ##  compared; this is useful for redirecting a request to  an  HTTPS
    ##  listener on the same server as the HTTP listener.
    RewriteLocation 0

    ##  If  1 force Pound to change the Destination: header in requests.
    ##  The header is changed to point to the back-end itself  with  the
    ##  correct protocol. Default: 0.
    RewriteDestination 1

    ## allow PUT and DELETE also (by default only GET, POST and HEAD)?:
    #xHTTP 0
    xHTTP 2

    Service
        BackEnd
             Address YOUR_SERVER_IP
             Port 8090
        End
    End
End

ListenHTTPS
    Address YOUR_SERVER_IP
    Port 443
    Cert "YOUR_SSL_CERTIFICATE.pem"
    Ciphers "EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-GCM-SHA256:AES256+EECDH:DHE-RSA-AES128-GCM-SHA256:$

    ##  If  1  force Pound to change the Location: and Content-location:
    ##  headers in responses. If they point to the back-end itself or to
    ##  the  listener (but with the wrong protocol) the response will be
    ##  changed to show the virtual host  in  the  request.  Default:  1
    ##  (active).  If the value is set to 2 only the back-end address is
    ##  compared; this is useful for redirecting a request to  an  HTTPS
    ##  listener on the same server as the HTTP listener.
    RewriteLocation 2

    ##  If  1 force Pound to change the Destination: header in requests.
    ##  The header is changed to point to the back-end itself  with  the
    ##  correct protocol. Default: 0.
    RewriteDestination 1

    ##  Behave  like an HTTP/1.0 server for HTTPS clients. If this value
    ##  is 0 disable the check. If the value is 1 do not allow  multiple
    ##  requests on SSL connections. If the value is 2 (default) disable
    ##  multiple requests on SSL  connections  only  for  MSIE  clients.
    #NoHTTPS11 0

    AddHeader "Ssl-Offloaded: on"

    ## allow PUT and DELETE also (by default only GET, POST and HEAD)?:
    #xHTTP 0
    xHTTP 2

    Service
        BackEnd
            Address YOUR_SERVER_IP
            Port 8090
        End
    End
End

2) Start or restart Pound service.

ADDISON74 commented 8 years ago

APACHE CONFIGURATION

1) Disable ssl module: a2dismod ssl.

2) Comment these lines in /etc/apache2/apache2.conf:

#SSLProtocol ALL -SSLv2 -SSLv3
#SSLCipherSuite ECDHE-RSA-AES256-SHA384:AES256-SHA256:AES256-SHA256:RC4:HIGH:MEDIUM:+TLSv1:+TLSv1.1:+TLSv1.2:!MD5:!ADH:!aNULL:!eNULL:!NULL:!DH:!ADH:!EDH:!AESGCM

3) Edit .htaccess in Magento root and add at the end of the file these directives:

SetEnvIf Ssl-Offloaded on HTTPS=on
SetEnvIf X-Forwarded-For "^(\d{1,3}+\.\d{1,3}+\.\d{1,3}+\.\d{1,3}+).*" XFFCLIENTIP=$1

I am not using mod_rpaf or mod_remoteip. Second line above is for extracting the first position in X-Forwareded-For. I am using that variable XFFCLIENTIP in file /etc/apache2/apache2.conf in LogFormat directive:

LogFormat "%{XFFCLIENTIP}e %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

In this way I am getting in my Apache logs visitor IP address.

4) Listen port in /etc/apache2/ports.conf must be 8080. Also check your virtual host files in /etc/apache2/sites-available/ for:

<VirtualHost YOUR_IP_SERVER:LISTEN_PORT>

5) Restart Apache service.

ADDISON74 commented 8 years ago

VARNISH 4 CONFIGURATION

1) Edit /etc/default/varnish file and replace YOUR_SERVER_IP:

DAEMON_OPTS="-a YOUR_SERVER_IP:8090 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -p feature=+esi_ignore_other_elements \
             -p vcc_allow_inline_c=on \
             -p cli_buffer=16384 \
             -s malloc,1G"

2) Restart Varnish service.

ADDISON74 commented 8 years ago

MAGENTO CONFIGURATION

1) For Turpentine: System -> Configuration -> Turpentine -> Caching Options

Backend Host: YOUR_SERVER_IP
Backend Port: 8080

2) For SSL: System -> Configuration -> General -> Web

Unsecure

Base URL: http://YOUR_SERVER_IP/

Secure

Base URL: https://YOUR_SERVER_IP/
Use Secure URLs in Frontend: Yes
Use Secure URLs in Admin: Yes
Offloader header: HTTP_SSL_OFFLOADED

ADDISON74 commented 8 years ago

Bonus: Using Pound for listening ports 80 and 443 in front of Varnish gives you flexibility in skipping Varnish when you need it. Just edit in Pound configuration file those two BackEnd's switching from Varnish port (8090) to Apache port (8080) and restart Pound service. Your Magento store will continue working just perfect for both protocols, but without Varnish.

nexcess / magento-turpentine

A crawler gets a valid session value if switching from secured to unsecured pages #1052