Multi-site configuration question

[inspiredearth]

Preamble

I feel a bit out of place asking this on the "Issues" area, as it's not really an issue per se. However, a search online didn't turn up where there is active support / discussion taking place for debops-wordpress. If there's somewhere better I could post this question, please point me in that direction.

Domain structure for WP multisite ?

I deployed debops-wordpress multisite. The domains provided in /host_vars/vars.yml are in the form of : primarydomain.com, www.primarydomain.com, networkdomain1.com, www.networkdomain1.com, networkdomain2.com, www.networkdomain2.com, etc.

I have completed the setup on WP. Each site has been created. The wp-config.php looks fine. The nginx/domain.conf looks fine (at least the server part near the top has the correct domains in it).

The issue is that whilst the primary domain loads fine, all the other networked sites cause a ERR_EMPTY_RESPONSE error. I have checked over everything many times, and can't see why this is occurring.

[carlalexander]

Would it be possible to see what's in your vars.yml file?

[inspiredearth]

Hi Carl, thanks for chiming in. It's been a long week of trying to get this up and running. Help is greatly appreciated.

Here's /inventory/host_vars/www.PRIMARYDOMAIN.nz/vars.yml file...

# inventory/host_vars/www.PRIMARYDOMAIN.nz
wordpress__multisite: True
wordpress__multisite_subdomains: True
wordpress__nginx__server_name: [ 'www.PRIMARYDOMAIN.nz','PRIMARYDOMAIN.nz','NETWORK-DOMAIN.org.uk','www.NETWORK-DOMAIN.org.za','NETWORK-DOMAIN.org.za','NETWORK-DOMAIN.ca','www.NETWORK-DOMAIN.ca','www.NETWORK-DOMAIN.org.uk' ]
wordpress__admin_email: 'MYEMAIL@gmail.com'
wordpress__admin_username: 'USERNAME'
wordpress__title: 'SITE TITLE'

As far as I can tell, that results in a correctly configured nginx conf file. Here's the relevant snip from that...

server {

        listen 8080 default_server ipv6only=off;

        server_name www.PRIMARYDOMAIN.nz;
        server_name PRIMARYDOMAIN.nz;
        server_name NETWORK-DOMAIN.org.uk;
        server_name www.NETWORK-DOMAIN.org.za;
        server_name NETWORK-DOMAIN.org.za;
        server_name NETWORK-DOMAIN.ca;
        server_name www.NETWORK-DOMAIN.ca;
        server_name www.NETWORK-DOMAIN.org.uk;

        root /var/www/www.PRIMARYDOMAIN.nz;

        keepalive_timeout 60;

        access_log /var/log/nginx/www.PRIMARYDOMAIN.nz_access.log;
        error_log /var/log/nginx/www.PRIMARYDOMAIN.nz_error.log;
        index index.html index.htm index.php;

[inspiredearth]

My hunch is that err_empty_response is likely a networking related error, as opposed to an issue in the application or the stack. Although I could be wrong. Either way, I have no idea what else to try in order to fix it.

I have noted that when I make a request to any of the networked site domains, it generates no live activity in varnishlog, no entry in the Nginx access.log or www.primarydomain.nz_access.log. It would seem one new entry appears in varnish.www.primarydomain.nz_access.log when I make a request to a networked domain.

So, I gather from that, requests on those domains don't reach Nginx, and they only reach Varnish to the point that the initial request is logged, but no activity is generated by Varnish (at least none that shows up whilst the live varnishlog is running.

In which case perhaps it's not a network issue, because then I suspect Varnish wouldn't even log the access attempt. Although my knowledge of how all that works is too limited to make any serious theories or conclusions.

[inspiredearth]

@carlalexander : In case you get back around to assisting in troubleshooting this issue, I've posted a lot more data in a thread on Server Fault. You'll also see there is some discussion between myself and Drifter104 in which he is questioning the fact varnish appears to be listening on port tcp:6081 and tcp6:80, but not on tcp 80. So he was questioning how varnish is actually functioning at all. Perhaps you can shed some light on that? I've been trying to get this stack up and running for 10 days now. So if you have any chance to help resolve this, it would be greatly appreciated.

[carlalexander]

I've never tried to setup something like this. At first glance, your vars.yml does seem correct. The server fault details are useful as well so thanks for that!

A setup like your needs something like the domain mapping plugin to work. Do you have it installed? That's the first thing that came to mind when I saw this.

Everything else seems ok. But you did say that the request hit the varnish vhost, but not the backend one right? Do you see anything in the error logs?

[inspiredearth]

From what I can tell, since WP version 4.5, it doesn’t need a domain mapping plugin. It can handle the domain mapping without. Although, I took me a while to discover that, since most of the online guides all (out-dated) refer to needing such a plugin. So, I had in fact set it up with a plugin, and that’s the scenario in which the issue has been occurring. I’ve since removed the plugin (since the plug-in developer himself recommends not using it on 4.5+, as it’s redundant. The problem persists.

The only log that changes in any way when requesting a multisite domain is the one I mentioned. No others (that I know of) change. If there is a specific log you’d like me to check, please advise what the path is, so I can be sure we’re talking about the same logs.

On 18/07/2017, at 4:22 AM, Carl Alexander <notifications@github.com mailto:notifications@github.com> wrote:

I've never tried to setup something like this. At first glance, your vars.yml does seem correct. The server fault details are useful as well so thanks for that!

A setup like your needs something like the domain mapping plugin https://wordpress.org/plugins/wordpress-mu-domain-mapping/ to work. Do you have it installed? That's the first thing that came to mind when I saw this.

Everything else seems ok. But you did say that the request hit the varnish vhost, but not the backend one right? Do you see anything in the error logs?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/carlalexander/debops-wordpress/issues/173#issuecomment-315798836, or mute the thread https://github.com/notifications/unsubscribe-auth/AAovA2-I8URWK3iLv4FZfe1Xsvo0AGHNks5sO4otgaJpZM4OY4yE.

https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png https://github.com/carlalexander/debops-wordpress https://wordpress.org/plugins/wordpress-mu-domain-mapping/ https://github.com/carlalexander/debops-wordpress/issues/173#issuecomment-315798836

[inspiredearth]

In terms of setting up “something like this,” what aspect of this is outside the scope of setting up wordpress multisite as per the instructions you’ve provided? Even unmapped domains like site2.primarydomain.nz http://site2.primarydomain.nz/ have the same no response error. If you can help me understand where my particular configuration has strayed off the beaten path of Wordpress Multisite, I’ve love to know. Perhaps that’ll help me look for the right solutions. Hopefully. :-)

On 18/07/2017, at 4:22 AM, Carl Alexander <notifications@github.com mailto:notifications@github.com> wrote:

I've never tried to setup something like this. At first glance, your vars.yml does seem correct. The server fault details are useful as well so thanks for that!

A setup like your needs something like the domain mapping plugin https://wordpress.org/plugins/wordpress-mu-domain-mapping/ to work. Do you have it installed? That's the first thing that came to mind when I saw this.

Everything else seems ok. But you did say that the request hit the varnish vhost, but not the backend one right? Do you see anything in the error logs?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/carlalexander/debops-wordpress/issues/173#issuecomment-315798836, or mute the thread https://github.com/notifications/unsubscribe-auth/AAovA2-I8URWK3iLv4FZfe1Xsvo0AGHNks5sO4otgaJpZM4OY4yE.

https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png https://github.com/carlalexander/debops-wordpress https://wordpress.org/plugins/wordpress-mu-domain-mapping/ https://github.com/carlalexander/debops-wordpress/issues/173#issuecomment-315798836

[inspiredearth]

@carlalexander Would you please explain to me the purpose of there being two Nginx .conf files? One with the varnish prefix, and one without. Thanks.

Currently I've got all domains working, but only with Varnish out of the picture. So I am trying to figure out how to bring Varnish back in to listen on port 80, and pass requests to Nginx on 8080. Although I see that the backend port seems to be 6081, which is another mystery to me. Any light you could shed on all this would be greatly appreciated. If we manage to resolve this issue, the Docs can be updated with the necessary instructions for wordpress multisite, with custom domains, on Azure (if that makes any difference -- which I doubt at this stage).

[carlalexander]

The purpose of the two conf is because varnish doesn't do SSL termination. The varnish config file does SSL termination and then forwards the request to varnish. The second vhost listening to port 8080 is the one that hosts the WordPress site.

When I was asking for log details, I was asking for any of the error logs in /var/log/nginx. There's a good chance that they're all empty. You can also test to see if it's a varnish issue by turning off varnish by doing sudo service varnish stop. The upstream will failover to the WordPress vhost when it can't reach varnish.

[inspiredearth]

Okay. Thanks. At this point I've got as far as removing ipv6only=off from the various listen lines in the conf files. That got the sites loading for the first time, but witout varnish. When I do netstat -tulpn (even on the fresh deployment, it never shows varnish is listening on port 80. Are you in a position to help me get this up and running? I can even pay you for your time, if you'd prefer. My situation is that I offered to get this set up for a non-profit organisation, and it was meant to be up and running first thing last week. I now just need to get it sorted out, whatever it takes.

[carlalexander]

You also have two access log. One for varnish which starts with varnish_ and another for the WordPress backend. Are you seeing the request in the backend log?

[inspiredearth]

At this point I can't say, as I am in the middle of attempting to get Nginx and Varnish set up (in terms of their ports) in a way that matches what pretty much every howto suggests. Varnish listen on port 80, Nginx listen on port 8080. I come a bit unstuck on the fact that port 6081 comes into the picture so often in the debops-wordpress configuration. When the config was fresh, the only thing logged was what I already reported. Although, I am not aware of a "wordpress backend" log. Where do I find that?

[carlalexander]

Same place as the varnish log in /var/log/nginx. If you do ls -l /var/log/nginx, you should see two logs per domain.

[inspiredearth]

I see: -rw-r----- 1 www-data adm 0 Jul 16 06:25 access.log -rw-r----- 1 www-data adm 420 Jul 14 13:35 access.log.1 -rw-r----- 1 www-data adm 4367 Jul 18 10:30 error.log -rw-r----- 1 www-data adm 2649 Jul 17 23:50 error.log.1 -rw-r----- 1 www-data adm 230 Jul 16 11:03 error.log.2.gz -rw-r----- 1 www-data adm 683 Jul 16 01:28 error.log.3.gz -rw-r----- 1 www-data adm 49294 Jul 18 10:23 varnish.www.PRIMARYDOMAIN.nz_access.log -rw-r----- 1 www-data adm 213048 Jul 18 06:24 varnish.www.PRIMARYDOMAIN.nz_access.log.1 -rw-r----- 1 www-data adm 5469 Jul 17 03:37 varnish.www.PRIMARYDOMAIN.nz_access.log.2.gz -rw-r--r-- 1 root root 19951 Jul 16 04:47 varnish.www.PRIMARYDOMAIN.nz_access.log.3.gz -rw-r----- 1 www-data adm 0 Jul 18 06:25 varnish.www.PRIMARYDOMAIN.nz_error.log -rw-r----- 1 www-data adm 3822 Jul 17 23:15 varnish.www.PRIMARYDOMAIN.nz_error.log.1 -rw-r--r-- 1 www-data root 222 Jul 17 02:32 varnish.www.PRIMARYDOMAIN.nz_error.log.2.gz -rw-r----- 1 www-data adm 28421 Jul 18 10:23 www.PRIMARYDOMAIN.nz_access.log -rw-r----- 1 www-data adm 99542 Jul 18 05:32 www.PRIMARYDOMAIN.nz_access.log.1 -rw-r----- 1 www-data adm 567 Jul 17 02:32 www.PRIMARYDOMAIN.nz_access.log.2.gz -rw-r--r-- 1 root root 4787 Jul 16 01:38 www.PRIMARYDOMAIN.nz_access.log.3.gz -rw-r--r-- 1 www-data root 0 Jul 14 13:37 www.PRIMARYDOMAIN.nz_error.log

[inspiredearth]

So is ...log and ...log.1 the "two per domain" you are referring to? Or is it the varnish.www.domain... and www.domain... you are referring to as "two per domain" ?

[carlalexander]

www.PRIMARYDOMAIN.nz_access.log is your backend log. varnish.www.PRIMARYDOMAIN.nz_access.log is your varnish log. When you make a request, you should see it appear on both. Based on the timestamps, it does look like requests are making it to the backend.

[carlalexander]

Both error logs are empty so there are no errors. You want to look at www.PRIMARYDOMAIN.nz_access.log and see what nginx is saying about those. If you're still getting 204 errors there, the issue is either with PHP or WordPress itself.

[inspiredearth]

As mentioned, from the start, I have been able to make requests to the primary domain. That works fine, and is logged in both logs. But, on the default configuration, no requests for the multi-side domains got through. They would result in one line going into varnish...

[carlalexander]

Can you DM me or share your actual domain names instead of placeholders? I can't test routing issues without it. All I have is your ip.

[inspiredearth]

Yep. Happy to. Only thing is, in the past couple of years of using github, I've never been able to figure out how to direct message other users. :-)

[carlalexander]

Just DM me on twitter. @twigpress

[inspiredearth]

Please keep in mind, the way I now have the ports set up is nothing like the default deployment. Currently, I've made changes, which have allowed ALL sites to load, but varnish seems to have vanished out of the game.

[inspiredearth]

Well, at least I think Varnish is not working. I have been using http://www.isvarnishworking.com/ to test for that. But I now see that requests are firing off a lot of activity on varnishlog

[carlalexander]

I'm seeing varnish header, but only misses.

[carlalexander]

I don't use the X-Varnish header so that site won't work. You're looking for X-Cache and X-Cache-Hits.

[inspiredearth]

Yes, I see your requests coming through on varnishlog and varnishhist. Do if they are misses, what does that suggest? Currently I've changed the ports so that ALL nginx config files are telling it to listen on 8080. And ALL varnish config files (there are various files tucked away) are set to listen on port 80. Only when i did that did netstat -tulnp start showing that Varnish is listening on 80, and Nginx is listening on 8080.

I don't use the X-Varnish header so that site won't work. You're looking for X-Cache and X-Cache-Hits.

Okay. That's super good to know! 👍

[inspiredearth]

I just made a request to the same domain twice, and the second time around varnishlog showed:

-   RespHeader     X-Cache: HIT
-   RespHeader     X-Cache-Hits: 1

The first request produced,

-   RespHeader     X-Cache: MISS
-   RespHeader     X-Cache-Hits: 0

Does that mean Varnish is in fact working?

[carlalexander]

Yes! Actually, the mistake was on me. I had "Disable cache" on in Chrome Dev Tools.

[carlalexander]

I'm seeing Varnish working right now

[inspiredearth]

Third refresh I again saw MISS. Fourth and fifth time, I got HIT.

[inspiredearth]

Okay. I seem to be consistently getting cache HIT now. This is major progress. Now, the only thing is, I have made quite a few changes to the configuration, which will all get obliterated if/when I refresh the stack through debops-wordpress. Is there a way I can add custom settings to the debops-wordpress .yml (or in whatever config file) so that the next time it deploys, it will use those changes?

It would also be interesting to know (perhaps for you too) why those changes were necessary, and whether there's a better way to go about it (in terms of a better way to customise it, as opposed to the rather hit and miss way I've gone about it).

[carlalexander]

The cache is only configured for 1 minute for WordPress pages. You should see higher numbers for assets. Also keep in mind that we're using cache header so assets might be coming from memory which means that the number won't be accurate.

[carlalexander]

In general, I would take those numbers as just general guides. If they're changing, Varnish is doing its job. But it's not an accurate way to track hits anymore due to Varnish changes in the recent version.

[inspiredearth]

Okay. Thanks. That's good to know. Is it necessary to have it as short as 1 minute? What's the reasoning behind that?

[inspiredearth]

Is there a way I can add custom settings to the debops-wordpress .yml (or in whatever config file) so that the next time it deploys, it will use those changes?

[carlalexander]

In general, I wouldn't run debops wordpress like that once you configured a server. You should try to use tags if you can. Some of the wiki articles tell you which tags to use.

But I get what you mean, I'm honestly not sure what you changed and what actually fixed the issue either. So it's hard to say what you should do or what you should to prevent those changes from being reverted.

I'll try to replicate the setup that you're trying to do in the coming weeks and add some wiki for it.

[carlalexander]

For Varnish, it's a tradeoff behind performance and perceived responsiveness. 1 minute cache can handle most loads and if for some reason a page isn't updating, it will be invalid in a minute.

[inspiredearth]

I am compiling a list of the changes I made. Will have it ready in a couple of minutes.

[carlalexander]

I think it's a real use case (with the using this with Azure). I just need to document it.

[inspiredearth]

Here's all the changes I made:

$ /etc/default/varnish

    DAEMON_OPTS="-a :80 \
                 -T localhost:6082 \
                 -f /etc/varnish/default.vcl \
                 -S /etc/varnish/secret \
                 -s malloc,256m"

$ /etc/nginx/sites-available/default

    server {
        listen 8080;
    #   listen [::]:80 default_server;

$ /etc/varnish/default.vcl

    backend wordpress {
      .host = "127.0.0.1";
      .port = "8080";

$ /etc/nginx/sites-available/www.PRIMARYDOMAIN.nz.conf
    server {

          listen 8080 default_server;
          server_name www.DOMAIN.nz;

$ /etc/nginx/sites-available/varnish.www.PRIMARYDOMAIN.nz.conf
    server {
            listen 8080;

            server_name www.DOMAIN.nz;

$ /etc/systemd/system/varnish.service
    ExecStart=/usr/sbin/varnishd -j unix,user=vcache -P /run/varnishd.pid -F -a *:80 -T localhost:6082 -t 120 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,1715.0

Removing ipv6only=off was what got the multi-side domains loading. I read somewhere (lost track of where) that this is a glitch in Nginx, that that has to be removed for Wordpress MS to work with custom domains.

Then, the other change was to basically change all settings for Nginx to listen to port 8080, and all for Varnish to listen on 80.

[UPDATED]

[inspiredearth]

One concern I have is whether or not this will screw things up for SSL (443) when I implement it.

Also, it would be great to know what changes I can make on the debops-wordpress side to make those changes on the Ansible controller. That way if I ever do need to redeploy it, I safely can, AND if I need to set up a new server, I can have those settings handled on the Ansible controller side.

[inspiredearth]

With those changes (some of which may be unnecessary), the following is what's listening to the various ports:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1972/mysqld
tcp        0      0 127.0.0.1:29131         0.0.0.0:*               LISTEN      2796/mdsd
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      1545/redis-server l
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1372/varnishd
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      1518/nginx -g daemo
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1449/sshd
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      2672/master
tcp        0      0 127.0.0.1:6082          0.0.0.0:*               LISTEN      1372/varnishd
tcp6       0      0 :::80                   :::*                    LISTEN      1372/varnishd
tcp6       0      0 :::22                   :::*                    LISTEN      1449/sshd
tcp6       0      0 :::25                   :::*                    LISTEN      2672/master
udp        0      0 0.0.0.0:68              0.0.0.0:*                           1010/dhclient

I am testing which of the posted changes are in fact not required. I'll update you when I have figured that out.

[carlalexander]

Yup, that's what I want to document. (The ansible controller options.) As for SSL, this is also going to be tricky. I don't think the project can handle SSL for such a setup as it stands. You're going to need multiple certificates which I think means multiple vhosts. I would hold off doing that or do it by hand if you want to do it.

[carlalexander]

Based on your changes, it seems all you did is go to varnish directly vs through the nginx vhost. I would point out that both the varnish vhost and backend vhost are identical. This means that nginx is picking one of the two at random. It just happened to pick the backend one which is a coincidence so keep that in mind.

[carlalexander]

Removing ipv6only=off was what got the multi-side domains loading. I read somewhere (lost track of where) that this is a glitch in Nginx, that that has to be removed for Wordpress MS to work with custom domains.

This is good to know. Will run tests on that.

[carlalexander]

I think I should be able to figure out a configuration that should work based on your changes. That said, I'm not sure for the SSL stuff. That's a change that's a lot more complex and of larger scope.

[carlalexander]

You might be able to do it by using a variation of this wiki, but I'm really not sure. I'll need to do a lot of testing for that.

[inspiredearth]

I am not sure what you mean by "is picking one of the two at random".

For some reasons, unknown to me, when accessng varnish by "going through the nginx vhost" as opposed to directly, nothing would get through to Nginx or Varnish on the multi-side domains. Only the primary domain functions as expected. So there's something about going through the nginx vhost that prevented it from working. I wonder if that's something that can be corrected, so that it is again possible to go through the nginx vhost, and therefore leave the port settings as they were by default.

[carlalexander]

Whenever nginx receives a request, it tries to match the domain name and port to a vhost. Since both vhosts listen to the same domain and port, nginx is picking the first one it sees.

I wonder if that's something that can be corrected, so that it is again possible to go through the nginx vhost, and therefore leave the port settings as they were by default.

Yes, I think so! Just need to run some tests on my end.

[inspiredearth]

Well, I am very pleased to have figured out how to get this working. Thanks for your help. It was kinda wild flying blind for days on end, but I've learnt a lot in the process. This is the first time I've set up a hosting server, and messed around with it. I have a load of IT experience in other areas (over 28 years of it, professionally) so that at least gave me the wherewithal to figure things out as I went along. But it was a long-haul. I don't think I've ever spent anywhere near 10+ days trying to figure out how to get something going on a server or computer or IT environment before. What a marathon!

If you can figure out how to incorporate changes into the debops-wordpress project to make this work by default, that would be great. If you can also figure out how to get SSL to work, that would be even greater.

If it would help to have some money donated toward your time on the SSL side of things, that's something I might be able to contribute toward. In this day and age, I think it's pretty vital to be able to implement SSL/TLS when required.