nginx broke after upgrade

[un-precedented]

my nginx is not coming up after the upgrade. I can tell something is up with the data volume path and the paths/mounts expected by the new nginx. I did not run the data volume container command again because my original volume container has data from an year ago, which is good data. I can log in to the ssh and see the config files, and the web/html files. But when I bash into the nginx, it's /data seems to be clean. The www folder is empty and the conf.d folder is empty too.

CONTAINER ID IMAGE COMMAND CREATED STATUS
17fa31eaa647 nazarpc/webserver:nginx "/webserver-entrypoin" 12 minutes ago Restarting (0) 28 seconds ago

[nazar-pc]

First of all, if you have original data-only container then your data are still there and can be recovered.

The original folders used in older images are now gone and everything is stored within /data directory, which is the only volume exposed by data image.

These lines provide conversion from older directories structure to the new one. So check if those files were actually moved under /data/nginx first. If they weren't moved properly, let me know what didn't work. Also take a look at logs (docker-compose logs -f nginx).

[un-precedented]

Yes, the change in directory structure seems to be the problem. Do you recommend creating a new data volume container first, and then re-run the docket compose with latest images? Will the init script automatically bring over older data? What's the command to backup the old data volume container ?

[nazar-pc]

OK, what I did for my old containers (now sure they were exactly as old as yours) is following: 1) Pull latest backup and restore images (they support both old and new structures without issues as they are generic tools) 2) Make backup of whatever you have currently using command in readme 3) docker-compose pull to grab new images 4) docker-compose up -d (at this point data should be converted, this is a good time to check if /data/*/* contain actual directories or old symlinks, if symlinks - take a look at logs to figure out why) 5) If everything went well on previous step then make backup of data-only container (remember, we have 2 backups already: before and after conversion) 6) Remove data-only container 7) Create data-only container with latest image 8) Restore data from latest backup (created after conversion) 9) If something was not moved properly you can move it now manually in new data-only image 10) docker-compose up -d

This is only necessary for this particular migration in order to get rid of unnecessary volumes exposed by data-only container (they do not hurt at all, but kind of unnecessary and I like to keep my system clean). In future this will be simpler because of 2 measures I did recently:

• only /data volume is exposed and used (there is single exception for SSH image)
• containers are now versioned, -v1 means you don't need to do something as lengthy as above until -v2 which will be explicitly obvious and will have proper documentation about how to do this.

Currently all you need is to look at /data/nginx and explore what is there, also you didn't post any logs yet.

[un-precedented]

I will try that today.

I also noticed I can no longer remote login into my SSH container. Probably private key error stemming from the directory mismatch. Once I create a new data container and copy/restore all the existing data, will it take care of my keys as well?

[nazar-pc]

Your private key is on your local machine. SSH container is looking for /data/.ssh/authorized_keys file with your public key, if you bring it back SSH should work fine (you can also define PUBLIC_KEY variable in `docker-compose.yml and this file will be created/updated for you, take a look at updated readme).

[un-precedented]

I don't remember how I got my private key from local machine into the previous SSH container, but I guess I can ignore that now, and simply copy the file again into a location that I reference in the docker compose script.

[nazar-pc]

Yes, but not private key (it should be kept secret), but public key, typically ~/.ssh/id_rsa.pub

[un-precedented]

The nginx is still not coming up. I think my data volume container's directory structure is different than what you've specified in the new webserver-entrypoint.sh.

I can tell my data container has the directories listed below mounted by doing the docker inspect command. I can also tell the /data/nginx/config is not a symlink as you're checking in the script.

            "Destination": "/usr/local/etc",
            "Destination": "/usr/share/nginx/html",
            "Destination": "/var/lib/consul",
            "Destination": "/data",
            "Destination": "/etc/consul.d",
            "Destination": "/etc/mysql",
            "Destination": "/etc/nginx",
            "Destination": "/etc/ssh",
            "Destination": "/var/lib/mysql",
            "Destination": "/var/log/mysql",
            "Destination": "/var/log/nginx",

[nazar-pc]

These are directories previously exposed by data-only container, it is all fine and correct. Current image only contains /etc/ssh and /data, but extra volumes shouldn't really hurt much.

What about finally posting some logs of Nginx container?

[un-precedented]

what's the location for log files of the nginx container?

[un-precedented]

I see this as the last message in the error.log

/data/nginx/log# tail error.log

2017/04/20 03:30:40 [emerg] 1#1: open() "/data/nginx/config/nginx.conf" failed (2: No such file or directory)

[un-precedented]

the config folder within /data/nginx is completely empty

[nazar-pc]

What about /etc/nginx? If latter contains configuration try to just move configs to the new location and update paths:

mv /etc/nginx/* /data/nginx/config/
sed -i 's/\/etc\/nginx/\/data\/nginx\/config/g' `find /data/nginx/config -type f`
sed -i 's/\/var\/log\/nginx/\/data\/nginx\/log/g' `find /data/nginx/config -type f`
sed -i 's/\/usr\/share\/nginx\/html/\/data\/nginx\/www/g' `find /data/nginx/config -type f`
echo -e "daemon off;\n$(cat /data/nginx/config/nginx.conf)" > /data/nginx/config/nginx.conf

Also check /data/www and its contents, if story is the same - then check /usr/share/nginx/html and if everything is fine there then just move website files to correct place:

mv /usr/share/nginx/html/* /data/nginx/www/
mv /usr/share/nginx/html/.* /data/nginx/www/

/data/nginx/log should be just an empty directory (or you can move old logs there is you want).

After files being moved to the right places Nginx should start and work fine.

[un-precedented]

/etc/nginx is empty /data/nginx/www <-- has website data, that I moved from /usr/share/nginx/html

It seems like my original nginx config directory got overwritten somehow. That's where all the config files were. I wonder why it didn't create new config files from scratch?

[nazar-pc]

Well, you configs should persist in the first place if data volume is mounted, I have no idea how that might happen while at the same time /data/nginx/www is not a symlink and still contains files.

Defaults are available in /etc/nginx_dist, you can copy them into the right place with this:

cp -a /etc/nginx_dist/* /data/nginx/config/

[un-precedented]

Just checked the backup of my data container that I made before today's exercise. It has the config files intact for nginx. I am going to copy them in, and then restart the nginx

data volume has been persistent all along; I suspect some directories got overwritten.

[nazar-pc]

Just make sure you always have working backup:) Let me know if it ends up well.

[un-precedented]

ok, brought back the config folder, and gave ownership to git

root@c572b064cede:/data/nginx# chown git:git -R config root@c572b064cede:/data/nginx# ls -l total 12 -rw-r--r-- 1 root root 0 Apr 10 2016 before_start.sh drwxrwxr-x 3 git git 4096 Apr 20 05:17 config drwxr-xr-x 2 git git 4096 Apr 20 04:17 log drwxr-xr-x 11 git git 4096 Apr 20 04:57 www root@c572b064cede:/data/nginx# ls config/ conf.d fastcgi_params koi-utf koi-win mime.types nginx.conf scgi_params uwsgi_params win-utf

[nazar-pc]

No need to give ownership, container is ready for such manipulations and will restore correct ownership upon restart.

Does container start and work now?

[un-precedented]

it's up, almost.

I cleaned up all my older images and stopped containers...except for my original data container. then started docker-compose from scratch. it was still failing one last thing I did was setting the environment using these commands sed -i 's/\/etc\/nginx/\/data\/nginx\/config/g' find /data/nginx/config -type f sed -i 's/\/var\/log\/nginx/\/data\/nginx\/log/g' find /data/nginx/config -type f sed -i 's/\/usr\/share\/nginx\/html/\/data\/nginx\/www/g' find /data/nginx/config -type f echo -e "daemon off;\n$(cat /data/nginx/config/nginx.conf)" > /data/nginx/config/nginx.conf

now the nginx is up, but my wordpress site is appearing appearing all crazy. seems like it is not able to read some directories correctly

[un-precedented]

Also, I did copy my local machine's public key to the /data/ssh/config, but I can't ftp into the ssh.

I first ftp'ed to my host, and then copied within the container. docker cp id_rsa.pub ssh_1:/data/ssh/config

[un-precedented]

any ideas on why ftp into the ssh container is not working? perhaps the git password needs to be set up?

[un-precedented]

also my websites are opening up, but the format is messed up. Seems like some config / meta-data files are not being properly processed.

[un-precedented]

I think there should be a synlink by the name of modules that didn't get copied over to /data/nginx/config

[un-precedented]

created the symlink for modules, but no success. looks like php or some other major module is not working.

[nazar-pc]

Also, I did copy my local machine's public key to the /data/ssh/config, but I can't ftp into the ssh.

Once again, you can just add your public key to docker-compose.yml, otherwise correct path is /data/.ssh/authorized_keys file (copy contents of you public key file into this file, create file if it is not there), not /data/ssh/config (latter is used for sshd configuration).

Website looks working now, but it was 9 hours since.

It is hard t help when you don't follow suggestions I make.

[un-precedented]

Ignore my last message because my browser was showing content from the cache. Nginx has been working fine after I created the symlink for modules in the config directory.

[nazar-pc]

Which modules are you talking about? There should be none, otherwise you have non-standard config which might fail again in future.

[un-precedented]

On ssh container access via ftp, looks like I need to create the file authorized_keys. Can I specify the key directly in the compose file or do I need to specify location of the key file?

[un-precedented]

Re: modules There was a symlink to modules directory in my original config directory. When I copied over everything from old config to new config directory, it didn't copy the symlink. That's why the Wordpress was coming up crazy in my web browser initially. After I created the symlink manually, it's been fine.

It's a standard config.

[nazar-pc]

Can I specify the key directly in the compose file or do I need to specify location of the key file?

Just look at readme, copy, paste, uncomment 2 lines, it will all just work. Should be something like:

    environment:
      PUBLIC_KEY: 'ssh-rsa AAAAB3....Q=='

After first SSH start you can remove it if you want.

When I copied over everything from old config to new config directory, it didn't copy the symlink. Is it in /data/nginx/config? Why do you need a symlink there?

[un-precedented]

I will send you the symlink that I am referring to when I get to my computer in some hours.

[un-precedented]

below is/was my previous config directory. When I copied it manually, the symlink to modules directory got left out. That's why my wordpress site was not properly loading. Perhaps the php or fastcgi module was not loading, and therefore the website was not showing fine. After I crated the symlink manually, it's been all well.

drwxr-xr-x 2 git git 4096 Apr 19 21:35 conf.d -rw-r--r-- 1 git git 1007 Apr 19 21:35 fastcgi_params -rw-r--r-- 1 git git 2837 Apr 19 21:35 koi-utf -rw-r--r-- 1 git git 2223 Apr 19 21:35 koi-win -rw-r--r-- 1 git git 3957 Apr 19 21:35 mime.types lrwxrwxrwx 1 git git 22 Apr 4 15:30 modules -> /usr/lib/nginx/modules -rw-r--r-- 1 git git 671 Apr 19 21:35 nginx.conf -rw-r--r-- 1 git git 636 Apr 19 21:35 scgi_params -rw-r--r-- 1 git git 664 Apr 19 21:35 uwsgi_params -rw-r--r-- 1 git git 3610 Apr 19 21:35 win-utf

[nazar-pc]

Well, modules symlink should have being absolute, in which case it doesn't matter where you move configs, it will always point to the right location. I'm wondering if in older versions packages config was different, but it is not the case right now, so I suppose we can ignore that issue now.

Can we close this issue as resolved?