Issue - nginx: [emerg] host not found in upstream

[kanenses]

Hi @spurin My kid and I have sarting learning at packtpub, but getting a few "issues"

Log from Portal:

[emerg] 1#1: host not found in upstream "ubuntu1" in /etc/nginx/conf.d/default.conf:45
nginx: [emerg] host not found in upstream "ubuntu1" in /etc/nginx/conf.d/default.conf:45

And none of the ubuntu containers, starting, with the log error:

No log line matching the '' filter

Running lab at Proxmox, Ubuntu container.

Please, can you give a help? thank you

[spurin]

Hi @kanenses

Great to hear from you and kudos for supporting your prodigy!

This environment is certainly a new one for me but never less, let's see what we can do. I also have some alternative back up options but we'll explore this first.

Firstly could you please tell me more about the environment in which this is running. You mentioned that the hypervisor is Proxmox, what is the virtual machine that you're using running (*edit/update: I see you mention that it is Ubuntu, what version please, also share the full log from when you run the docker-compose up) ?

Thanks

James

[bsahane]

Facing Same Issue.. nginx: [emerg] host not found in upstream "ubuntu1" in /etc/nginx/conf.d/default.conf:45 After execute command docker-compose up

$ docker-compose up
Creating ubuntu2  ... done
Creating centos3  ... done
Creating centos1  ... done
Creating ubuntu3  ... done
Creating ubuntu-c ... done
Creating centos2  ... done
Creating ubuntu1  ... done
Creating docker   ... done
Creating portal   ... done
Attaching to ubuntu3, ubuntu2, ubuntu-c, docker, centos3, ubuntu1, centos2, centos1, portal
ubuntu2 exited with code 255
ubuntu-c exited with code 255
ubuntu3 exited with code 255
ubuntu1 exited with code 255
portal      | 2021/11/14 14:20:36 [emerg] 1#1: host not found in upstream "ubuntu1" in /etc/nginx/conf.d/default.conf:45
portal      | nginx: [emerg] host not found in upstream "ubuntu1" in /etc/nginx/conf.d/default.conf:45
portal exited with code 1

---

OS Details: Fedora Linux 35 (Workstation Edition) Docker Version:

$ docker --version
Docker version 20.10.10, build b485636

Docker-Compose Version:

$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c

[spurin]

@bsahane

Am really glad you've contacted me about this and correspondingly, your feedback may also help @kanenses

Fedora 35 in your case is a bleeding edge distribution and subsequently, as you may know it uses newer versions of components. Traditionally, many distro's of Linux use cgroups v1 whereas Fedora has now switched to cgroups v2.

Historically, the fix on the course for this issue was to download cgroups v2 back to v1, reboot, remove the lab and then start it back up. I have however, been working on a brand new version of the lab which in my testing is working well for both cgroups v1 and v2.

Would you please take this opportunity to test this out and see if it fixes it for your case. Here's the commands I'd like you to run, assuming that you used git originally to clone the repository -

docker-compose rm
git reset --hard
git fetch -v
git checkout release-candidate
git pull
docker-compose pull
docker-compose up

Just to note, we do the reset above as I want it also to discard the changes you have made to the .env file. I've found in my testing that Docker has caught up on some of the historic issues with volume mounting and alas, going forward updating this file may be an unnecessary step.

Please let me know how you get on.

Thanks

James Spurin

[spurin]

@bsahane I just tried it out successfully on Fedora 34 and all is working well on that, if I don't hear from you before I'll setup a Fedora 35 for testing. It would be very helpful for me when reporting back on your testing if you can share the following output (run as root) -

docker --version
docker-compose --version
/usr/lib/systemd/systemd --version
mkdir /tmp/test; mount -t cgroup2 none /tmp/test; ls /tmp/test

@kanenses - The version of Ubuntu that you installed, was it 21.10 by any chance? This I believe also uses cgroups v2 and the steps I've shared for @bsahane should also work for you. I am also setting up a 21.10 instance for testing.

[spurin]

Have just completed testing of the release candidate on those mentioned above and it's working well. For reference I tested with the following -

Distribution	Docker Version	Docker Compose Version	SystemD Version	Cgroups	Lab Starts	Lab Restarts	Login Successful
Fedora 34	20.10.7	1.28.6	248	v2	✔️	✔️	✔️
Fedora 35	20.10.9	1.29.2	249	v2	✔️	✔️	✔️
Ubuntu 21.10	20.10.8	1.25.5	248-3-1ubuntu8	v2	✔️	✔️	✔️

Please let me know how you both get on - @kanenses @bsahane

[bsahane]

@spurin Thank You for providing update.

Your Provided Commands are working perfectly. and Now I can access http://localhost:1000/ portal.

Please find output for your reference..

~/diveintoansible-lab $ docker-compose rm
git reset --hard
git fetch -v
git checkout release-candidate
git pull
docker-compose pull
docker-compose up
Going to remove portal, centos2, docker, ubuntu2, ubuntu3, ubuntu1, centos1, ubuntu-c, centos3
Are you sure? [yN] y
Removing portal   ... done
Removing centos2  ... done
Removing docker   ... done
Removing ubuntu2  ... done
Removing ubuntu3  ... done
Removing ubuntu1  ... done
Removing centos1  ... done
Removing ubuntu-c ... done
Removing centos3  ... done
HEAD is now at 67f5d84 Update README.md
POST git-upload-pack (155 bytes)
From https://github.com/spurin/diveintoansible-lab
 = [up to date]      master             -> origin/master
 = [up to date]      cloudshell-gcp     -> origin/cloudshell-gcp
 = [up to date]      nonpersistent_home -> origin/nonpersistent_home
 = [up to date]      release-candidate  -> origin/release-candidate
 = [up to date]      tests              -> origin/tests
Branch 'release-candidate' set up to track remote branch 'release-candidate' from 'origin'.
Switched to a new branch 'release-candidate'
Already up to date.
Pulling ubuntu-c ... done
Pulling ubuntu1  ... done
Pulling ubuntu2  ... done
Pulling ubuntu3  ... done
Pulling centos1  ... done
Pulling centos2  ... done
Pulling centos3  ... done
Pulling docker   ... done
Pulling portal   ... done
Creating ubuntu3  ... done
Creating centos2  ... done
Creating centos3  ... done
Creating ubuntu2  ... done
Creating centos1  ... done
Creating ubuntu1  ... done
Creating docker   ... done
Creating ubuntu-c ... done
Creating portal   ... done
Attaching to ubuntu3, centos2, ubuntu2, centos3, docker, centos1, ubuntu-c, ubuntu1, portal

[spurin]

Fantastic, thanks @bsahane

Really great to hear. I've put weeks into the quirks of getting cgroups v2 support so am pleased to know it's working for you. Will keep the issue open for now to support @kanenses

If the updates on this issue are no longer of interest please unsubscribe and best of luck with the course!

[kanenses]

Hi @spurin Thank you for your words, indeed, I believe, that kids needs some challenges beside just playing a few games :-)

Sorry for late reply, I have out of home, here is my settings:

Running lastest release of Proxmox (7.0-13) with a running LXC container, with 2 vCPU and 4GB RAM, with follow options:

OS Release:

NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Docker version: Docker version 20.10.10, build b485636 Docker Compose: docker-compose version 1.29.2, build 5becea4c

I can confirm that running with excelent performance and no issues, a few container with docker.

[spurin]

Thanks @kanenses

Appreciate the info. So just to confirm on this, is the release-candidate branch working for you?

[kanenses]

I @spurin I have not try yet, I will do it now, A few minutes, and I can report it.

thank you

[kanenses]

Hi again, @spurin

I can confirm, that release-candidate branch working ;-)

Thank you!! next week, can start, the continuos of study with my son :-)

[spurin]

Fantastic news! Thanks @kanenses

Best of luck with the course and keep up the great influence with your prodigy 🚀👍

[kanenses]

Thank you alot @spurin !! I'am going close issue, as it resolved!

Have a great week! :-)

[spurin]

@kanenses one more thing, you might need to add the line

- LOCALHOST_OVERRIDE=1.2.3.4

With your IP address of the vm to the portal environment section, in the docker-compose.yaml

This fixes the links in the portal so rather than using localhost, they use the ip referenced

[kanenses]

@kanenses one more thing, you might need to add the line

- LOCALHOST_OVERRIDE=1.2.3.4

With your IP address of the vm to the portal environment section, in the docker-compose.yaml

This fixes the links in the portal so rather than using localhost, they use the ip referenced

Awesome @spurin - I was thinking tomorrow, take a look at that point. In that case, is more "fast" make change.

Again, thank you for your time!! :-) And great work!