Port mapping(?) on clean Ubuntu 16.04 install?

[evansharp]

Hello,

I am having difficulty accessing my newly-installed wok/ kimchi 2.5 implementation on a fresh Ubuntnu 16.04 box over a LAN. Connecting on my local network to https://severip:8001 times out. Ping shows clean network access to the server. I am working on it over ssh.

1) Brand new server machine, nothing besides Nginx installed to conflict with 2) Installation and start of Nginx and Wok are apparently ok:

evan@bigjimmy:/var/log/wok$ sudo systemctl status nginx ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2017-09-11 13:52:57 PDT; 1 day 18h ago Main PID: 1846 (nginx) Tasks: 9 Memory: 9.0M CPU: 29.864s CGroup: /system.slice/nginx.service ├─1846 nginx: master process /usr/sbin/nginx -g daemon on; master_process on ├─1847 nginx: worker process
├─1848 nginx: worker process
├─1849 nginx: worker process
├─1850 nginx: worker process
├─1851 nginx: worker process
├─1852 nginx: worker process
├─1853 nginx: worker process
└─1854 nginx: worker process

Sep 11 13:52:57 bigjimmy systemd[1]: Starting A high performance web server and a reverse proxy server... Sep 11 13:52:57 bigjimmy systemd[1]: Started A high performance web server and a reverse proxy server.

evan@bigjimmy:/var/log/wok$ sudo systemctl status wokd ● wokd.service - Wok - Webserver Originated from Kimchi Loaded: loaded (/lib/systemd/system/wokd.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/wokd.service.d └─kimchi.conf Active: active (running) since Mon 2017-09-11 13:53:42 PDT; 1 day 18h ago Docs: https://github.com/kimchi-project/wok/wiki Main PID: 2404 (python2) Tasks: 15 Memory: 67.5M CPU: 59.812s CGroup: /system.slice/wokd.service ├─2404 python2 /usr/bin/wokd └─2645 python2 /usr/bin/wokd

Sep 11 13:53:42 bigjimmy systemd[1]: Started Wok - Webserver Originated from Kimchi. Sep 12 07:49:20 bigjimmy systemd[1]: Started Wok - Webserver Originated from Kimchi. Sep 12 07:49:36 bigjimmy systemd[1]: wokd.service: Dependency After=wokd.service dropped Sep 12 07:49:39 bigjimmy systemd[1]: Started Wok - Webserver Originated from Kimchi. Sep 12 15:42:39 bigjimmy systemd[1]: Started Wok - Webserver Originated from Kimchi.

3) Firewall rules to open ports 8000 and 8001 added to iptables and then also ufw when I decided to run that on top.

4) netstat shows:

Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:8010 0.0.0.0: LISTEN 2404/python2
tcp 0 0 0.0.0.0:80 0.0.0.0: LISTEN 1846/nginx -g daemo tcp 0 0 192.168.122.1:53 0.0.0.0: LISTEN 2249/dnsmasq
tcp 0 0 127.0.1.1:53 0.0.0.0: LISTEN 1872/dnsmasq
tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN 1819/sshd
tcp 0 0 127.0.0.1:64667 0.0.0.0: LISTEN 2645/python2
tcp 0 0 10.1.1.8:22 10.1.102.248:57048 ESTABLISHED 4612/sshd: evan [pr tcp6 0 0 :::80 ::: LISTEN 1846/nginx -g daemo tcp6 0 0 :::22 ::: LISTEN 1819/sshd
udp 0 0 0.0.0.0:36909 0.0.0.0: 1872/dnsmasq
udp 0 0 192.168.122.1:53 0.0.0.0: 2249/dnsmasq
udp 0 0 127.0.1.1:53 0.0.0.0: 1872/dnsmasq
udp 0 0 0.0.0.0:67 0.0.0.0: 2249/dnsmasq
udp 0 0 0.0.0.0:68 0.0.0.0:* 1863/dhclient

...hmmm, I don't see :8001, but I see the PID of wokd listening on :8010...

5) wok-error.log shows this:

[11/Sep/2017:13:53:45] ENGINE Listening for SIGHUP. [11/Sep/2017:13:53:45] ENGINE Listening for SIGTERM. [11/Sep/2017:13:53:45] ENGINE Listening for SIGUSR1. [11/Sep/2017:13:53:45] ENGINE Bus STARTING [11/Sep/2017:13:53:45] ENGINE Serving on http://127.0.0.1:8010 [11/Sep/2017:13:53:45] ENGINE Bus STARTED

There is nothing in the documentation about having to set up port mapping/ forwarding. Is this in the triggers for Nginx when you install or start Wokd? For kicks I have tried opening port 8010, but the connection times out.

Thoughts? Help? Thanks in advance!

[alinefm]

Hi @evansharp

Let me first explain the ports used by Wok:

Wok launches a cherrypy instance on localhost:8010, ie, it is not exposed outside. It is just for security matters as Wok runs as root on system to do the management.

The ports 8000 (HTTP) and 8001 (HTTPS) are the ports exposed outside by nginx. Wok relies on nginx as a reverse proxy to communicate with the local cherrypy instance running.

That said, you need to open the ports 8000 and 8001 on firewall, or in case of Debian/Ubuntu, on ufw. I am not full familiar with Debian/Ubuntu but from a quick search over the internet I found those commands:

sudo ufw allow https
sudo ufw allow https
sudo ufw allow 8000
sudo ufw allow 8001

sudo ufw allow proto tcp from any to any port 8000,8001

For reference: https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands

Could you try that and check it works?

[evansharp]

Hi @alinefm, thanks for the quick reply.

The ports are definitely open.

Is there any test I can perform to verify the reverse proxy is configured? What do you make of my netstat output above that dosen't show anything listening on port 8001/ 8000, even though wok and nginx services are up?

I've never used Nginx before, but I have extensive experience with Apache.

Thanks!

[alinefm]

Have you already tried to restart nginx service? Is there any warning or error message on logs? You can also check the wok configuration file related to nginx (/etc/nginx/conf.d/wok.conf)

[evansharp]

I'll have a look at the config. I have restarted the services and whole machine several times.

[alinefm]

Are you able to access it locally by localhost:8001 ?

[evansharp]

Thanks again for your continued help @alinefm!

wok.conf does contain the default:

location / {

Default cherrypy port for Wok is 8010

    # DO NOT forget to update cherrypy_port parameter in /etc/wok/wok.conf
    # when changing this value
    proxy_pass http://127.0.0.1:8010;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    # Update location path for https for relative path
    # e.g.: proxy_redirect http://127.0.0.1:8010/ https://$host:8001/wok/;
    proxy_redirect http://127.0.0.1:8010/ https://$host:8001/;
}

Which indicates that even though nothing is "listening" on :8001 for netstat to report, connections there should get to :8010 for cherrypy ok. Does this look ok to a Nginx person?

The server is a headless rack-mount unit offsite, so getting a localhost GUI isn't possible without a lot of VNC overhead I don't want on the box.

Also, I just grep-ed through the kernel and system logs to confirm that no relevant traffic is being blocked.

Other thoughts?

[evansharp]

I turned up something in the wok-error.log that Google isn't clear about. Does this 'internal error' matter for basic operation? It looks like it's related to a firewall, but I've been really thorough on that front.

Kimchi: Running feature tests Service Libvirtd running ...: True internal error: Child process (/sbin/showmount --no-headers --exports 127.0.0.1) unexpected exit status 1: clnt_create: RPC: Port mapper failure - Unable to receive: errno 111 (Connection refused)

NFS Target Probe support ...: True Fibre Channel Host support .: True Kernel VFIO support ........: True Network Manager running ....: True Memory Hotplug support .....: True Kimchi: Feature tests completed

My issue could be similar to kimchi-project/kimchi #1051:

root@bigjimmy:/var/log/wok# systemctl status libvirt-bin ● libvirt-bin.service - Virtualization daemon Loaded: loaded (/lib/systemd/system/libvirt-bin.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2017-09-11 13:53:07 PDT; 2 days ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 2008 (libvirtd) Tasks: 18 Memory: 36.9M CPU: 7.078s CGroup: /system.slice/libvirt-bin.service ├─2008 /usr/sbin/libvirtd ├─2249 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper └─2250 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper

Sep 11 13:53:09 bigjimmy dnsmasq[2249]: reading /etc/resolv.conf Sep 11 13:53:09 bigjimmy dnsmasq[2249]: using nameserver 127.0.1.1#53 Sep 11 13:53:09 bigjimmy dnsmasq[2249]: read /etc/hosts - 5 addresses Sep 11 13:53:09 bigjimmy dnsmasq[2249]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Sep 11 13:53:09 bigjimmy dnsmasq-dhcp[2249]: read /var/lib/libvirt/dnsmasq/default.hostsfile Sep 11 13:53:09 bigjimmy libvirtd[2008]: libvirt version: 1.3.1, package: 1ubuntu10.14 (Jorge Niedbalski jorge.niedbalski@canonical.com Thu, 10 Aug 2017 22:50:46 -0400) Sep 11 13:53:09 bigjimmy libvirtd[2008]: hostname: bigjimmy Sep 11 13:53:09 bigjimmy libvirtd[2008]: cannot open path '/var/lib/kimchi/isos': No such file or directory Sep 11 13:53:09 bigjimmy libvirtd[2008]: Failed to autostart storage pool 'ISO': cannot open path '/var/lib/kimchi/isos': No such file or directory Sep 14 13:15:28 bigjimmy libvirtd[2008]: internal error: Child process (/sbin/showmount --no-headers --exports 127.0.0.1) unexpected exit status 1: clnt_create: RPC: Port mapper failure - Unable to receive: errno 111 (Connection refused)

/var/lib/kimchi/isos does exist though. It is root:root and drwxr-xr-x. This is all 'as installed', is this a bug?

[evansharp]

... I 777-ed the isos/ and the error goes away for libvirt-bin. Still left with clnt_create: RPC: Port mapper failure - Unable to receive: errno 111 (Connection refused) though.

[alinefm]

The /etc/nginx/conf.d/wok.conf file should look like https://github.com/kimchi-project/wok/blob/master/src/nginx/wok.conf

See the server instance configuration at line 33:

listen 0.0.0.0:8001 ssl;

I have a guess about the problem. Please, paste me the content of /etc/nginx directory and the content of /etc/nginx/nginx.conf:

The /etc/nginx/nginx.conf file should have something like below to load wok configuration:

    # Load modular configuration files from the /etc/nginx/conf.d directory.    
    # See http://nginx.org/en/docs/ngx_core_module.html#include                 
    # for more information.                                                     
    include /etc/nginx/conf.d/*.conf;

[evansharp]

evan@bigjimmy:/etc/nginx$ ls -ll
total 60
drwxr-xr-x 2 root root 4096 Sep 14 12:46 conf.d
-rw-r--r-- 1 root root 1077 Feb 11  2017 fastcgi.conf
-rw-r--r-- 1 root root 1007 Feb 11  2017 fastcgi_params
-rw-r--r-- 1 root root 2837 Feb 11  2017 koi-utf
-rw-r--r-- 1 root root 2223 Feb 11  2017 koi-win
-rw-r--r-- 1 root root 3957 Feb 11  2017 mime.types
-rw-r--r-- 1 root root 1462 Feb 11  2017 nginx.conf
-rw-r--r-- 1 root root 1462 Sep 10 14:16 nginx.conf.backup
-rw-r--r-- 1 root root  180 Feb 11  2017 proxy_params
-rw-r--r-- 1 root root  636 Feb 11  2017 scgi_params
drwxr-xr-x 2 root root 4096 Sep 11 13:51 sites-available
drwxr-xr-x 2 root root 4096 Sep 10 14:06 sites-enabled
drwxr-xr-x 2 root root 4096 Sep 11 13:51 snippets
-rw-r--r-- 1 root root  664 Feb 11  2017 uwsgi_params
-rw-r--r-- 1 root root 3071 Feb 11  2017 win-utf

---

 evan@bigjimmy:/etc/nginx$ sudo vi nginx.conf
 user www-data;
 worker_processes auto;
 pid /run/nginx.pid;

 events {
         worker_connections 768;
         # multi_accept on;
 }

 http {

         ##
         # Basic Settings
         ##

         sendfile on;
         tcp_nopush on;
         tcp_nodelay on;
         keepalive_timeout 65;
         types_hash_max_size 2048;
         # server_tokens off;

         # server_names_hash_bucket_size 64;
         # server_name_in_redirect off;

         include /etc/nginx/mime.types;
         default_type application/octet-stream;

         ##
         # SSL Settings
         ##

         ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
         ssl_prefer_server_ciphers on; 

         ##
         # Logging Settings
         ##

         access_log /var/log/nginx/access.log;
         error_log /var/log/nginx/error.log;

         ##
         # Gzip Settings
         ##

         gzip on;
         gzip_disable "msie6";

         # gzip_vary on;
         # gzip_proxied any;
         # gzip_comp_level 6;
         # gzip_buffers 16 8k;
         # gzip_http_version 1.1;
         # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

         ##
         # Virtual Host Configs
         ##

         include /etc/nginx/conf.d/*.conf;
         include /etc/nginx/sites-enabled/*;
} 

#mail {
#       # See sample authentication script at:
#       # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
#       # auth_http localhost/auth.php;
#       # pop3_capabilities "TOP" "USER";
#       # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
#       server {
#               listen     localhost:110;
#               protocol   pop3;
#               proxy      on;
#       }
#
#       server {
#               listen     localhost:143;
#               protocol   imap;
#               proxy      on;
#       }
#}

The include you mention is present.

[alinefm]

Hrmm.. Your /etc/nginx/nginx.conf file already has the statement include /etc/nginx/conf.d/*.conf;

Please, try to copy/paste the wok.conf file under /etc/nginx/sites-enabled/* as well, restart nginx service and check if it helps in some way.

[evansharp]

Nginx will not restart after copying /etc/wok/wok.conf to /etc/nginx/sites-enabled/ because of a syntax error in wok.conf. Comparing /etc/nginx/sites-enabled/default to the copiedwok.conf, they do not seem to be the same type of thing; the formatting is quite different.

Am I looking at the wrong conf?

[evansharp]

Ok I resolved this: Indeed there was no wok.conf in my Nginx sites-available/. I copied the sourcefile from the project and got it working.

I now see:

sudo service nginx status
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-09-18 15:16:38 PDT; 1s ago
  Process: 3516 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
  Process: 3524 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 3520 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 3526 (nginx)
    Tasks: 9
   Memory: 3.3M
      CPU: 15ms
   CGroup: /system.slice/nginx.service
           ├─3526 nginx: master process /usr/sbin/nginx -g daemon on; master_process on
           ├─3527 nginx: worker process                           
           ├─3528 nginx: worker process                           
           ├─3529 nginx: worker process                           
           ├─3530 nginx: worker process                           
           ├─3531 nginx: worker process                           
           ├─3532 nginx: worker process                           
           ├─3533 nginx: worker process                           
           └─3534 nginx: worker process                           

Sep 18 15:16:38 bigjimmy systemd[1]: Starting A high performance web server and a reverse proxy server...
Sep 18 15:16:38 bigjimmy systemd[1]: Started A high performance web server and a reverse proxy server.

So that's promising. netstat confirms the server is listening on :8001 now:

sudo netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8010          0.0.0.0:*               LISTEN      3112/python2    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      3526/nginx -g daemo
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      2205/dnsmasq    
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN      1901/dnsmasq    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1797/sshd       
tcp        0      0 127.0.0.1:64667         0.0.0.0:*               LISTEN      3118/python2    
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      3526/nginx -g daemo
tcp        0      0 0.0.0.0:8001            0.0.0.0:*               LISTEN      3526/nginx -g daemo
tcp6       0      0 :::80                   :::*                    LISTEN      3526/nginx -g daemo
tcp6       0      0 :::22                   :::*                    LISTEN      1797/sshd

And my firewall is definitely open:

sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere                  
8001                       ALLOW       Anywhere                  
80                         ALLOW       Anywhere                  
443                        ALLOW       Anywhere                  
8000                       ALLOW       Anywhere                  
2222                       ALLOW       Anywhere                  
80,443/tcp                 ALLOW       Anywhere                  
8000,8001/tcp              ALLOW       Anywhere                  
Nginx Full                 ALLOW       Anywhere                  
8000/tcp                   ALLOW       Anywhere                  
8001/tcp                   ALLOW       Anywhere                  
8010/tcp                   ALLOW       Anywhere                  
8010                       ALLOW       Anywhere                  
22 (v6)                    ALLOW       Anywhere (v6)             
8001 (v6)                  ALLOW       Anywhere (v6)             
80 (v6)                    ALLOW       Anywhere (v6)             
443 (v6)                   ALLOW       Anywhere (v6)             
8000 (v6)                  ALLOW       Anywhere (v6)             
2222 (v6)                  ALLOW       Anywhere (v6)             
80,443/tcp (v6)            ALLOW       Anywhere (v6)             
8000,8001/tcp (v6)         ALLOW       Anywhere (v6)             
Nginx Full (v6)            ALLOW       Anywhere (v6)             
8000/tcp (v6)              ALLOW       Anywhere (v6)             
8001/tcp (v6)              ALLOW       Anywhere (v6)             
8010/tcp (v6)              ALLOW       Anywhere (v6)             
8010 (v6)                  ALLOW       Anywhere (v6)

This is pretty epic as far as setup ghosts go. Any other ideas @alinefm?

[alinefm]

So have you copied wok.conf flie under sites-available/ and sites-enable/, restarted nginx and wok service, nfw is proper configure and it is still not working?

Are you able to access it locally localhost:8000?

Also, in Fedora, I need to do some SElinux configuration. Is there SElinux on Debian? Is it proper configured?

[evansharp]

The server is headless, so no browser.

I can ping localhost successfully, which means :80 is open. To check :8001 I used nmap:

Starting Nmap 7.01 ( https://nmap.org ) at 2017-09-19 13:09 PDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000066s latency).
Other addresses for localhost (not scanned): ::1
PORT     STATE SERVICE
8001/tcp open  vcom-tunnel

Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds

SELinux is not part of Debian/ Ubuntu. The similar default is Apparmor. All things are default as this is a clean new server.

[evansharp]

Resolved!

In the end, the issue was both due to firewall and me having to specify the "https://" protocol in my browser, since the wok.conf specifically listens to :8001 ssl.

Thank you for all your help @alinefm!