search

juju-solutions / bigtop

Mirror of Apache Bigtop
Apache License 2.0
0 stars 2 forks source link

zeppelin charm should expose ZEPPELIN_WAR_TEMPDIR as config option #57

Open marfrit opened 6 years ago

marfrit commented 6 years ago

I just created a zeppelin installation using "juju deploy zeppelin". Here's where I got stuck (the installation is created using a KVM based virtual machine managed by a MAAS pod):

ubuntu@v010203:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            477M     0  477M   0% /dev
tmpfs           100M  100M     0 100% /run
/dev/vda1        92G  6.5G   81G   8% /
tmpfs           496M     0  496M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           496M     0  496M   0% /sys/fs/cgroup
tmpfs           100M     0  100M   0% /run/user/119
tmpfs           100M     0  100M   0% /run/user/1000
ubuntu@v010203:~$ sudo du -hs /run/*
4.0K    /run/acpid.pid
0       /run/acpid.socket
0       /run/agetty.reload
4.0K    /run/atd.pid
8.0K    /run/blkid
20K     /run/cloud-init
4.0K    /run/crond.pid
0       /run/crond.reboot
0       /run/dbus
0       /run/dmeventd-client
0       /run/dmeventd-server
0       /run/initctl
4.0K    /run/initramfs
4.0K    /run/irqbalance.pid
4.0K    /run/iscsid.pid
0       /run/lock
2.5M    /run/log
0       /run/lvm
4.0K    /run/lvmetad.pid
0       /run/lxcfs
4.0K    /run/lxcfs.pid
0       /run/lxd-bridge
8.0K    /run/mdadm
0       /run/motd.dynamic.new
0       /run/mount
24K     /run/network
4.0K    /run/ntpd.pid
0       /run/puppet
12K     /run/resolvconf
4.0K    /run/rsyslogd.pid
0       /run/screen
0       /run/sendsigs.omit.d
0       /run/shm
0       /run/snapd-snap.socket
0       /run/snapd.socket
0       /run/sshd
4.0K    /run/sshd.pid
0       /run/sudo
164K    /run/systemd
0       /run/thermald
4.0K    /run/tmpfiles.d
0       /run/ubuntu-fan
304K    /run/udev
0       /run/user
4.0K    /run/utmp
0       /run/uuidd
0       /run/xtables.lock
97M     /run/zeppelin

From /var/log/zeppelin/zeppelin-zeppelin-v010203.log:

WARN [2018-05-06 17:35:36,994] ({main} ZeppelinConfiguration.java[create]:97) - Failed to load configuration, proceeding with a default
 INFO [2018-05-06 17:35:37,291] ({main} ZeppelinConfiguration.java[create]:109) - Server Host: 0.0.0.0
 INFO [2018-05-06 17:35:37,291] ({main} ZeppelinConfiguration.java[create]:111) - Server Port: 9080
 INFO [2018-05-06 17:35:37,292] ({main} ZeppelinConfiguration.java[create]:115) - Context Path: /
 INFO [2018-05-06 17:35:37,304] ({main} ZeppelinConfiguration.java[create]:116) - Zeppelin Version: 0.7.2
 INFO [2018-05-06 17:35:38,030] ({main} Log.java[initialized]:186) - Logging initialized @47225ms
 INFO [2018-05-06 17:35:38,563] ({main} ZeppelinServer.java[setupWebAppContext]:343) - ZeppelinServer Webapp path: /var/run/zeppelin/webapps
 INFO [2018-05-06 17:35:43,421] ({main} ZeppelinServer.java[main]:187) - Starting zeppelin server
 INFO [2018-05-06 17:35:43,426] ({main} Server.java[doStart]:327) - jetty-9.2.15.v20160210
 WARN [2018-05-06 17:35:46,046] ({main} WebAppContext.java[doStart]:514) - Failed startup of context o.e.j.w.WebAppContext@44f75083{/,null,null}{/usr/lib/zeppelin/zeppelin-web-0.7.2.war}
java.io.IOException: No space left on device
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:326)
        at org.eclipse.jetty.util.IO.copy(IO.java:162)
        at org.eclipse.jetty.util.IO.copy(IO.java:118)
        at org.eclipse.jetty.util.resource.JarResource.copyTo(JarResource.java:240)
        at org.eclipse.jetty.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:468)
        at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:72)
        at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:468)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:504)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
        at org.eclipse.jetty.server.Server.start(Server.java:387)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.Server.doStart(Server.java:354)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.zeppelin.server.ZeppelinServer.main(ZeppelinServer.java:189)
 INFO [2018-05-06 17:35:46,463] ({main} AbstractConnector.java[doStart]:266) - Started ServerConnector@11e21d0e{HTTP/1.1}{0.0.0.0:9080}
 INFO [2018-05-06 17:35:46,464] ({main} Server.java[doStart]:379) - Started @55741ms
 INFO [2018-05-06 17:35:46,464] ({main} ZeppelinServer.java[main]:194) - Done, zeppelin server started

I looks to me that the default zeppelin directory defaults to /run which is created with a (small) size in Ubuntu 16.04 (nowadays?).

kwmonroe commented 6 years ago

@marfrit apologies for the delayed response.. You're right, 16.04 symlinks /var/run to /run:

$ ls -la /var/run
lrwxrwxrwx 1 root root 4 Jul 27 08:28 /var/run -> /run

And in your case, you're out of room in /run. As a workaround, you could move zeppelins webapps someplace else (/var/lib/zeppelin seems appropriate):

$ juju ssh zeppelin/0
...
ubuntu@ip-172-31-6-139:~$ cd /etc/zeppelin/conf
ubuntu@ip-172-31-6-139:/etc/zeppelin/conf$ sudo cp zeppelin-env.sh zeppelin-env.sh.orig
## sudo vi zeppelin-env.sh
ubuntu@ip-172-31-6-139:/etc/zeppelin/conf$ diff -u zeppelin-env.sh.orig zeppelin-env.sh
--- zeppelin-env.sh.orig        2018-08-06 15:56:36.802403316 +0000
+++ zeppelin-env.sh     2018-08-06 15:52:58.308325812 +0000
@@ -18,7 +18,7 @@
 export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
 export ZEPPELIN_LOG_DIR=/var/log/zeppelin
 export ZEPPELIN_PID_DIR=/var/run/zeppelin
-export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
+export ZEPPELIN_WAR_TEMPDIR=/var/lib/zeppelin/webapps
 export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
 export MASTER=local[*]
 export SPARK_HOME=/usr/lib/spark
ubuntu@ip-172-31-6-139:/etc/zeppelin/conf$ exit
...
$ juju run-action --wait zeppelin/0 restart
unit-zeppelin-0:
  id: 3af66652-d463-486c-8842-30a2321c2752
  status: completed
  timing:
    completed: 2018-08-06 15:57:04 +0000 UTC
    enqueued: 2018-08-06 15:56:55 +0000 UTC
    started: 2018-08-06 15:56:56 +0000 UTC
  unit: zeppelin/0

Bigtop does allow the war tmpdir to be declared at install time, but the zeppelin charm does not expose this as a config option.

I'll leave this open as a feature request, but hopefully the above workaround gets you unblocked.