silent operation

[dreamcat4]

Right... with newest v1.8.2 we are always outputting certain things during the startup and shutdown. That is good for debugging problems, and OK when running things in daemon mode. However for single-shot cmdline operation, is not desirable.

I wonder what would be the best way to do this in the stageN scripts ?

Here is a real-world example:

**s6-overlay v1.8.2 output***

s$ docker run dreamcat4/tvheadend-uk-atlas --help
[fix-attrs.d] applying owners & permissions fixes...
[fix-attrs.d] applying 00-runscripts...  exited 0
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
/usr/bin/tvheadend -u hts -g video -c /config --help
Usage: /usr/bin/tvheadend [OPTIONS]

Generic Options

  -h, --help                  Show this page
  -v, --version               Show version infomation

Service Configuration

  -c, --config                Alternate config path
  -B, --nobackup              Do not backup config tree at upgrade
  -f, --fork                  Fork and run as daemon
  -u, --user                  Run as user
  -g, --group                 Run as group
  -p, --pid                   Alternate pid path
  -C, --firstrun              If no user account exists then create one with
                              no username and no password. Use with care as
                              it will allow world-wide administrative access
                              to your Tvheadend installation until you edit
                              the access-control from within the Tvheadend UI
  -U, --dbus                  Enable DBus
  -e, --dbus_session          DBus - use the session message bus instead system one
  -a, --adapters              Only use specified DVB adapters (comma separated)
      --satip_xml             URL with the SAT>IP server XML location

Server Connectivity

  -6, --ipv6                  Listen on IPv6
  -b, --bindaddr              Specify bind address
      --http_port             Specify alternative http port
      --http_root             Specify alternative http webroot
      --htsp_port             Specify alternative htsp port
      --htsp_port2            Specify extra htsp port
      --useragent             Specify User-Agent header for the http client
      --xspf                  Use xspf playlist instead M3U

Debug Options

  -d, --stderr                Enable debug on stderr
  -s, --syslog                Enable debug to syslog
  -l, --logfile               Enable debug to file
      --debug                 Enable debug subsystems
      --trace                 Enable trace subsystems
      --fileline              Add file and line numbers to debug
      --threadid              Add the thread ID to debug
      --uidebug               Enable webUI debug (non-minified JS)
  -A, --abort                 Immediately abort
  -D, --dump                  Enable coredumps for daemon
      --noacl                 Disable all access control checks
      --nobat                 Disable DVB bouquets
  -j, --join                  Subscribe to a service permanently

TODO: testing

      --tsfile_tuners         Number of tsfile tuners
      --tsfile                tsfile input (mux file)

For more information please visit the Tvheadend website:
  https://tvheadend.org

/entrypoint.sh exited 0. Stopping the supervision tree.
[s6-finish] syncing disks.
if: fatal: child crashed with signal 11
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

Desired output: - s6-overlay is silent

$ docker run --entrypoint="tvheadend" dreamcat4/tvheadend-uk-atlas --help
Usage: tvheadend [OPTIONS]

Generic Options

  -h, --help                  Show this page
  -v, --version               Show version infomation

Service Configuration

  -c, --config                Alternate config path
  -B, --nobackup              Do not backup config tree at upgrade
  -f, --fork                  Fork and run as daemon
  -u, --user                  Run as user
  -g, --group                 Run as group
  -p, --pid                   Alternate pid path
  -C, --firstrun              If no user account exists then create one with
                              no username and no password. Use with care as
                              it will allow world-wide administrative access
                              to your Tvheadend installation until you edit
                              the access-control from within the Tvheadend UI
  -U, --dbus                  Enable DBus
  -e, --dbus_session          DBus - use the session message bus instead system one
  -a, --adapters              Only use specified DVB adapters (comma separated)
      --satip_xml             URL with the SAT>IP server XML location

Server Connectivity

  -6, --ipv6                  Listen on IPv6
  -b, --bindaddr              Specify bind address
      --http_port             Specify alternative http port
      --http_root             Specify alternative http webroot
      --htsp_port             Specify alternative htsp port
      --htsp_port2            Specify extra htsp port
      --useragent             Specify User-Agent header for the http client
      --xspf                  Use xspf playlist instead M3U

Debug Options

  -d, --stderr                Enable debug on stderr
  -s, --syslog                Enable debug to syslog
  -l, --logfile               Enable debug to file
      --debug                 Enable debug subsystems
      --trace                 Enable trace subsystems
      --fileline              Add file and line numbers to debug
      --threadid              Add the thread ID to debug
      --uidebug               Enable webUI debug (non-minified JS)
  -A, --abort                 Immediately abort
  -D, --dump                  Enable coredumps for daemon
      --noacl                 Disable all access control checks
      --nobat                 Disable DVB bouquets
  -j, --join                  Subscribe to a service permanently

TODO: testing

      --tsfile_tuners         Number of tsfile tuners
      --tsfile                tsfile input (mux file)

For more information please visit the Tvheadend website:
  https://tvheadend.org

[glerchundi]

Hi @dreamcat4,

I firmly believe that entrypoint is not being overriden in your running context:

Without entrypoint:

$> docker run -ti base
[fix-attrs.d] applying owners & permissions fixes...
[fix-attrs.d] applying 00-runscripts...  exited 0
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.

With entrypoint:

$> docker run --entrypoint="ls" base -la /root
total 16
drwxr-xr-x  2 root root 4096 Mar  7 12:00 .
drwxr-xr-x 57 root root 4096 Mar 22 19:15 ..
-rw-r--r--  1 root root 2550 Mar  7 12:00 .bashrc
-rw-r--r--  1 root root  140 Feb 20  2014 .profile

Perhaps you are suffering from this, docker/docker#5539?

[dreamcat4]

Thanks Gorka. I edited the original example to give correct command. Although your example is better / smaller. Thanks for that.

[glerchundi]

Do you really think that we need to hide supervision related logs when running a one-shot command? I don't think so! One-shot, or not supervised logs, it was thought for debugging purposes, this overlay is all about supervision of more or less long-lived processes...If you want to avoid that kind of logs, just use S6_USE_CATCHALL_LOGGER=1 and you're done. I think hiding how our system works will obscure to the end-user the real behaviour of our images.

Anyway, something is nagging me from your comment, why in the "real" output appears --entrypoint="tvheadend" after supervision tree was up? Can put the Dockerfile and I will give it a try :-)

What do you think?

[dreamcat4]

Can you please give some example how to use S6_USE_CATCHALL_LOGGER=1 to make the stout/stderr silent? Is that a way to solve the issue?

I would not worry of the output from my example… was not fully corrected. It isn't relevant but i edited again just to show how it appears ATM.

[glerchundi]

Supervised:

[...]
ENV S6_USE_CATCHALL_LOGGER 1
ENTRYPOINT [ "/init" ]
CMD [ "/bin/ls" ]

$> docker run -ti base
bin  boot  dev  etc  home  init  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
/bin/ls exited 0

Or overriding entrypoint, it can be done via --entrypoint="/bin/ls" or using ENTRYPOINT in your custom image:

$> docker run -ti base
bin   dev  home  lib    media  opt   root  sbin  sys  usr
boot  etc  init  lib64  mnt    proc  run   srv   tmp  var

The problem with the first case is that CMD itself is overriden if you provide custom arguments, so that this syntax is not possible, but that is something related to docker and not to our overlay!

Although I found a bug during these tests, it should work in the way I show below :-)

$> docker run -ti base -la
exec: fatal: unable to exec -la: No such file or directory

EDIT: Fixed in #27 :-), 1.8.4 should be available in a few seconds.

[dreamcat4]

OK Gorka. The first option is really what I wanted… as we have not yet documented most of these new things I guess that is why I didn't know about it :)

I don't mind the using the variable ENV S6_USE_CATCHALL_LOGGER 1. However perhaps it would not hurt, be more readable / obvious to users who want to silence s6. To also have in addition an extra environment variable like S6_SILENT or something. Which just was a simple alias to set the S6_USE_CATCHALL_LOGGER. Or S6_SILENT_STDERR etc. Whatever I don't mind.

Thanks for fixing the issue #27 too while at it. Appreciate it.

[glerchundi]

Nice to hear that :-)

For now, and until no others complain about this i'm not going to include more envs to configure how does it behaves. Because I don't want to hide what it really does, being silent is just a collateral effect of S6_USE_CATCHALL_LOGGER and I think that whoever which uses this overlay must know what is every env used for. If you want to achieve that using this env, it's up to you. But it's important to keep our overlay clean and maintainable.

Anyway, just for knowing what's your use case. Are you including server & client binaries inside the same docker container?

[dreamcat4]

I think you are too close to the problem and not seeing why I mentioned it:

The fact is I could not determine S6_USE_CATCHALL_LOGGER actually did that, and raised bug report on it. Until you told me it did that too as some kind of a side-effect.

S6_USE_CATCHALL_LOGGER. Just means 'use catchall logger'. It does not say 'don't use stout / std err'. or state anything one way or another about the effect on stout / std err…

I'm not saying they should or should not be independent of one another. Just that the full state behaviour is not reflected or entirely clarified by that particular ENV variable name alone.

So from that naiive user perspective, who are not knowing, it can log potentially to both destinations simultaneously and not be switching off the other one. = unclear. The fact that it does not do both is not possible to be deduced (by the user) without requiring or relying upon a documentation note to say otherwise.

2) As for use of it:

For me it's because I want my images to be usable in all possible ways… which includes getting --help and single-shot command line usage. Because then users can start using the image straight away, without needing to edit any configuration, read documentation, etc.

But in many other cases a complex regular command (not a server) can also create extra orphaned processes and need to be supervised. Process supervision is not something only required by server processes. If a command invokes java and other stuff to do it's work, for example say, a video transcoding job, like handbrake, it can easily be spawning a lot of processes. As a single-shot command. Therefore, the behaviour of setting S6_USE_CATCHALL_LOGGER flag is good for such purposes. When the task is on the terminal, in the foreground. Or the output is being piped | into other unix commands.

[glerchundi]

Ok, so it is more a matter of semantics!

But we'll have the same feeling with: S6_SILENT or S6_SILENT_STDERR. It hides everything but doesn't mean that they are being saved inside the container, concretely in /var/log/s6-uncaught-logs, I think that we need to invest our time and effort creating a very good documentation about this overlay and try to find good namings for all environment variables. This is a work in progress project, and it's not even beta yet, we have the possibility to change everything because no one is using it right now and we don't need to provide backward compatibility. I have used USE_CATCHALL_LOGGER because it was its main purpose and silently hides everything as a side-effect, so if you feel more comfortable with another env name which more or less merges both meaning/purposes I will be delighted to hear it and we'll discuss it here.

@jprjr what do you think about this?

2) I agree completely :-)

[dreamcat4]

Thank you Gorka. Sorry I don't have a magic solution if 1 flag do 2 different things at the same time.

Actually it can instead be written as S6_LOGGING=catchall,stderr,<other logging target(s)> comma separated list. With a default value of stderr. Where the text string is read for certain keywords. But that way is extra complex for the execline code to take as an input. example which was implemented in a regular shell script (not execline).

Also: When talking about a single config flag. The logic can instead be reversed. E.g. S6_LOGGING_STDERR=0. Default true. Just another way of doing things. TBH I'm not really in preference either particular way. Guess it depends which in general is best for the users to understand.

[glerchundi]

I have changed S6_USE_CATCHALL_LOGGER for S6_LOGGING:

• 0: Outputs everything to stdout/stderr
• 1: Uses internal catchall logger

For the newcomer, it will be easier to understand how to achieve things like this...

[dreamcat4]

Thanks Gorka! I agree it is a good improvement. Excellent.

[jprjr]

Sorry guys, I didn't see you mention me and ask for my input!

I think the S6_LOGGING is the way to go for sure!

[glerchundi]

i'm going to close this issue and create a new one pointing the need to have verbosity levels.