Supervisor / supervisor

Supervisor process control system for Unix (supervisord)
http://supervisord.org
Other
8.46k stars 1.24k forks source link

Option to create directory of specified log files #120

Open felipemachado-sambatech opened 12 years ago

felipemachado-sambatech commented 12 years ago

I tried to set the logfile to a file in a directory that doesn't exist. Shouldn't there be an option to automatically create these directories?

mnaberez commented 12 years ago

I'd rather not add this option, sorry. The config file is already quite verbose and this option may not provide much benefit over mkdir. It would probably be less effort to just run mkdirthan to shut down supervisord, edit the configuration file to enable the automagic directory option, and start supervisord again.

sherbert commented 12 years ago

For the record, we would also appreciate this feature. We keep our log files in a path that is periodically purged of files and directories that haven't been touched in some amount of time (and there's nothing I can do about this). This means that if we shut down a process for a month, its log directory will be gone when we try to restart it, and so it would be nice for that directory to be automatically recreated.

RobWin commented 10 years ago

+1 for this option

jaanauati commented 10 years ago

+1

felipou commented 9 years ago

As a related note, this diminishes the reliability of supervisor. I don't know if that's been solved somehow in the master branch, but in version 3.1.1, someone accidentally removed a log directory in our server, and supervisor simply stopped working, taking all other programs with it, not just the one with the log directory missing.

I think the scalability of supervisor is also compromised because of this, since one problem in one of the programs can end up taking all others down.

I'd be happy to contribute with a pull request, I just need some pointers as to where I should start in the code.

Also, I don't know if I should create another issue for this...

coderlol commented 9 years ago

+1...very reasonable request, why not honoring it?

rb2nem commented 8 years ago

Missing this too....

paambaati commented 8 years ago

Is there a workaround that will at least let us create the directory from within the config file? Maybe run mkdir inside?

kujiy commented 8 years ago

+1

We docker users should do this

  1. ENTRYPOINT["sh", "/entrypoint.sh"] in Dockerfile
  2. In /entrypoint.sh, create a log directory and log files before calling supervisord

This is a shell script for /entrypoint.sh to create them from supervisord.conf automatically.

# Create log dirs and files
mkdir -p $( dirname $(cat /etc/supervisord.conf  | grep logfile= | grep "\.log" | sed s/.*logfile=// ) )
touch $( cat /etc/supervisord.conf  | grep logfile= | grep "\.log" | sed s/.*logfile=// )

# Then run supervisord
/usr/bin/supervisord

Is there any good idea?

timmyomahony commented 8 years ago

+1

supervisord doesn't start on boot for Arch Linux as the /var/log/supervisord.log doesn't exist when booting.

rb2nem commented 7 years ago

Here are more details about why this would be useful in my case. I use supervisord in a docker to run some software for which I want to persist the logs, so people are mounting a volume. But with supervisord not creating the needed subdirectories, it requires users to make the directories on the host and set owner and permission themselves. And it is not possible to put it in the command directive, as supervisord complains and stops even before looking at the command.

@mnaberez why sticking to your refusal? I have seen more solid arguments in favor of this than in your rejection.... (but I may be missing something, so more explanations welcome ;-)

aujang commented 7 years ago

This continues to be a problem, specifically for users deploying supervisor within Docker in an ElasticBeanstalk environment. If you specify logging in Dockerrun, the service wipes the specified directories during deployment, deleting anything created while Docker is spinning up. This in turn breaks supervisor, because it expects files where none exist. If supervisor simply created directories and logs when none existed, it would completely solve the issue. As it stands, I had to disable discrete log files (and use syslog instead) in order to get my app to deploy.

Niyojan commented 7 years ago

same here, I used the following workaround : docker run -v /mnt/my_logs/celery:/var/log/celery -v /mnt/my_logs/supervisor:/var/log/supervisor -v /mnt/my_logs/supervisor:/var/log/supervisor -v /mnt/my_logs/nginx:/var/log/nginx -v /mnt/my_media:/home/project/media -p 80:80 -p 5555:5555 -d my_project

adamreisnz commented 7 years ago

I am also running into this now. I have a Raspberry Pi which has /var/log mounted on tmpfs to prevent writes to the SD card. After a reboot of the device, the /var/log/supervisor folder is gone and supervisor won't start. Auto creating this directory would be a much welcome feature.

@mnaberez would you mind reconsidering? There have been offers of PR's for this feature as well.

For context, I've spent about 4 hours debugging this issue today, and could not find any useful information or error messages informing me that a missing log directory causes supervisor to not start at all. That should really not be happening, and I don't know of any other software that doesn't know how to create it's own log folders, let alone refuses to run if they don't exist 😢

corck commented 7 years ago

:+1: I just ran into this as well.

adamatan commented 7 years ago

+1 👍 Would be very helpful for me as well.

Nidhoggur1993 commented 6 years ago

A kindly remind here. If supervisor was installed by default and do not change it's log path out of /tmp/supervisord.log, in certain system like CentOS 7.3 or higher, the systemd-tmpfiles-clean.timer will do a cron to delete everything under /tmp everyday. We have suffer this a lot...

mavci commented 6 years ago

I just edited /etc/supervisor/supervisord.conf file and changed these lines;

logfile=/var/log/supervisor/supervisord.log to logfile=/var/log/supervisord.log childlogdir=/var/log/supervisor to childlogdir=/var/log/

and fixed for me. And I was the excatly the same situation with @adamreisnz :)

adililhan commented 6 years ago

docker wasn't around when this issue created. so, we use supervisord in docker nowadays. thanks it is nice tool.

however, if we want to redirect the outputs then we have to add new line (and it creates new layer) to dockerfile such as mkdir /var/log/supervisord/"

if supervisord creates the directory then it would be good. thanks anyway.

Destroy666x commented 6 years ago

I don't know of any other software that doesn't know how to create it's own log folders, let alone refuses to run if they don't exist

Bump

derekargueta commented 6 years ago

+1

mikelibg commented 6 years ago

@mnaberez First of all, supervisor is a great tool. Due to the fact that supervisor is the first one to run (before the log directory can be created by the program it is starting) it is intuitive that it can create the log directory by itself, if it does not exists yet. Is there any other logic explanation not to add this option besides the comment "...the config file is already quite verbose..."?

mnaberez commented 6 years ago

Due to the fact that supervisor is the first one to run (before the log directory can be created by the program it is starting)

If you are starting myprogram directly:

command = myprogram

You could instead start it with something like:

command = bash -c 'mkdir /foo && exec myprogram'
injaon commented 6 years ago

:+1: +1 to this request

FAKERINHEART commented 6 years ago

1063

await commented 6 years ago

+1 would like to to use program name in the log path. it's not completely obvious why supervisor doesn't start when this occurs

Dayjo commented 6 years ago

Also getting this same issue recently. Was quite suprised it didn't handle this internally.

rb2nem commented 6 years ago

@mnaberez You suggest to put it in command, but this does not work for supervisord's logs, as I put it in a comment in this thread:

And it is not possible to put it in the command directive, as supervisord complains and 
stops even before looking at the command
mnaberez commented 6 years ago

Can you create the directories before starting supervisord, e.g. in the init script or equivalent that launches supervisord? Why not?

adamreisnz commented 6 years ago

From the general sentiment in this issue, it would appear to me that most people expect a piece of software that requires certain directories to work, to be able to create those directories for itself.

It is also quite apparent that the majority of software out there does in fact do that, so it's fair that people have come to expect that.

Moreover, from reading the feedback in this thread, it appears that the way this error is handled is unclear and confusing to people, and that a lot of them don't understand why supervisor won't work.

I therefore think it's not a question of whether we can or can't have our initialisation scripts create the directory -- we can, and in our case we have -- but a question of why Supervisor can't simply do it itself.

There are 26 participants in this thread alone, that means there's at least 10x as many people who don't bother to comment here that are feeling the same way about this issue. If there's such a strong desire for what is at first glance a pretty trivial feature to ask for, then why not just implement it or have someone submit a PR?

adamreisnz commented 6 years ago

I would also like to add that for a problem this big, e.g. the whole application not starting if a directory that needs to be manually created doesn't exist, there's surprising little documentation around this fact.

http://supervisord.org/logging.html http://supervisord.org/faq.html http://supervisord.org/installing.html

Neither of these pages mention anything about needing to create log directories manually.

mnaberez commented 6 years ago

From the general sentiment in this issue, it would appear to me that most people expect a piece of software that requires certain directories to work, to be able to create those directories for itself.

It is also quite apparent that the majority of software out there does in fact do that, so it's fair that people have come to expect that.

I can't speak for the other maintainers, but I don't have experience with programs that do this. It might be helpful if someone named specific daemons in common use that have this behavior. This may be why I don't immediately recognize why this is needed or desirable. For all the other daemons that I run, I create the needed directories before I start them. You have to do work before starting the daemon anyway, e.g. you have to make a config file. Why can't the directory be created then?

Moreover, from reading the feedback in this thread, it appears that the way this error is handled is unclear and confusing to people, and that a lot of them don't understand why supervisor won't work.

This is the actual error message:

Error: The directory named as part of the path /nonexistant/cat/stdout.log does not exist in section 'program:cat' (file: 'supervisord.conf')
For help, use /usr/local/bin/supervisord -h

If this is unclear, how can it be reworded?

mnaberez commented 6 years ago

If there's such a strong desire for what is at first glance a pretty trivial feature to ask for, then why not just implement it or have someone submit a PR?

mnaberez commented 6 years ago

I would also like to add that for a problem this big, e.g. the whole application not starting if a directory that needs to be manually created doesn't exist, there's surprising little documentation around this fact.

Fail-fast is a common approach when daemons and other server software are started with a bad configuration. There's are no documentation stating which configuration options are allowed to be invalid and which aren't because none of them are.

In this case, the user has specified an optional logfile but supervisord can't create that logfile because the directory specified by the user doesn't exist. The existing implementation wasn't written to automatically create directories so supervisord outputs a descriptive error message and refuses to start until the configuration is fixed. It does that not only with this condition but with as many configuration issues as possible.

The alternative would be to somehow enable supervisord to partially start with a known broken configuration, perhaps trying to ignore only those options that are invalid. This would make its operation unpredictable and delay fixing the problems.

rb2nem commented 6 years ago

@mnaberez My primary reason to ask this feature was that I used supervisord in docker. Docker containers don't have an init system and are designed to run only one command. When people need to run multiple processes in one container, suprevisord is a great fit, except for this problem. When users want to persist logs of the commands started by supervisord, they mount a volume where supervisord saves the app's logs. Due to this problem, they have to know the hierarchy of directories to create.

Of course there's alway a justification possible to refuse to implement it ("if docker's for one command, you don't use it as meant if you need supervisord", or "people should know what happens in the image") , but why not just ease the life of your users by integrating it?

I have tried to find a reasonable solution to create directories where supervisord sends the logs, but none was working (add creation of the logfile in command="mkdir blah && command") and easy and maintainable (parse the log file to create the directories before starting supervisord).

mnaberez commented 6 years ago

Of course there's alway a justification possible to refuse to implement it ("if docker's for one command, you don't use it as meant if you need supervisord", or "people should know what happens in the image") , but why not just ease the life of your users by integrating it?

I've already reopened this issue after it being closed for years, which should be an obvious sign that we're openly reconsidering it.

I wrote detailed responses above (here and here). Those need to be answered to make progress on this issue. Those questions are open to anyone asking for this change.

With regards to "why not ease the life of your users?" and similar comments, I obviously spent a lot of time replying to comments here and trying to understand this issue better. I do care about users which is why I have spent nearly ten years maintaining this software, fixing issues, and responding to the mailing list. When you send comments like that to open source maintainers, it makes it an already thankless task miserable and leads to burnout.

Edit: Reworded; I confused similar posts from two different users.

adamreisnz commented 6 years ago

In this case, the user has specified an optional logfile but supervisord can't create that logfile because the directory specified by the user doesn't exist

So that's the thing, in my case, supervisord is installed by another piece of software (Dataplicity), to monitor/start their application. I don't configure supervisord myself, so I don't specify where it should log or if it should log at all.

All I know is that it gave me a lot of grief because of a missing folder, and I had to modify our startup scripts to create the folder manually after every reboot (because it's on tmpfs).

So who's responsible in this case? You say supervisord doesn't log by default, so Dataplicity uses supervisord and (presumably) configures it to use a log file. But then Dataplicity doesn't know anything about the state of the machine it's running on, so they can't really know what log folders to create either. The user is stuck in the middle because he doesn't know he needs to create a log folder until after he encounters the error after a reboot. So it's a bit of a chicken and egg situation.

For all the other daemons that I run, I create the needed directories before I start them. You have to do work before starting the daemon anyway, e.g. you have to make a config file. Why can't the directory be created then?

I get your point and now that I understand that the log file is not a default setting, I think I understand why you don't bother creating the folder. After all if the user is specifying the folder, then they should make sure it exists. Completely agree.

So I think the problem might come in part from the fact that other software uses supervisord to start and manage their apps, and they maybe make assumptions about the log configuration and folders without being clear about it to their end users.

I still feel the simplest solution would be if supervisord would just create the folder, using sensible defaults and best practices for how this is done. I don't think the perceived issue of configuration error and then ending up with two different log folders is a biggie. That sounds like a very odd scenario with minimal impact if it does eventuate, and would be a clear user error.

felipou commented 6 years ago

@mnaberez Thanks a lot for re-considering this and for all the replies. I've been reviewing my original comments (from so many years ago) to better understand why I created this issue in the first place.

I believe my original wish was that supervisor should be able to create the directory of the log files of the programs it manages, not the directory of its own logs. And this is still something that I have to do some workarounds nowadays. Perhaps because my use of supervisor is somewhat different from expected: I update it constantly, adding and removing programs multiple times per minute.

I share many of your concerns regarding the creation of supervisor's own log directories. And come to think of it, I solved both problems the same way: for supervisor, I create the log directories in the init file (systemd nowadays), and for the programs, I use a kind of "init script" (bash) for then, and this script is responsible for creating the log directories if they don't exist (and "execs" the program after).

All things considered, I'm not really sure if this is a feature that is really required, but perhaps we should just add a little more documentation about this, since it seems to be a pain for many people.

Dayjo commented 6 years ago

In response to everyone's comments here. I would say additionally that absolutely if this is something that is difficult to achieve, simply improving the error messages and actually documenting this in various places and in the setup process, it will simplify the process for users.

The main reason I came here was that I spent a good hour re-installing, changing settings, re-reading documentation to try and work out what the problem was and why it didn't 'just work' after following the install instructions. If in those instructions it had said "If you want to customise the path of the logs, you will need to make sure that the directory first exists and has appropriate permissions", then I wouldn't have gotten stumped at all.

Folder permissions is a common issue so I don't think having a bit of documentation on this as part of the setup would negatively impact that process.

rb2nem commented 6 years ago

With regards to "why not ease the life of your users?" and similar comments, I obviously spent a lot of time replying to comments here and trying to understand this issue better. I do care about users which is why I have spent nearly ten years maintaining this software, fixing issues, and responding to the mailing list. When you send comments like that to open source maintainers, it makes it an already thankless task miserable and leads to burnout.

@mnaberez I'm happy you reconsider this request, and I appreciate your dedication to your project. My comments were always meant to be respectful and I feel my question is justified for a request with users' recurring interest since 2012.

Regarding the request itself, the problem I experienced is that supervisord complains about the inexisting log dir before starting the command. The easiest would be to let the user create the directories either by combining it in the command ( like mkdir /var/log/supervisord && mycommand), but that might have some impact on the internals, or by having a PreCommand directive for which the presence of the log dir is not required.

mnaberez commented 6 years ago

My comments were always meant to be respectful and I feel my question is justified for a request with users' recurring interest since 2012.

Negative comments like your "why not just ease the life of your users?" are never respectful or justified.

No maintainer of this project is intentionally trying to make life difficult for users. We are users of the software ourselves. There is no incentive to make life more difficult for users. It only creates more support requests for us to deal with. There is almost always another reason why a particular thing doesn't get done in a particular way other than the maintainers don't care about users. There were sound reasons for the existing behavior, which were explained in the thread already.

I'm not sure what should be done on this issue, but I agree with @adamreisnz and @Dayjo that updating the documentation to clarify the current behavior is a good first step.

rb2nem commented 6 years ago

Negative comments like your "why not just ease the life of your users?" are never respectful or justified.

As my last comment on this issue, I encourage you to read http://www.dealingwithdisrespect.com . I feel my comment is acceptable, and far from disagreeable.

mnaberez commented 6 years ago

So that's the thing, in my case, supervisord is installed by another piece of software (Dataplicity), to monitor/start their application. I don't configure supervisord myself, so I don't specify where it should log or if it should log at all.

All I know is that it gave me a lot of grief because of a missing folder, and I had to modify our startup scripts to create the folder manually after every reboot (because it's on tmpfs).

So who's responsible in this case?

I'm not familiar with Dataplicity but you may want to report this issue to Dataplicity as well.

Supervisor itself does not handle installation. It's just a Python package. It does not provide official init scripts or even a config file, it only provides a command echo_supervisord_conf that prints a sample config file to stdout. The person repackaging or installing Supervisor is responsible for creating a configuration compatible with the target system.

If I understand correctly, Dataplicity installed Supervisor for you but the way it was installed would not survive a reboot on your system. The log directories were created initially by Dataplicity but did not persist after reboot because they were on tmpfs. If Dataplicity is creating the init script that starts Supervisor, they may be able to change the init script so it checks for the directories first and creates them if needed (the same solution you did). They will also probably want to track this ticket because even if a change is made to Supervisor or a new feature is added, Dataplicity may not pick up the new release automatically if they have pinned to a specific version.

Radoslaw-K commented 6 years ago

@adamreisnz @mnaberez

Our Dataplicity installation agent downloads and installs supervisor. When it is downloaded it has all the settings as configured upstream, we don't change anything. The default directory that supervisor uses for logging can be seen in the file below. This is what you would probably want to adjust if you'd like your logs to be stored elsewhere.

cat /etc/supervisor/supervisord.conf

; supervisor config file

[unix_http_server]
file=/var/run/supervisor.sock   ; (the path to the socket file)
chmod=0700                       ; sockef file mode (default 0700)

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)

--- ... shortened output ... ---

As for Dataplicity specific logs, below is our unit file for supervisor that specifies where we keep them.

cat /etc/supervisor/conf.d/tuxtunnel.conf

--- ... shortened output ... ---

stdout_logfile=/var/log/dataplicity.log
stderr_logfile=/var/log/dataplicity.log

Alternatively if you'd like to keep the log files where they are and just make sure supervisor works with tmpfs, here's something you can add to your /etc/rc.local file:

if [ ! -d /var/log/supervisor ]; then
        mkdir /var/log/supervisor;
        touch /var/log/supervisor/supervisord.log
        chown -R dataplicity:dataplicity /var/log/supervisor;
        service supervisor restart
fi

Hope this helps, Radoslav from Dataplicity

adamreisnz commented 6 years ago

@mnaberez @Radoslaw-K

Our Dataplicity installation agent downloads and installs supervisor. When it is downloaded it has all the settings as configured upstream, we don't change anything. The default directory that supervisor uses for logging can be seen in the file below

So these are the default settings of supervisor? From what I understood from what @mnaberez wrote;

In this case, the user has specified an optional logfile but supervisord can't create that logfile because the directory specified by the user doesn't exist.

I thought Supervisor doesn't create any logs or specify any log directories by default.

So either Dataplicity must be specifying those directories, or supervisor must have them as the default. If it's the former, it probably shouldn't be done. If it's the latter, then I think the defaults might need to be changed to not have any logging at all, or to auto create the default directory.

It doesn't make much sense imo to have a default directory but then be unable to start the application because the directory doesn't exist.

Radoslaw-K commented 6 years ago

@adamreisnz

These are the default settings of supervisor, correct.

You are right that:

"It doesn't make much sense imo to have a default directory but then be unable to start the application because the directory doesn't exist."

The emonpi guys who back in the past were installing dataplicity on their system had the same problem and I think it was reported to the developers of Supervisor but their response was that it's user's responsibility to create that directory if it doesn't exist.

Best, Radoslav from Dataplicity

mnaberez commented 6 years ago

These are the default settings of supervisor, correct.

This is not correct because this project (Supervisor) does not supply init scripts or config files at all (see more above). We only provide a command, echo_supervisord_conf, that writes a sample config to stdout. I looked at your configuration above, and it does not match what is output by echo_supervisord_conf (e.g. the directories /var/run and /var/log do not appear anywhere in that sample output). You are using a configuration that was created and written to disk by someone else.

Our Dataplicity installation agent downloads and installs supervisor. When it is downloaded it has all the settings as configured upstream, we don't change anything.

The Supervisor project only provides Python packages that are published to PyPI. Our packages ares usually installed with pip. Our packages do not contain init scripts or config files. It sounds like you are installing the package from another source, e.g. maybe you installed it with a command like apt or rpm. Those are third party packages that are created by others who aren't involved with the main Supervisor project. We don't know much about these packages and we can't make changes to them. Any init scripts or config files come from them. I suggest contacting the maintainers of the third party package you are installing. They are the only ones that would be able to modify their default init script to create the directories.

barfle commented 6 years ago

Hi there, I'm just writing in to provide a little backfill here and to clarify one or two things since Dataplicity has been referenced more than once here :-)

First, thanks everyone for the renewed interest in this issue, and thanks to the supervisor devs in particular for devoting so much time and effort to the supervisor daemon that we rely on to run Dataplicity Agent on devices.

The precise issue for us is already pretty well described above. The behaviour in respect of creating the log directory if it doesn’t exist is not without precedent. In particular, nginx has similar behaviour (not creating the directory), as do a number of other mainstream services. As it happens, @adamreisnz has produced a pretty good list in the emonpi repos where every service that has this issue has had to be fudged in precisely the same way.

If I were to make a pretty wide observation, what is new now as opposed to five years ago is that logging to volatile storage is becoming increasingly common. More precisely, with the en masse arrival of IoT devices (such as Raspberry Pi) and systems such as Docker and Amazon EBS, people are increasingly relying on tmpfs for a fast logging system that doesn’t wear out flash storage. This is not a distribution level issue - Ubuntu, Arch and Slackware are all similarly affected.

In the medium term, I take the view that the use of volatile storage for logging has become sufficiently common that treating its use as a downstream, deployment related issue is no longer appropriate. If I am right, this will require a small change in every service that presently logs directly to a subdirectory of the system log directory.

In the case of Dataplicity, it affects us because SBCs including Raspberry Pi commonly use tmpfs to store logs to prevent flash wear. And on most of these systems that means /var/log is a fresh file system with no directories in it the moment it boots up. Which then means that supervisor breaks at startup because it can't access a directory that it's expecting.

For clarity: 1) dataplicity agent is the device-side daemon that's running on user devices, commonly a Raspberry Pi, to connect the device to the dataplicity service. 2) The agent uses supervisor to ensure we're running as a non-privileged user on the system, and to handle things like auto-restart of agents etc.
3) The Dataplicity installer uses apt-get on systems where it is available, which means yes, the installation process for supervisor derives from a combination of the software source (here) and the precise installation instructions baked into the apt package by the distribution people at Debian/Ubuntu etc.

There are three places we (Dataplicity) could put something in to create this directory on boot, but the only one (for us) without significant drawbacks is if supervisor was to create the expected directory itself on startup.

Tier 3 (at the dataplicity installer/agent level)

Bottom line is that it ends up a bit of a fudge that after we do "apt-get install supervisor" we then have to do "mkdir /var/log/supervisor/" just to make it work.

Tier 2 - Distribution

Tier 1 - Supervisor itself.

I hope that helps.

Elliot (Dataplicity)

mnaberez commented 6 years ago

Bottom line is that it ends up a bit of a fudge that after we do "apt-get install supervisor" we then have to do "mkdir /var/log/supervisor/" just to make it work.

Please report these same details to the maintainers of the package that you are installing. They may be able to help.

barfle commented 6 years ago

Hi there

We have a few options, but this particular one is likely to take me into retirement: making this change for all the platforms we support amounts to every distribution from red hat to arch via Ubuntu and Debian in between. And it also misses the point that out of the box supervisor doesn’t work on systems with a volatile log store.

Is there something I am missing here that presents either a particular challenge or an argument against doing so? Is it likely to cause maintenance or deployment issues of which I am unaware?

Best Elliot