Logging for apps

[nikochiko]

For CGI apps, a simple logging hack could be to set sys.stderr to a file handle.

For logs of deployment, and pre/post-deploy scripts, we would need to decide on conventions. Maybe the same file, or a different file.

[anandology]

how many logs are we going to have and where will they go?

Also, keep in mind that we are using Apache as the webserver as of now, but that may change eventually.

[nikochiko]

I think I'll have to look at how Heroku and others do it to get a good answer.apa

But I feel it should be good enough to have a single log file. It might not be enough to just take logs from stderr of apps, because static sites are served directly by web server and a separate arrangement would be needed for access logs.

We can define a convention for logs (a format and order), and the webserver would need to conform to that.

Also, keep in mind that we are using Apache as the webserver as of now, but that may change eventually.

I had similar concerns for some other issue and found that there are projects to convert configuration between popular web servers (apache/nginx/caddy)

[nikochiko]

Logs should go into $APP_ROOT/logs.

Users can add multiple loggers in their apps if they want and store this here. The logs created by hamr could go into a main.log file or app.log file.

[anandology]

Users can add multiple loggers in their apps if they want and store this here. The logs created by hamr could go into a main.log file or app.log file.

See https://12factor.net/logs

[anandology]

Start with identifying the multiple sources of logs and put all of them in a single directory. Redirecting all of them to a single file, might be tricky and I wouldn't recommend you to get to that right now.

[nikochiko]

Logs could be coming from:

• deployment (git pull) and deployment scripts
• access to deployed sites (maybe doesn't have to be logged, but mentioning here for completeness)
• stdout from the application that doesn't go into CGI
• stderr from the application (maybe stdout/stderr for an app can go to the same file)
• cron jobs if they are added

primarily, as of now: deploy logs, and app logs (stdout/stderr combined)

[nikochiko]

Friday and today I fiddled around with different ways to do this in apache - which would be the ideal way to perform logging because we are not touching the CGI script when the data can be served statically. But I haven't found a way to do this in a dynamic fashion - depending on the subdomain.

It is possible to have a separate AccessLog and ErrorLog for each app, and it is possible to have macros in apache. That way we can define configuration of many apps and still be make changes that apply to all by editing only a single template.

[nikochiko]

As for the deployment logs - these should include the logs during the git pull and of the post-build script.

[nikochiko]

Update:

I was able to get this to work by using a configuration like this:

CustomLog "| ${HAMR_DIR}/split_logs.py ${HAMR_DIR}" our_custom_log_format_starting_with_hostname

Where the logs are piped to the python process like in a shell.

And in split_logs.py:

split_logs.py reads each log, assumes that the first word in it is the hostname, grabs the subdomain, and then writes logs to that file.

#!/home/kaustubh/hamr/venv/bin/python

import os
import sys

def get_log_file(hamr_dir, host):
    subdomain = host.split(".", maxsplit=1)[0]
    return os.path.join(hamr_dir, f"apps/{subdomain}/logs/access.log")

hamr_dir = sys.argv[1] if len(sys.argv) > 1 else ""

for line in sys.stdin:
    host = line.split(maxsplit=1)[0]
    with open(get_log_file(hamr_dir, host), "a") as f:
        f.write(line)

This could work, and we could give the access / error / deploy as an argument to save to the correct file.

The pros of this approach being that we don't have to give up on the simple "serve this hamr directory" route and add an extra build step for adding to the apache config.

[nikochiko]

21 has support for access.log and error.log. deployment log is also logged to error.log as its redirected to stderr.

for each app, the requests and the stderr can be viewed there. I had to do some troubleshooting to get it to work for ErrorLog (this thread: https://bz.apache.org/bugzilla/show_bug.cgi?id=14090 - last message). We should add that to our deployment instructions or add it in a troubleshooting section.