Leaf doesn't write log files

[artgoldberg]

Hi Folks

Without warning the Leaf API has stopped writing log files. Stopping and restarting Leaf doesn't create a log file.

This is weird and incomprehensible. It’s also hard to debug, as we don’t have logs of exceptions that would be raised when Leaf cannot write to a log.

I don't think Leaf's code or its configuration (running on Linux) has changed, although I'm check with other people who have access to the server.

SERILOG_DIR=/var/log/leaf/ is still defined in leaf_api.service.conf. Thoughts on how to debug?

Arthur

[ndobb]

That's extremely odd; I can't say we've run into that. @mh2727 any ideas? Any other Apache logs we could look at?

@artgoldberg, is the Leaf application itself still up, but is just not logging? Also just to knock out obvious possible issues, is there enough space on disk? And write permissions on the log directory haven't changed?

[artgoldberg]

Yes, x-odd. Yes, Leaf is working, and not logging. And yes, there's plenty of disk space. Regarding write permissions on the log directory, I've not changed them and I've asked other folks who've been working on the server whether they have. Can it be configured to log to stdout and a file?

[mh2727]

Hi @artgoldberg.

Did you find out from the others if they made any changes? Also, you mentioned restarting the API, did you attempt to compile again to see if that resolves the issue?

Another item to check might be the log rotation settings on your machine. It could be that the files have moved and so it's still writing but we're not looking at the right place. Here's more details for log rotation: https://www.tecmint.com/install-logrotate-to-manage-log-rotation-in-linux/.

Lastly, how is the .Net service running? Did you set it up in a container or is it running as a SystemD service? If latter, then you can use the journalctl command to get more details.

Hope this helps.

-- Mehadi Hassan

[artgoldberg]

Thanks @mh2727

I've returned from taking Thurs. & Fri. off. And I find that the Leaf logs started working again on Fri. Dec. 10, which is good. But I don't know why, which is troubling. I'll investigate.

In response to your questions

• No, I did not compile again.
• I doubt the files moved, as I searched the entire file system for files with the right name pattern and didn't find any in the wrong directory.
• Yes, I'm using SystemD. I'll try the journalctl command.

I see that Leaf also writes messages* logs in /var/log. How do they compare with the leaf-api-<date>.log files? In particular, could I use messages* logs to recreate parts of missing logs in /var/log/leaf?

Thanks Arthur

[mh2727]

Hi @artgoldberg

The log files in /var/log/messages are the same as the leaf-api-<date>.log files. The difference is in the additional configs for the serilog files. More details on the differences can be found here: https://github.com/uwrit/leaf/blob/master/src/server/API/Program.cs#L32.

And yes, it should be possible to recreate the missing logs.

Leaf writes to stdout as well as to the serilogdir path. Standard output generally writes to /dev/null or the journal files which can be inside the unit file dir or /var/log/messages. This is why you're seeing Leaf write to /var/log.

re: log files moved; I'm still unclear on what happened here. When you are recreating the logs, perhaps you can take a look in /var/log/messages and see if there are any errors/something unusual which caused Leaf not to write to serilogdir path.

Hope this is helpful and answers your questions.

-- Mehadi Hassan

[artgoldberg]

Hi @mh2727 Thank you very much. Arthur