search

tstack / lnav

Log file navigator
http://lnav.org
BSD 2-Clause "Simplified" License
8.06k stars 324 forks source link

Command line flags to load logs since a timestamp #1188

Open roylez opened 1 year ago

roylez commented 1 year ago

Is your feature request related to a problem? Please describe.

A lot of times I find myself loading huge log files all together, however, I am only interested in examing entries after a specific timestamp. There is no way right now tell lnav not to load all lines, and with all the extra unecessary lines it slows down lnav.

Describe the solution you'd like

Too extra command line options --after and --before which should accept a datetime argument that could be used to limit the lines that are read into lnav. The rest of lines should be skipped at loading time.

Describe alternatives you've considered

Nil

Additional context

Nil

tstack commented 1 year ago

There are :hide-lines-after and :hide-lines-before commands that you can pass from the command line with the -c option.

extra unecessary lines it slows down lnav.

Is the initial indexing slow or interactive use? The initial index can't really be avoided right now.

roylez commented 1 year ago

Initial indexing. I regularly deal with tarballs of multiple GBs, and sometimes lnav crashes/hangs during initial indexing.

tstack commented 1 year ago

Initial indexing. I regularly deal with tarballs of multiple GBs, and sometimes lnav crashes/hangs during initial indexing.

What version are you using? Can you send some of the crash logs?

If you have some time, can you try one of the artifacts attached to one of the nightly builds to see if that fixes anything:

https://github.com/tstack/lnav/actions/runs/6093182503

roylez commented 1 year ago

I do not have any crash logs right now. It does not happen everyday, but I will collect one next time.

I am using 0.11.2, will try a nightly build next time when it crashes.

roylez commented 1 year ago

Today it crashed again when loading a 8.6GB directory. I tried both 0.11.2 and nightly build, they both crashed.

Unfortunately neither version generated any crash logs. There is one crash log from May 29th, which I guess could be by an older version. I am putting it in a gist any way, and hopefully it has what you are looking for.

crash

tstack commented 1 year ago

Today it crashed again when loading a 8.6GB directory. I tried both 0.11.2 and nightly build, they both crashed.

Sorry, that sucks. Thanks for trying out the nightly build.

I would like to try to figure out what the problem is and get it fixed. Since I can't seem to replicate it, I would need some help from you. One option is for you to send the files to support@lnav.org , but they probably have sensitive data. You can get a debug log from lnav by passing -d /path/to/lnav.log on the command-line and then sending that. You could try starting lnav and then attaching to it with gdb/lldb, then you might be able to get a backtrace when it crashes (but, you would probably need to build from source to get reasonable line numbers).

roylez commented 1 year ago

It is probably a memory issue.

The machine I use to load log files only had 8GB of memory, and after increasing it to 16GB, I have tried loading the directory twice, it crashes no more.

Still, it would be great if we could filter out some data at loading time and save some memory.

graceman9 commented 4 months ago

@tstack Unfortunately the flag :hide-lines-before doesn't work exactly as I expect it to. I also have a file ~5Gb and it takes quite long time to load. The option does not reduce load time. Moreover, the search also works throughout the entire file despite the option. Is it the only option to limit lines?

tstack commented 3 months ago

Moreover, the search also works throughout the entire file despite the option. Is it the only option to limit lines?

@graceman9, can you give some more details? Do you see log messages with timestamps before the time passed to :hide-lines-before? The search should only be happening for displayed lines.

graceman9 commented 3 months ago

@tstack My bad, probably I used this option incorrectly. Tried again - works great! Thanks lnav -c ":hide-lines-before 2024-07-25" var/log/php.log

stokito commented 2 months ago

Speaking more generally it would be nice to have same set of filtering options as tail and journalctl have:

-S --since=DATE Show entries not older than the specified date -U --until=DATE Show entries not newer than the specified date -n --lines[=[+]INTEGER] Number of journal entries to show -f --follow Follow the journal

And there are more options (check journalctl -h) but this options are definitely the most used. My main interest is to use the lnav as a highlighter for plain file logs i.e. as a direct replacement for tail -f.

piotr-dobrogost commented 2 months ago

Is the initial indexing slow or interactive use? The initial index can't really be avoided right now.

When using :hide-lines-before and :hide-lines-after lnav could establish starting point using binary search together with reading small chunks of file and index content only till reaching the ending point. This would allow to work with very big files exceeding the size of available memory.

FaffeF commented 2 months ago

I agree. I have previously exceeded the file number limit of lnav when opening a directory full of daily logs from many different sources. Being able to specify the date range of the log events I'm interested in would spare me a lot of file globbing.

piotr-dobrogost commented 1 month ago

I haven't thought about this case but yes, if lnav can see there are no entries from given file in the specified datetime range then it can disregard this file and close it reducing the number of simultaneously opened files. However it should keep information about such files in case user changes the range later so that it could reopen them and check if there are entries in the new range specified by user.