jtmoon79 / super-speedy-syslog-searcher

Speedily search and merge log messages by datetime
MIT License
43 stars 2 forks source link

chained block reads #14

Open jtmoon79 opened 2 years ago

jtmoon79 commented 2 years ago

Problem

Currently, only one "depth" of compressed or archived file is supported. e.g. can read syslog stored in logs.tar. Cannot read syslog.gz in logs.tar, nor logs.tar stored in logs.tar.xz. e.g. can read syslog stored in syslog.gz. Cannot read syslog.gz stored in syslog.gz.xz. Cannot read a the special gzip+tar file logs.tgz.

Related, only plain text files are extractable from compressed files or archived files. EVTX, Journal files, and utmp files stored as a compressed or archived file are not readable. See FileType

Solution

Refactor BlockReader reading to handle arbitrary "chains" of reads for text files and UTMPX files.

Currently, JournalReader reads Journal files using libsystemd calls for reading. BlockReader is not used by the JournalReader. Processing Journal files that are compressed or archived are outside the scope of this issue.

Currently, EvtxReader reads EVTX files using EvtxParser. BlockReader is not used by the EvtxReader. Processing EVTX files that are compressed or archived are outside the scope of this issue.

Relates to Issue #7.

jtmoon79 commented 2 years ago

This is a decent amount of work. Additionally, if done well enough, it could be it's own separate rust library.

jtmoon79 commented 2 years ago

Also touches Issue #13