is it possible to use S_UNIXTIME in network source?

[GoogleCodeExporter]

This is just a question, not an issue. 

What is the expected output? 
Have ELSA to store and use the original event timestamp (S_UNIXTIME) instead of 
the received timestamp (R_UNIXTIME). This comes handy for syslog forwarders 
that perform local buffering in case of lack of connectivity, etc.

Is this possible at all? I mean if I go and change R_UNIXTIME for S_UNIXTIME in 
the network source template, will it break something inside ELSA's logic? 

If this is not possible at the moment, would the best approach (by now) be to 
rewrite the S_UNIXTIME as part of the message, modify patterndb to catch the 
timestamp, and then create a new field and store the timestamp value as a 
string (to avoid losing the original timestamp)?

Thx!

Original issue reported on code.google.com by rhatu...@gmail.com on 27 Feb 2013 at 12:04

[GoogleCodeExporter]

Sure, you can make that change with no problems.  R_UNIXTIME is defaulted 
because it is guaranteed to be correct (cannot be misparsed or tampered with).  
The import script uses S_UNIXTIME already, so this is well tested.  If you want 
to make the manual change, you should add the following to /etc/elsa_vars.sh:

USE_LOCAL_SYSLOG_NG_CONF="1"

Put the new template to your local customizations file 
(/etc/elsa_syslog-ng.conf) to ensure that your changes will be preserved across 
upgrades to the node.

Original comment by mchol...@gmail.com on 27 Feb 2013 at 4:25

• Changed state: Done
• Added labels: Type-Other
• Removed labels: Type-Defect

[GoogleCodeExporter]

That's great news, thanks a lot!

Original comment by rhatu...@gmail.com on 27 Feb 2013 at 6:36

[GoogleCodeExporter]

Sorry, I'm back with some doubts. I was going through the code to understand 
the impact of this change, and I can use your help to double check something. I 
had the impression that at some point ELSA was using the timestamps in a query, 
and found this code:

        $query = sprintf("SELECT MIN(id) AS min_id, MAX(id) AS max_id FROM %s", $full_table);
            $sth = $self->db->prepare($query);
            $sth->execute();
            my $row = $sth->fetchrow_hashref;
            ($min, $max) = ($row->{min_id}, $row->{max_id});
            $query = sprintf("SELECT FROM_UNIXTIME(timestamp) AS timestamp FROM %s WHERE id=?",
                $full_table);
            $sth = $self->db->prepare($query);
            $sth->execute($min);
            $row = $sth->fetchrow_hashref;
            $start = $row->{timestamp};
            $sth->execute($max);
            $row = $sth->fetchrow_hashref;
            $end = $row->{timestamp};

Now, I'm not sure this is truly a problem, from just from here I understand 
that ELSA is assuming the min_id would be the oldest timestamp and that max_id 
would be the newest event. 
If we start using S_UNIXTIME, this scenario is no longer valid... or not? :-)

Thanks!

Original comment by rhatu...@gmail.com on 27 Feb 2013 at 6:55

[GoogleCodeExporter]

This is a fair point.  I've just released code that will address this by
always tracking the earliest/latest timestamps for a batch and updating the
directory appropriately.

On Wed, Feb 27, 2013 at 12:55 PM, <
enterprise-log-search-and-archive@googlecode.com> wrote:

Original comment by mchol...@gmail.com on 27 Feb 2013 at 9:28