No timestamp column for a custom format

huntc commented 8 years ago

Given a custom format (attached), I'm not seeing columns such as timestamp appear when performing an SQL query - in fact I'm not even convinced that I've got the custom format right. I'm raising the issue here though because the custom format looks right after reading the doc quite a few times.

Here's my command:

echo "2016-09-17T19:55:07Z ip-10-0-2-202.us-west-2.compute.internal DEBUG AgentInfo akka.tcp://conductr-agent@10.0.2.202:2552/user/reaper/agent-info - akka.tcp://conductr-beta2@10.0.2.205:9004/user/reaper/agent-monitor/conductr-agent@10.0.2.202:2552 requested agent info data."|lnav

When I go to query mode I see:

screen shot 2016-09-21 at 12 11 24 pm

...so why is there no timestamp column available?

Here's the custom json:

{
    "conductr" : {
        "title" : "ConductR's log file format",
        "description" : "The log file format for ConductR.",
        "url" : "http://conductr.typesafe.com",
        "regex" : {
            "std" : {
                "pattern" : "^(?<timestamp>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}Z)\\s(?<hostname>\\S+)\\s(?<level>\\S+)\\s+(?<logger>.+)(?<mdc>\\[.*\\])?\\s-\\s(?<body>.*)"
            }
        },
        "level-field" : "level",
        "level" : {
            "error" : "ERROR",
            "warning" : "WARNING",
            "info" : "INFO",
            "debug" : "DEBUG"
        },
        "value" : {
            "hostname" : {
                "kind" : "string",
                "collate" : "ipaddress",
                "identifier" : true
            },
            "logger" : {
                "kind" : "string",
                "identifier" : true
            },
            "mdc" : {
                "kind" : "string",
                "identifier" : false
            }
        },
        "sample" : [
          {
              "line" : "2015-08-24T08:08:02Z Christophers-MacBook-Pro.local INFO  Slf4jLogger  - Slf4jLogger started"
          },                                                                                                                                                                                   
          {
              "line" : "2015-08-24T08:08:03Z Christophers-MacBook-Pro.local INFO  ActorSystemImpl [ActorSystem=conductr] - ConductR is starting up..."
          },                                                                                                                                                   
          {
              "line" : "2015-08-24T08:08:05Z Christophers-MacBook-Pro.local INFO  ActorSystemImpl [ActorSystem=conductr] - Preparing to shutdown"
          },                                                                                                                                                        
          {
              "line" : "2015-08-24T08:08:05Z Christophers-MacBook-Pro.local INFO  ActorSystemImpl [ActorSystem=conductr] - Actor system is terminated [false]"
          },                                                                                                                                           
          {
              "line" : "2015-08-24T08:08:05Z Christophers-MacBook-Pro.local INFO  ActorSystemImpl [ActorSystem=conductr] - Shutting down actor system"
          },                                                                                                                                                   
          {
              "line" : "2015-08-24T08:08:05Z Christophers-MacBook-Pro.local INFO  ActorSystemImpl [ActorSystem=conductr] - Awaiting termination..."
          },
          {
              "line" : "2015-08-24T08:08:05Z Christophers-MacBook-Pro.local INFO  RemoteActorRefProvider$RemotingTerminator akka.tcp://conductr@127.0.0.1:9004/system/remoting-terminator - Shutting down remote daemon."
          },                                                                               
          {
              "line" : "2015-08-24T08:08:05Z Christophers-MacBook-Pro.local INFO  RemoteActorRefProvider$RemotingTerminator akka.tcp://conductr@127.0.0.1:9004/system/remoting-terminator - Remote daemon shut down; proceeding with flushing remote transports."
          },                                       
          {
              "line" : "2016-08-05T00:59:45Z 7b1767ba733f INFO  ScaleScheduler [sourceThread=conductr-akka.actor.default-dispatcher-19, akkaSource=akka://conductr/user/reaper/scale-scheduler, sourceActorSystem=conductr, requestId=2c9494b4-bb55-4f42-8ca7-89757cf2bed9, bundleId=79d74c307f29e08640ce705f39670848, scale=1, tag=conductr.scaleScheduler.scaleBundleRequested, akkaTimestamp=00:59:45.062UTC] - Scale bundle requested: scale=1"
          }
        ]
    }
}

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/37980282-no-timestamp-column-for-a-custom-format?utm_campaign=plugin&utm_content=tracker%2F449456&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F449456&utm_medium=issues&utm_source=github).

tstack commented 8 years ago

Sorry, the timestamp field gets mapped to the log_time column in the sqlite mapping. It's a bit non-obvious. You can take a look at all of the table definitions by executing the following in lnav:

;.schema

That should dump the full schema.

huntc commented 8 years ago

Ah nice. Would it be useful to show all of those columns available via .schema into the "known message fields" associated with the screen shot above?

tstack / lnav

No timestamp column for a custom format #351