add record_type autodetection based on a sample

Hurence / logisland

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.

https://logisland.github.io

Other

110 stars 28 forks source link

add record_type autodetection based on a sample #143

Open oalam opened 7 years ago

oalam commented 7 years ago

given a sample of log lines or raw records (100 or more), this processor should give back an avro schema and a Processor parser config (regex pattern and son on)

first detect log nature (raw, json, xml)
then infer pattern & schema

oalam commented 7 years ago

https://www.loggly.com/docs/automated-parsing/