nielsbasjes
commented
4 years ago Yes, we talked about it (I think it was Barcelona) and I really tried to create one.
I have created it for my other project https://yauaa.basjes.nl/UDF-ApacheNifi.html but for this log parsing library it was harder.
The main hurdle I was unable to take is that the configuration of this logparser is a two step thing.
1) First dialog: You specify the access logformat (and optional extra dissectors and mappings).
- The system then analyzes this and gives a list of possible output values.
- Some of those values are wildcards (like for extracting the name of a cookie or query parameter). 2) Second dialog: Show all of these possible values and allow editing (i.e. wildcards and selecting which fields are wanted) and specifying the name of the attributes the results must be stored in. In case of a wildcard you can either leave it (so you get a map / key-value structure) or you specify one or more explicit values (so you get only those).
What I remember from Andy's talk in Barcelona is that this should be possible.
Yet I have not been able to find an example that do something this. Note that my frontend (javascript/html/css and such) skills are too limited to figure this out myself.
So if you can give this a try then please do so. I'm fine with such a processor being either in this project of being a native part of Nifi itself. Note that this library is already bundled with Apache Pig and Apache Drill, so adding Nifi to this list seems like a good idea.
Is there a NiFi processor? I thought last time we talked, you had one. If not, maybe I'll take a look. Always love your libraries.