ntop / nProbe

Open source components and extensions for nProbe
http://ntop.org
GNU General Public License v2.0
1.64k stars 44 forks source link

Problem with exporting HTTP info to Elasticsearch [failed to execute bulk item] #45

Closed lzalewsk closed 8 years ago

lzalewsk commented 8 years ago

Hi

nProbe v.7.3.160313 (r4932) for x86_64-unknown-linux-gnu
with native PF_RING acceleration.
Copyright 2002-16 ntop.org
Build OS:      Debian GNU/Linux 8.2 (jessie)
SystemID:      3C0E6232B206AB23
Edition:       nProbe Pro
[...]

With active HTTP module and nprobe.conf option where -T=%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %OUT_PKTS %L4_DST_PORT %L4_SRC_PORT %IN_BYTES %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %PROTOCOL %IN_SRC_MAC %OUT_DST_MAC %TCP_FLAGS %APPL_LATENCY_MS %SRC_IP_COUNTRY %DST_IP_COUNTRY %L7_PROTO %L7_PROTO_NAME %ICMP_TYPE %IP_PROTOCOL_VERSION %NUM_PKTS_UP_TO_128_BYTES %NUM_PKTS_128_TO_256_BYTES %NUM_PKTS_256_TO_512_BYTES %NUM_PKTS_512_TO_1024_BYTES %NUM_PKTS_1024_TO_1514_BYTES %NUM_PKTS_OVER_1514_BYTES %DNS_QUERY %DNS_QUERY_TYPE %DNS_RET_CODE %DNS_NUM_ANSWERS %DNS_TTL_ANSWER %DNS_RESPONSE %HTTP_URL %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST %HTTP_SITE %DOT1Q_SRC_VLAN %DOT1Q_DST_VLAN %SRC_IP_CITY %DST_IP_CITY

During exporting data to Elasticsearch there are a lot of notes / errors in ES logs.

[2016-03-13 22:39:01,029][DEBUG][action.bulk              ] [node-1] [nprobedev-2016.03.13][0] failed to execute bulk item (index) index {[nprobedev-2016.03.13][flows][AVNx6wUkGYu_AAvmVAjQ], source[{"IPV4_SRC_ADDR":"5.134.213.61","IPV4_DST_ADDR":"77.114.78.4","IN_PKTS":6,"OUT_PKTS":1,"L4_DST_PORT":49837,"L4_SRC_PORT":80,"IN_BYTES":4452,"OUT_BYTES":467,"FIRST_SWITCHED":1457905119,"LAST_SWITCHED":1457905120,"PROTOCOL":6,"IN_SRC_MAC":"50:C5:8D:1E:8F:C4","OUT_DST_MAC":"02:03:00:11:98:00","TCP_FLAGS":18,"APPL_LATENCY_MS":17.310,"SRC_IP_COUNTRY":"PL","DST_IP_COUNTRY":"PL","L7_PROTO":7,"L7_PROTO_NAME":"HTTP","ICMP_TYPE":0,"IP_PROTOCOL_VERSION":4,"NUM_PKTS_UP_TO_128_BYTES":3,"NUM_PKTS_128_TO_256_BYTES":0,"NUM_PKTS_256_TO_512_BYTES":2,"NUM_PKTS_512_TO_1024_BYTES":0,"NUM_PKTS_1024_TO_1514_BYTES":3,"NUM_PKTS_OVER_1514_BYTES":0,"HTTP_URL":"img16.staticclassifieds.com/images_tablicapl/350409411_1_261x203_grubosciowkastrugarkaheblarka-zywiec.jpg^Q�Dj^?","HTTP_METHOD":"GET","HTTP_RET_CODE":200,"HTTP_REFERER":"olx.pl/dom-ogrod/narzedzia/q-grubo%C5%9Bci%C3%B3wkaotwierdza-plotki-zrobila-to-ku-przera,nId,2161801-m-n-ts-rd-u-co-re-r-kg-g.","HTTP_UA":"Mozilla/5.0 (Windows NT 6.2; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0iowkastrugarkaheblarka-zywiec.jpg^Q�Dj^?","HTTP_MIME":"image/jpeg-ogrod/narzedzia/q-grubo%C5%9Bci%C3%B3wkaotwierdza-plotki-zrobila-to-ku-przera,nId,2161801-m-n-ts-rd-u-co-re-r-kg-g.","HTTP_HOST":"img16.staticclassifieds.comq-grubo%C5%9Bci%C3%B3wkaotwierdza-plotki-zrobila-to-ku-przera,nId,2161801-m-n-ts-rd-u-co-re-r-kg-g.","HTTP_SITE":"staticclassifieds.comNT 6.2; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0iowkastrugarkaheblarka-zywiec.jpg^Q�Dj^?","DOT1Q_SRC_VLAN":1480,"DOT1Q_DST_VLAN":1480,"SRC_IP_CITY":"Grupa","DST_IP_CITY":"Warsaw","@version":"1","@timestamp":"2016-03-13T21:38:39Z", "EXPORTER_IPV4_ADDRESS":"10.221.1.44"}]}
MapperParsingException[failed to parse [HTTP_URL]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 17)): has to be escaped using backslash to be included in string value
 at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@1ebbe99e; line: 1, column: 749]];
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:343)
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:318)
        at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:445)
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:271)
        at org.elasticsearch.index.mapper.DocumentParser.innerParseDocument(DocumentParser.java:131)
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:79)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:304)
        at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:500)
        at org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:481)
        at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:214)
        at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:223)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:595)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:263)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260)
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 17)): has to be escaped using backslash to be included in string value
 at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@1ebbe99e; line: 1, column: 749]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1581)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)
ValentinaViscarelli commented 8 years ago

I have made a fix. Now everything should be fine. Please give it a try.

lzalewsk commented 8 years ago

Hi, Confirm. Works.