alidisi
commented
5 years ago @double0331 have you ever resolve the problem, can show the way to resolve the problem, i will appreciate that
Open double0331 opened 7 years ago
alidisi
commented
5 years ago @double0331 have you ever resolve the problem, can show the way to resolve the problem, i will appreciate that
kujon
commented
4 years ago Did anyone manage to figure out a workaround?
double0331
commented
4 years ago I change the avro file to txt file and then use the logstash...
netsplit
commented
3 years ago I experienced the same in combination with Confluents Schema Registry and messages that were encoded with Confluents Avro Wire Frame format. Using https://github.com/revpoint/logstash-codec-avro_schema_registry instead of the default plugin solved this problem for me.
For all general issues, please provide the following details for fast resolution:
Version: 5.4.1
Operating System: linux uname -a Linux dssubuntu02 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Config File: dssqa@dssubuntu02:~/lisa/logstash-5.4.1/config$ cat avro_to_es.conf
For detail structure of this file
Set: https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
input { file { path => "/home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro" type => "avro" codec => avro { schema_uri => "/home/dssqa/lisa/inputData/schema.avsc" } start_position => "beginning" sincedb_path => "/home/dssqa/lisa/lisatest.sincedb" } } filter {
Only matched data are send to output.
} output { stdout{ codec => json } }
-Schema file: cat schema.avsc { "type": "record", "name": "extraction", "fields": [ {"name": "schedule_name", "type":["string","null"]} ] }
Sample Data: I just use a company levle .avro file from kafka as input.
Steps to Reproduce: run ./bin/logstash -f config/avro_to_es.conf --debug and then below shown in debug mode [2017-06-22T10:21:44,096][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2017-06-22T10:21:44,100][DEBUG][logstash.inputs.file ] each: file grew: /home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro: old size 23554, new size 1259334 [2017-06-22T10:21:44,102][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro", :text=>"\xA9a\x11C\t\x0E61\x00\xB5\x9F\x0Ee\x15V\xAB\x05!\x16\x00i\x0E\r\x99\x18setSwap\x89\xB7\x050\xCD\x1FN\xF8\x05:\x16\x15\bLPC\x01\x12\x12 \x11\x00,.\x0E\x006\x8F\x00\x1D'\x04An\xAD!!j\x16&\x0F\"\xA2\x14J\x13\x00\x82\x9CI\x15=\bTen\x12\xD08>U\x00\x1E\xA9\x14:l\x00\r\xE4\x15?>\x1E\x15>0\x00\x1E}\x15\x11\x18\x12^r54\x19\x16\xAA\x11\x9A\xA5U>)\x03F\x94P\x04Bi:\xDF\x96\r\x11:\xE9\x02\x00BF\x16\x97F\QFC\x06\xADA:T\x04\x00B\x0E\x91O\xC5\x83Ag\x11\xDE\x19\x15>|\x00\x01\x1A\"\xE80./\x00\t\x15\x11/\x10Bonusm\x1A\x05\f\x10Extra\x0E\x99\x13\x01g\x05\x12\x14OldHol\x0E\x87R]\x0E\x14rokerL\x16\x8D;\r^\x1Aw \x0E\xCFN\x19)\x12\x13O:\xE5N\x10Adfin:\x19\e2\x14\x00\x0E\xD0\x184sCESREEARegula\x16\x83\xC9\x16\xEB\x90%\xD0\bNPV\x0E\xB4\x16.\x12\xDD\x19\r\xAD\xF6\x18ageOfPa\x0E#\e\x11\xA3\x0E\x12\x12\x00DB\xE2\x01\x05\x171\x12\x1D+\"\xC8\xF5\e\x05B7\x05\x04Co\x0E\xB6\xC9ao\x16~\xE46$T\x0E\xB9\x19\x1E\xD0\xAA\x12u\t:o\xA7\x11\x14m\x00\x05\x0F\x10ibute\x162\x15!\x88\x15\x14@\xFA\r\xA8N\xAC\t\r\e\x1D6\xBD2\x1D\x16\x01r\x1A\xAA\x19J\x16\x00\x851\x1D\xFB\x00u>3\x9A\xBA\x12\x00\t/\"\x8B\xCB\r\xA5\x9B\xCC\"\x9A\x1EA\x90\bMov\x0E<\x14\x0EZ\x1D\x12\xAC\x9E\x05\x8E>1Ua\xB0\x14ayToDa\x16\x01\xDF>\xE8\a> \x00M\x19\x15'\x0E\x02\x1C\x00TR]S\x1A\xCFI\x10Model9\x04.\x16\x00R\x91\x00\r!\bSpe\x12\xF8\x18\x10Direc\x0E\x8BZ\x04ceu\xBB\x12\x9F\tI\xBF\x05\v!\b\xC1\x15\x05\b\x1A+\x9F\x01\f\x16\xF4\bQ\xAD\x95\xD8\x06 \x11\v\x1A"} [2017-06-22T10:21:44,106][DEBUG][logstash.inputs.file ] observe_read_file: general error reading /home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro - error: #<NoMethodError: undefined method
type_sym' for nil:NilClass> [2017-06-22T10:21:45,107][DEBUG][logstash.inputs.file ] each: file grew: /home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro: old size 23554, new size 1259334 [2017-06-22T10:21:45,107][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro", :text=>"\xF0BeAxYpTDhLAMm_Fifp_L9yQ aVnOgTYw51qPYVmPGZzZ8Q jP4scY8GfktsHhdA9m8si\xC9\xB18\xB0\x9D\xE7\xAF\xF4V\xF0\xA4\xAB\xD7\xF4V\x02\xC6\xAF\x05\a!2\b916\xC1\xB1\x18\f144544\x15\b)\xA3\x00\x90\"\xD3\x16-\x87\xA1\xCCF\xA3\tBg\x06\r.n\xB5\x01\r\x8A\x1D\x83)\xB5\x81\xCC)\xB5\x00&)\xB5\t\r\x00\x00\x16d.>\xB5\x01\b,04&%0\b2:1\x0E%0\b PM)\xEA%\xB8\x000\"\xCA\x1A\x12\xA3\x11462x4\x02|\x02\x06\x81\x95C\x8Bl\x1A2U\bJ\xB4\x01.\x13\x00\x12\xA0\x16\xB5\xD7\tbAL\xF0F\x90\x01J2ehcMt1pcJaotUIUDeQAw 5_qpI-PcFrgr3FvI7K-mjw MbZyqh4llh0xn_BILXWCXg \x01\xF84\xA0\x92\xAC\xD7\xF4V\xF0\x8D\xEB\xE0\xF4V\x02\xF2\t\a\x12N\x15\f6492\x16\x93\x11\f8014\x12U\b\x05\t)\xA6\x04\xB4\t\x12T\x0F&\x99\x0EiL6\xD0\x0E\x1A\xAD\x0E\x16)\x0F-\xA0\x12"} [2017-06-22T10:21:45,110][DEBUG][logstash.inputs.file ] observe_read_file: general error reading /home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro - error: #<NoMethodError: undefined methodtype_sym' for nil:NilClass> [2017-06-22T10:21:46,110][DEBUG][logstash.inputs.file ] each: file grew: /home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro: old size 23554, new size 1259334 [2017-06-22T10:21:46,111][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro", :text=>"\x16\xDC4TeIrXMia0UJ9-4c7EAdCutA^\x837PMIpBsC7mtPAc-BYF_rlve\x89W$\xF0\x83\x9E\xDF\xF4V\xB0\xF2\xDC\xC5\x0E?<\x00\xF5\x05\a!\x1D\x1086281\x16\xD52\xFE\xE4\x06\xFE\xE4\x06\xFE\xE4\x06\xFE\xE4\x06\xFE\xE4\x06n\xE4\x06-y=rY\x8F\x16\xD4\f\x8E\x82\x0E\xDD\xE4\x1090522\x89\xD4\x12~\x0E\b1.2\xCD\xE4I\x8F\x009\x12\x15y\x04\xC6\x04\x16\xE8)\x04\x95\x1As\t\x002\x1E\xF4'.\x8F\x02J\xE6\x06\xADr.\x8F\x02PG9wVJ-YGRMVdIXYpX9sWdb6aP7XhpLIqxOVVmtvKuspHX\x16\xE5\x118\x90\xD0\xDD\xC5\xF4V\xC0\xC5\xD2\xD6\xF4V\x02\xDA\xD2\x05\a!\x1D\f9099\x12\x97P\f6062\x0E7\v\t\b\xC9\xE6\x00\x80\"\xE5\x11\x1At\x0E\"X\x1C\x16+\x1C\x00,&\x95\x0E0MNACompanyRol: :.\xD5m0MNADealCancel\x1A\x14\r\x11\x14\blos\x1E'\r\r\x13>[\x16\r\x17\x00S\x1E\x9ET2\x10\x006Z\x12\x11\e\x16\x13\x16\x1A\":6\x15\x00\x1D16 \x00\x00T\x16\xA8{\r.\x01\x0E\x1D+\x0E\xBC\x1F$NAExpected:\xB5\x00\x11\x17J\xB9\x00\x00I\x12X4\x0E\xAB \tB\x04Is\x11-\x05D>\x15\x00RB\x00\x10Finan\x0E*T\x11.\x18Success\x12\x12 B\xA4\x00\x10Party\x16\xC1\f9\x9F\x1A\xCD;\x19\x11!\xB0\tO!\x996w\x00F\xCC\fMB];iX\x0Ew2eX\x04FQ\xA9\xE7\x12\xF7\x0F\xA1TN\xBB].\xAB\b\x0402\x12X\x12g\x19\x12\xAB\b\xA1\xEE\x10\b12.8\x1A/P\x12\x8Fn$9279x28\x02\xE2a\x16AR\x14tP@\x02pDR4"} [2017-06-22T10:21:46,114][DEBUG][logstash.inputs.file ] observe_read_file: general error reading /home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro - error: #<NoMethodError: undefined methodtype_sym' for nil:NilClass> [2017-06-22T10:21:47,114][DEBUG][logstash.inputs.file ] each: file grew: /home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro: old size 23554, new size 1259334 [2017-06-22T10:21:47,115][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro", :text=>"j~K\x1054206\x16e\x11\x1A~K2\xEA!\x002\x12\xE9&\x000\x1Ae\x11\x00\xE4\xCA\xC6\vM\xE1.\xA8\x02Tu9weTPQwow4pqiCFSIWmkA^\xC6\vT2omeCgcO6_gtaYcnWfSemw\x12\x03Z)\x0E\x04\xF0\x8D\x12\xE2&\x04\xB4\x96\x05\a%\e\x0E\xE9&&\xE1&\x1E\xF1&\xE9\xC4\xFE\xE9&\xFE\xE9&\xFE\xE9&\xFE\xE9&\xA6\xE9&-Y=RYmA\\\b\x00@M.m\x02Z\x8Ft\x00\"\x0E\xD3\x13\x16B\xA1\x182.46.05Io\x04\xD3g\x0E\x8At\x004\x0Et\f\x12\xD0D\x00\x18\x0E5\x0E\x1014734\x0E\xEB&\b\x9C\xE6\x01\x16\xEC&\x04B32z\f\xBD"} [2017-06-22T10:21:47,117][DEBUG][logstash.inputs.file ] observe_read_file: general error reading /home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro - error: #<NoMethodError: undefined methodtype_sym' for nil:NilClass> [2017-06-22T10:21:48,118][DEBUG][logstash.inputs.file ] each: file grew: /home/dssqa/lisa/inputData/us1s-dswsext01-20170425.avro: old size 23554, new size 1259334 . . . .