search

influxdata / influx-cli

CLI for managing resources in InfluxDB v2
MIT License
61 stars 22 forks source link

influx write incorrectly parses input data when --skipRowOnError is enabled #502

Closed lorinescu closed 3 months ago

lorinescu commented 1 year ago

I am trying to import an LP file exported from influxdb v1.8 into v2.7. When I use --skipRowOnError the input data is not correctly parsed. When I don't use this flag everything works as expected. I believe the problem was introduced in v2.4.0 when --skipRowOnError was enabled for LP format. What I think happens is that when the LineReader buffer fills up the data between the last \n in the buffer and the end of the buffer is not read by LineProtocolFilter.

Example:

# cat bug-report-data
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10149335861206055 1680843957087735646
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10599446296691895 1680844070301214389
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11674189567565918 1680844094394325888
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.12114214897155762 1680844139344799828
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11209964752197266 1680844301203178190
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1039726734161377 1680844347876425153
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1268465518951416 1680844370106414825
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.12303543090820312 1680844646311648005
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11537313461303711 1680844695335169551
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10717368125915527 1680845112709821861
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1027822494506836 1680845196950133820
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.12188148498535156 1680845320198611842
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10436582565307617 1680845335147764695
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11761021614074707 1680845473279430934
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10162472724914551 1680845611228521315
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.12307000160217285 1680845680031549517
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11197781562805176 1680845736813021569
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11208724975585938 1680846223638448981
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.0956425666809082 1680846370123147298
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1171119213104248 1680846439166725830
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1193227767944336 1680846639652055730
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11297607421875 1680846714839454444
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11377334594726562 1680846991135713316
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10156655311584473 1680847055803239552
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.09974861145019531 1680847194100005178
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1182551383972168 1680847198099239798
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680843957087735646
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844070301214389
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844094394325888
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844139344799828
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844301203178190
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844347876425153
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844370106414825
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844646311648005
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844695335169551
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845112709821861
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845196950133820
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845320198611842
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845335147764695
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845473279430934
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845611228521315
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845680031549517
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845736813021569
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846223638448981
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846370123147298
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846439166725830
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846639652055730
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846714839454444
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846991135713316
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680847055803239552

# cat bug-report-data.txt | nflux write dryrun --org foobar --skipRowOnError --precision ns --format lp --bucket foobar --token xyz 2> err 1> out

# cat err
2023/04/24 09:05:30 invalid point on line 25: unable to parse 'celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.ce': missing fields
2023/04/24 09:05:30 invalid point on line 50: unable to parse 'celery_task_duration,env=MOOO,host=foobar-batch,queue=high,servic': missing tag value

# cat out
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10149335861206055 1680843957087735646
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10599446296691895 1680844070301214389
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11674189567565918 1680844094394325888
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.12114214897155762 1680844139344799828
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11209964752197266 1680844301203178190
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1039726734161377 1680844347876425153
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1268465518951416 1680844370106414825
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.12303543090820312 1680844646311648005
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11537313461303711 1680844695335169551
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10717368125915527 1680845112709821861
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1027822494506836 1680845196950133820
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.12188148498535156 1680845320198611842
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10436582565307617 1680845335147764695
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11761021614074707 1680845473279430934
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10162472724914551 1680845611228521315
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.12307000160217285 1680845680031549517
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11197781562805176 1680845736813021569
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11208724975585938 1680846223638448981
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.0956425666809082 1680846370123147298
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1171119213104248 1680846439166725830
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1193227767944336 1680846639652055730
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11297607421875 1680846714839454444
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.11377334594726562 1680846991135713316
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.10156655311584473 1680847055803239552
lery_throughput_high duration=0.09974861145019531 1680847194100005178  <-------truncated
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration=0.1182551383972168 1680847198099239798
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680843957087735646
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844070301214389
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844094394325888
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844139344799828
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844301203178190
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844347876425153
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844370106414825
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844646311648005
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680844695335169551
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845112709821861
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845196950133820
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845320198611842
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845335147764695
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845473279430934
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845611228521315
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845680031549517
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680845736813021569
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846223638448981
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846370123147298
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846439166725830
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846639652055730
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846714839454444
celery_task_duration,env=MOOO,host=foobar-batch,queue=high,service=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680846991135713316
e=BATCH,task=monitoring.tasks.celery_throughput_high duration_limit_exceeded=0i 1680847055803239552 <--- truncated
leeparayno commented 4 months ago

Looks like this is still occurring in InfluxDB v2.7.5. @lorinescu did you ever try to create a pull request for this fix? Looks like it never made it into any of the released versions.

lorinescu commented 4 months ago

Looks like this is still occurring in InfluxDB v2.7.5. @lorinescu did you ever try to create a pull request for this fix? Looks like it never made it into any of the released versions.

@leeparayno yeap https://github.com/influxdata/influx-cli/pull/503