search

databricks / reference-apps

Spark reference applications
Other
656 stars 341 forks source link

when run the code get the RuntimeException #73

Open NewBoy825 opened 8 years ago

NewBoy825 commented 8 years ago

When I follow the README file's steps one by one, after I run spark-submit, I got the error: ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1).java.lang.RuntimeException: Error parsing logline, I know where it source from, but know idea how to resolve it, has any tips?

astroyang commented 8 years ago

Hey,

I also run into the Error parsing logline. I checked a bit and believed that it's the format of log file that causes this problem. I have to change the regex rule and actually delete one line of log to make it work. (the log file from "http://www.monitorware.com/en/logsamples/apache.php"). I wonder if you have figured out a better solution?

HamzaKhribi commented 6 years ago

Hey folks, Am having the same issue and i think that throwing a runtime exception is not the best idea to deal with mis formed logs. I mean why i would stop the excution of my program if one line of log doesn't respect my pattern? The problem is clearly in the log file so i think i will just skip that line. Anyone has a better solution?

tashoyan commented 6 years ago

This is the easiest way to handle malformed input, most suitable for educational purposes. In a production system you may implement a special error handling, like push malformed records to a special Kafka topic for advanced analysis. In an educational app the only thing needed is to quickly find out that the sample is bad.

HamzaKhribi commented 6 years ago

Thank you very much for your reply!! Now i get it. Have a nice day