Closed kmillanrfractal closed 4 years ago
Thanks for the detailed report. I assume you already did the pip install spark-nlp
with the same version. (just wanted to double-check)
Could you please do a .show() and .printSchema() on your df
?
df = kafkaDf.selectExpr("CAST(value AS STRING)")
.select(from_json('value', schema).alias('data'))
.select('data.text')
I want to see if you have a text
column and it is actually a String and noting else.
Hey Maziyar,
thank you for you quick response. Yeah I installed spark-nlp through pip. Sure thing, attached you will be able to view the required information. One image shows how the output looks on the console, in the other picture I saved some batches locally and used jupyter notebook to view the contents of the stream to obtain the .show() and .printSchema() output.
thank you once again for your help.
all the best,
Kevin
On Thu, May 7, 2020 at 1:57 PM Maziyar Panahi notifications@github.com wrote:
Thanks for the detailed report. I assume you already did the pip install spark-nlp with the same version. (just wanted to double-check)
Could you please do a .show() and .printSchema() on your df?
df = kafkaDf.selectExpr("CAST(value AS STRING)") .select(from_json('value', schema).alias('data')) .select('data.text')
I want to see if you have a text column and it is actually a String and noting else.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JohnSnowLabs/spark-nlp/issues/895#issuecomment-625407535, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO72KADNH3OU7UCK3QNGQDLRQLZAPANCNFSM4M3OLZKA .
One more thing I forgot to mention, the pipeline does actually work, but only a couple of times, after transforming 2 or 3 batches, sometimes even only 1. Then it exits with that error. Sorry for not mentioning that before.
All the best,
Kevin
On Thu, May 7, 2020 at 8:04 PM Kevin Millan kegamiro@gmail.com wrote:
Hey Maziyar,
thank you for you quick response. Yeah I installed spark-nlp through pip. Sure thing, attached you will be able to view the required information. One image shows how the output looks on the console, in the other picture I saved some batches locally and used jupyter notebook to view the contents of the stream to obtain the .show() and .printSchema() output.
thank you once again for your help.
all the best,
Kevin
On Thu, May 7, 2020 at 1:57 PM Maziyar Panahi notifications@github.com wrote:
Thanks for the detailed report. I assume you already did the pip install spark-nlp with the same version. (just wanted to double-check)
Could you please do a .show() and .printSchema() on your df?
df = kafkaDf.selectExpr("CAST(value AS STRING)") .select(from_json('value', schema).alias('data')) .select('data.text')
I want to see if you have a text column and it is actually a String and noting else.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JohnSnowLabs/spark-nlp/issues/895#issuecomment-625407535, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO72KADNH3OU7UCK3QNGQDLRQLZAPANCNFSM4M3OLZKA .
I think you forgot to attach the screenshot(s).
Sorry about that,
I have attached to pictures both in this email and on the github thread as well. Thank you.
all the best,
Kevin
[image: Screenshot from 2020-05-07 20-00-48.png] [image: Screenshot from 2020-05-07 20-01-28.png]
On Sat, May 9, 2020 at 9:49 AM Maziyar Panahi notifications@github.com wrote:
I think you forgot to attach the screenshot(s).
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JohnSnowLabs/spark-nlp/issues/895#issuecomment-626179011, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO72KADLZPED24LOCQOBACDRQVNM5ANCNFSM4M3OLZKA .
Thank you. So these are Tweets. Possible to have empty strings or become empty after tokenizing due to just having emojis etc. For some reason, that specific pipeline doesn't like empty strings. You need to do a filter to be sure the text is not empty in your initial DataFrame or even do a size control that if the Tweet is larger than 10-20 characters to pass it through:
Caused by: java.lang.NullPointerException
df = kafkaDf.selectExpr("CAST(value AS STRING)")
.select(from_json('value', schema).alias('data'))
.filter("data.text != ''")
.select('data.text')
or
import pyspark.sql.functions as F
df = kafkaDf.selectExpr("CAST(value AS STRING)")
.select(from_json('value', schema).alias('data'))
.withColumn('text_length', F.length(data.text))
.filter("text_length > 20")
.select('data.text')
PS: This may not happen in our new SentimentDL and the model/pipeline for it since it uses DL Transformers Universal Sentence Encoder in 2.5.0.
Thank you very much for your help, will give the solution a try and get back to you as soon as possible.
All the best,
Kevin
On Sat, May 9, 2020 at 1:16 PM Maziyar Panahi notifications@github.com wrote:
Thank you. So these are Tweets. Possible to have empty strings or become empty after tokenizing due to just having emojis etc. For some reason, that specific pipeline doesn't like empty strings. You need to do a filter to be sure the text is not empty in your initial DataFrame or even do a size control that if the Tweet is larger than 10-20 characters to pass it through:
Caused by: java.lang.NullPointerException
df = kafkaDf.selectExpr("CAST(value AS STRING)") .select(from_json('value', schema).alias('data')) .filter("data.text != ''") .select('data.text')
or
import pyspark.sql.functions as F
df = kafkaDf.selectExpr("CAST(value AS STRING)") .select(from_json('value', schema).alias('data')) .withColumn('text_length', F.length(data.text)) .filter("text_length > 20") .select('data.text')
PS: This may not happen in our new SentimentDL and the model/pipeline for it since it uses DL Transformers Universal Sentence Encoder in 2.5.0.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JohnSnowLabs/spark-nlp/issues/895#issuecomment-626207925, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO72KADBUVEXLIWC4EFBCZLRQWFVHANCNFSM4M3OLZKA .
Worked like a charm, thank you.
all the best,
Kevin
On Sat, May 9, 2020 at 4:29 PM Kevin Millan kegamiro@gmail.com wrote:
Thank you very much for your help, will give the solution a try and get back to you as soon as possible.
All the best,
Kevin
On Sat, May 9, 2020 at 1:16 PM Maziyar Panahi notifications@github.com wrote:
Thank you. So these are Tweets. Possible to have empty strings or become empty after tokenizing due to just having emojis etc. For some reason, that specific pipeline doesn't like empty strings. You need to do a filter to be sure the text is not empty in your initial DataFrame or even do a size control that if the Tweet is larger than 10-20 characters to pass it through:
Caused by: java.lang.NullPointerException
df = kafkaDf.selectExpr("CAST(value AS STRING)") .select(from_json('value', schema).alias('data')) .filter("data.text != ''") .select('data.text')
or
import pyspark.sql.functions as F
df = kafkaDf.selectExpr("CAST(value AS STRING)") .select(from_json('value', schema).alias('data')) .withColumn('text_length', F.length(data.text)) .filter("text_length > 20") .select('data.text')
PS: This may not happen in our new SentimentDL and the model/pipeline for it since it uses DL Transformers Universal Sentence Encoder in 2.5.0.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JohnSnowLabs/spark-nlp/issues/895#issuecomment-626207925, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO72KADBUVEXLIWC4EFBCZLRQWFVHANCNFSM4M3OLZKA .
Now, the error I get is as follows:
Any ideas on what might be the issue? If you need additional info on the error or code please don't hesitate to ask. I'm innitializing the spark script with the following command:
spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.5,com.johnsnowlabs.nlp:spark-nlp_2.11:2.4.5 --class com.inndata.StructuredStreaming.Kafka --master local[*] /home/kevin/kafka-pyspark-streaming.py