search

eljefe6a / beamexample

An example Apache Beam project.
111 stars 74 forks source link

HDFS example #4

Open amitsela opened 8 years ago

amitsela commented 8 years ago

Now that we've established that there is no bug in the SparkRunner over HDFS as described here. I was wondering if you have plans to try this on GS ?

amitsela commented 8 years ago

I'm also wondering why is shading them all together an issue ?

eljefe6a commented 8 years ago

Mostly the really long class path. Someone from the Apache Beam team is working an a GS demo.

amitsela commented 8 years ago

So you're probably better of with local mode. I don't see TextIO issues resolved within the next 48 hours 😞

amitsela commented 8 years ago

I don't think WordCount will work with HDFS with any runner..

eljefe6a commented 8 years ago

I have enough to demo Spark+Beam. I'm just going to show it with local files.