Closed geopolitis closed 2 years ago
@geopolitis Looking at this now...
I pushed a fix, the name for count/tital wasn't being called right.
Still fixing...
@geopolitis This is fixed.
@rjurney a few of the issues have been fixed, however, seem like the issues below insists and also after a few steps is missing the file /data/simple_flight_delay_features.jsonl.bz2 "Building a Classifier Model to Predict Flight Delays Loading Our Data"
Could you help?
Py4JJavaError Traceback (most recent call last) /home/vagrant/spark/python/pyspark/sql/utils.py in deco(*a, *kw) 62 try: ---> 63 return f(a, **kw) 64 except py4j.protocol.Py4JJavaError as e:
/home/vagrant/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name) 318 "An error occurred while calling {0}{1}{2}.\n". --> 319 format(target_id, ".", name), value) 320 else:
Py4JJavaError: An error occurred while calling o181.select.
: org.apache.spark.sql.AnalysisException: cannot resolve 'count
' given input columns: [company, name, title];;
'Project [name#1, 'count]
+- LogicalRDD [company#0, name#1, title#2]
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:88)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:85)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:268)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:268)
at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:279)
at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:289)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$1.apply(QueryPlan.scala:293)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:293)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$6.apply(QueryPlan.scala:298)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
at org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:298)
at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:268)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:78)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:78)
at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:91)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:52)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:67)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2884)
at org.apache.spark.sql.Dataset.select(Dataset.scala:1150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
During handling of the above exception, another exception occurred:
AnalysisException Traceback (most recent call last)
@rjurney regarding the /data/simple_flight_delay_features.jsonl.bz2 "Building a Classifier Model to Predict Flight Delays Loading Our Data"
I found a solution. The script in Chapter08 (ch08/download_data.sh ) downloads that files. However you need those files in chapter 02.
I fixed the download issue in https://github.com/rjurney/Agile_Data_Code_2/commit/1a55d930d3a12c6259673ce92706ccd7c4a0d281
Run git pull origin master
and you should get the update.
I'm looking at the rest...
@geopolitis I fixed all these to make the class go in 2021.
Please find below all the out of the Exercise. The step 10 is successful, the step 11 fails. Also fails the "Loading and Inspecting Parquet Files" find the output below.
================= Exercises First count the number of job titles for each executive, then create a pyspark.sql.Row for this result. In [10]:
records = csv_lines.map(lambda line: line.split(',')) records.collect() groups = records.groupBy(lambda x: x[0]) counts = groups.map(lambda x: (x[0], len(x[1]))) new_rows = counts.map(lambda x: Row(name=x[0], count=x[1])) new_rows.collect() Out[10]: [Row(count=1, name='Florian Liebert'), Row(count=2, name='Russell Jurney'), Row(count=1, name='Don Brown'), Row(count=1, name='Steve Jobs'), Row(count=1, name='Donald Trump')] In [11]:
rows.toDF().select("name","count").show()
Py4JJavaError Traceback (most recent call last) /home/vagrant/spark/python/pyspark/sql/utils.py in deco(*a, *kw) 62 try: ---> 63 return f(a, **kw) 64 except py4j.protocol.Py4JJavaError as e:
/home/vagrant/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name) 318 "An error occurred while calling {0}{1}{2}.\n". --> 319 format(target_id, ".", name), value) 320 else:
Py4JJavaError: An error occurred while calling o181.select. : org.apache.spark.sql.AnalysisException: cannot resolve '
count
' given input columns: [company, name, title];; 'Project [name#1, 'count] +- LogicalRDD [company#0, name#1, title#2]During handling of the above exception, another exception occurred:
AnalysisException Traceback (most recent call last)