search

TresAmigosSD / SMV

Spark Modularized View
Apache License 2.0
42 stars 22 forks source link

Better user experience when requesting help from the CLI tools #1446

Open jacobdr opened 5 years ago

jacobdr commented 5 years ago

It seems like smv-shell and smv-run no longer emit any output when run with the -h or --help flags.

$ ./tools/smv-shell -h

Detected python installation of pyspark. Adding it to the spark-submit candidates on the PATH
Using spark-submit to submit jobs
Using pyspark to start shells
2018-10-27 15:01:08 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoSuchMethodException: org.tresamigos.smv.SmvApp.main([Ljava.lang.String;)
    at java.lang.Class.getMethod(Class.java:1786)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:42)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2018-10-27 15:01:09 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-10-27 15:01:09 INFO  ShutdownHookManager:54 - Deleting directory /private/var/folders/j6/6t998hw57xb1q52ccrj516640000gn/T/spark-0c7d1641-6233-49d7-85a2-64319416cce2

Ditto for --help:

 $ ./tools/smv-shell --help

Additionally, I would expect an smv-run invocation without any parameters to produce a useful error message/help, but now it spills:

$ ./tools/smv-run

Traceback (most recent call last):
  File "/Users/jroberts/Documents/Projects/OpenSource/SMV/venv/lib/python2.7/site-packages/smv/tools/../src/main/python/scripts/runapp.py", line 17, in <module>
    SmvDriver().run()
  File "/Users/jroberts/Documents/Projects/OpenSource/SMV/venv/lib/python2.7/site-packages/smv/smvdriver.py", line 53, in run
    app = self.create_smv_app(smv_args, driver_args)
  File "/Users/jroberts/Documents/Projects/OpenSource/SMV/venv/lib/python2.7/site-packages/smv/smvdriver.py", line 24, in create_smv_app
    return SmvApp.createInstance(smv_args)
  File "/Users/jroberts/Documents/Projects/OpenSource/SMV/venv/lib/python2.7/site-packages/smv/smvapp.py", line 75, in createInstance
    cls._instance = cls(arglist, _sparkSession)
  File "/Users/jroberts/Documents/Projects/OpenSource/SMV/venv/lib/python2.7/site-packages/smv/smvapp.py", line 105, in __init__
    self.py_smvconf = SmvConfig(arglist, self._jvm)
  File "/Users/jroberts/Documents/Projects/OpenSource/SMV/venv/lib/python2.7/site-packages/smv/smvconfig.py", line 59, in __init__
    self.all_data_dirs()
  File "/Users/jroberts/Documents/Projects/OpenSource/SMV/venv/lib/python2.7/site-packages/smv/smvconfig.py", line 116, in all_data_dirs
    raise SmvRuntimeError("Must specify a data-dir either on command line or in conf.")
smv.error.SmvRuntimeError: Must specify a data-dir either on command line or in conf.