shell engine cannot support special operations

[mindflow94]

My linkis version is 1.0.2. When I use the shell engine to submit a shell job, the contents of the job are ls > tmp.txt. This job will report a mistake. I think some special operation can cause a job to fail, such as |、>.

[mindflow94]

I tried to modify shellEngineConnExecutor's code and this issue was resolved.

val processBuilder: ProcessBuilder = new ProcessBuilder(generateRunCodeToList(code))

private def generateRunCodeToList(code: String): util.List[String] = {
    util.Arrays.asList("sh", "-c", code)
  }

I didn't do the full test and didn't know if there would be any other risks.

[wForget]

I tried to modify shellEngineConnExecutor's code and this issue was resolved.

val processBuilder: ProcessBuilder = new ProcessBuilder(generateRunCodeToList(code))

private def generateRunCodeToList(code: String): util.List[String] = {
    util.Arrays.asList("sh", "-c", code)
  }

I didn't do the full test and didn't know if there would be any other risks.

This code seems to just change the scala.Array type to the java.util.List type. Can you display specific error messages?

[mindflow94]

Can you display specific error messages

Toggle the constructor of the ProcessBuilder, Pass List<String> instead of String..., To avoid parsing special symbols in scripts.

There is no explicit error message, just a prompt for the shell code is wrong.The following is a detailed error:

2021-12-02 14:24:04.538 ERROR [Linkis-Default-Scheduler-Thread-5] com.webank.wedatasphere.linkis.manager.engineplugin.shell.executor.ShellEngineConnExecutor 61 error - exitCode is 1
2021-12-02 14:24:04.539 ERROR [Linkis-Default-Scheduler-Thread-5] com.webank.wedatasphere.linkis.manager.engineplugin.shell.executor.ShellEngineConnExecutor 61 error - 
2021-12-02 14:24:04.555 ERROR [Linkis-Default-Scheduler-Thread-5] com.webank.wedatasphere.linkis.manager.engineplugin.shell.executor.ShellEngineConnExecutor 57 error - execute code failed! com.webank.wedatasphere.linkis.manager.engineplugin.shell.exception.ShellCodeErrorException: errCode: 50037 ,desc: shell code is wrong ,ip: ct6 ,port: 44542 ,serviceKind: linkis-cg-engineconn
    at com.webank.wedatasphere.linkis.manager.engineplugin.shell.executor.ShellEngineConnExecutor.executeLine(ShellEngineConnExecutor.scala:88) ~[linkis-engineplugin-shell-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10$$anonfun$apply$11.apply(ComputationExecutor.scala:179) ~[linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10$$anonfun$apply$11.apply(ComputationExecutor.scala:178) ~[linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:39) ~[linkis-common-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10.apply(ComputationExecutor.scala:180) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10.apply(ComputationExecutor.scala:174) [linkis-computation-engineconn-1.0.2.jar:?]
    at scala.collection.immutable.Range.foreach(Range.scala:160) [scala-library-2.11.12.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2.apply(ComputationExecutor.scala:173) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2.apply(ComputationExecutor.scala:149) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.common.utils.Utils$.tryFinally(Utils.scala:60) [linkis-common-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor.toExecuteTask(ComputationExecutor.scala:222) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$3.apply(ComputationExecutor.scala:237) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$3.apply(ComputationExecutor.scala:237) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.common.utils.Utils$.tryFinally(Utils.scala:60) [linkis-common-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.ensureIdle(AccessibleExecutor.scala:54) [linkis-accessible-executor-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.ensureIdle(AccessibleExecutor.scala:48) [linkis-accessible-executor-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor.ensureOp(ComputationExecutor.scala:133) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.ComputationExecutor.execute(ComputationExecutor.scala:236) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl.com$webank$wedatasphere$linkis$engineconn$computation$executor$service$TaskExecutionServiceImpl$$executeTask(TaskExecutionServiceImpl.scala:239) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1$$anonfun$run$1.apply$mcV$sp(TaskExecutionServiceImpl.scala:172) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1$$anonfun$run$1.apply(TaskExecutionServiceImpl.scala:170) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1$$anonfun$run$1.apply(TaskExecutionServiceImpl.scala:170) [linkis-computation-engineconn-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:39) [linkis-common-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.common.utils.Utils$.tryAndWarn(Utils.scala:68) [linkis-common-1.0.2.jar:?]
    at com.webank.wedatasphere.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1.run(TaskExecutionServiceImpl.scala:170) [linkis-computation-engineconn-1.0.2.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_281]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_281]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_281]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_281]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]

[wForget]

The two construction methods seem to be equivalent. I tested the shell and found no errors. Is it caused by others?

ls /home/hadoop > /home/hadoop/tmp.txt

[mindflow94]

For me, with the changes above, my Shell Engine can successfully execute ls > tmp.txt. If you can pass the ls /home/hadoop > /home/hadoop/tmp.txt test, I can't think of any other reason.