search

itsjafer / jupyterlab-sparkmonitor

JupyterLab extension that enables monitoring launched Apache Spark jobs from within a notebook
https://krishnan-r.github.io/sparkmonitor/
Apache License 2.0
92 stars 23 forks source link

Exception sending socket message while running jobs #18

Open dciangot opened 4 years ago

dciangot commented 4 years ago

The plugin seems to behave as expected getting port from the environment ( ). But when using the socket ( ) it gets an error Broken pipe (Write failed) e.g. ( ).

Be patient :) could you help me to investigate this?

Thanks in advance.

( * )

SPARKMONITOR_LISTENER: Started SparkListener for Jupyter Notebook
SPARKMONITOR_LISTENER: Port obtained from environment: 37836
SPARKMONITOR_LISTENER: Application Started: KubernetesSpark ...Start Time: 1589273651632

( ** ) https://github.com/itsjafer/jupyterlab-sparkmonitor/blob/9f5da7769b51886b57ac3ef477a7c21aceb371d8/scalalistener/CustomListener.scala#L54

( *** )

SPARKMONITOR_LISTENER: --------------Sending Message:------------------
{
  "msgtype" : "sparkJobStart",
  "jobGroup" : "null",
  "jobId" : 0,
  "status" : "RUNNING",
  "submissionTime" : 1589273663329,
  "stageIds" : [ 0 ],
  "stageInfos" : {
    "0" : {
      "attemptId" : 0,
      "name" : "count at <ipython-input-1-a5fef8d63630>:1",
      "numTasks" : 2,
      "completionTime" : -1,
      "submissionTime" : -1
    }
  },
  "numTasks" : 2,
  "totalCores" : 1,
  "appId" : "KubernetesSpark",
  "numExecutors" : 1,
  "name" : "count at <ipython-input-1-a5fef8d63630>:1"
}
SPARKMONITOR_LISTENER: -------------------------------------------------

SPARKMONITOR_LISTENER: Exception sending socket message:java.net.SocketException: Broken pipe (Write failed)
itsjafer commented 4 years ago

This usually happens because the front-end extension is not running. I think this error is actually linked to your previous issue. Are you able to successfully run the jupyter labextension install jupyterlab_sparkmonitor command?