Error Walking through the README quick start

[liammac]

I'm following the instructions for the quick start and I get to the point where I run the jes.py

It fails with the following

python jes.py
ERROR:inviso.jes:[Errno 32] Broken pipe
Traceback (most recent call last):
  File "jes.py", line 35, in main
    monitor.run()
  File "/home/liamm/invisio/inviso/jes/inviso/monitor.py", line 295, in run
    for f in listing:
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/client.py", line 139, in ls
    recurse=recurse):
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/client.py", line 1072, in _find_items
    fileinfo = self._get_file_info(path)
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/client.py", line 1202, in _get_file_info
    return self.service.getFileInfo(request)
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/service.py", line 35, in <lambda>
    rpc = lambda request, service=self, method=method.name: service.call(service_stub_class.__dict__[method], request)
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/service.py", line 41, in call
    return method(self.service, controller, request)
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py", line 267, in <lambda>
    self._StubMethod(inst, method, rpc_controller, request, callback))
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py", line 284, in _StubMethod
    method_descriptor.output_type._concrete_class, callback)
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/channel.py", line 411, in CallMethod
    self.send_rpc_message(method, request)
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/channel.py", line 309, in send_rpc_message
    self.write_delimited(rpc_request_header)
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/channel.py", line 238, in write_delimited
    self.write(encoder._VarintBytes(len(data)))
  File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/channel.py", line 235, in write
    self.sock.send(data)
error: [Errno 32] Broken pipe

[danielcweeks]

What version of hadoop are you running against?

[danielcweeks]

Took another look and it's probably trying to used the wrong port to talk to the name node (it defaults to 9000 which is what EMR uses). Try adding the namenode port to the paramaters in jes.py.

 for cluster in settings.clusters:
     monitors.append(HdfsMr2LogMonitor(jobflow=cluster.id,
     cluster_id=cluster.id,
     cluster_name=cluster.name,
     host=cluster.host,
     publisher=publisher,
     port=8020,  
     elasticsearch=settings.elasticsearch))

[monolive]

I had the same issue on a HDP cluster. You are correct specifying the port fixed the issue.

In a similar fashion, I also had to change the port in index_cluster_stats.py from 9026 to 8088.