discoproject / disco

a Map/Reduce framework for distributed computing
http://discoproject.org
BSD 3-Clause "New" or "Revised" License
1.63k stars 241 forks source link

Can't run example from documentation #630

Open BitResolution opened 9 years ago

BitResolution commented 9 years ago

Hi, I've set up a disco cluster on EC2 using ansible to provision the machines. I can successfully lauch disco, can see the black nodes on the status page, but when I try and execute the sample code (word count) from the documentation I get the following error.

disco@ip-172-31-30-166:~/disco$ ./disco-smoke-test.py 
/usr/local/lib/python2.7/dist-packages/disco/fileutils.py:141: UserWarning: Duplicate name: 'usr/local/lib/python2.7/dist-packages/disco/job.py'
  self.write(file, file)
/usr/local/lib/python2.7/dist-packages/disco/fileutils.py:141: UserWarning: Duplicate name: 'usr/local/lib/python2.7/dist-packages/disco/worker/classic/worker.py'
  self.write(file, file)
Job@5a0:7ba4d:4cf2:
Traceback (most recent call last):
  File "./disco-smoke-test.py", line 18, in <module>
    for word, count in result_iterator(job.wait(show=True)):
  File "/usr/local/lib/python2.7/dist-packages/disco/core.py", line 366, in wait
    event_monitor.refresh()
  File "/usr/local/lib/python2.7/dist-packages/disco/eventmonitor.py", line 184, in refresh
    status = self.status
  File "/usr/local/lib/python2.7/dist-packages/disco/eventmonitor.py", line 175, in status
    .format(tuple(self.stats)))
  File "/usr/local/lib/python2.7/dist-packages/disco/eventmonitor.py", line 166, in stats
    first = pipeline[0]
IndexError: list index out of range

I did see #443 and have execute the commands you asked them to:

disco@ip-172-31-30-166:~$ echo $LANG
en_US.UTF-8
disco@ip-172-31-30-166:~$ python --version
Python 2.7.6
disco@ip-172-31-30-166:~$ erl version
Erlang R16B03 (erts-5.10.4) [source] [64-bit] [async-threads:10] [kernel-poll:false]

I've also tried running the healthcheck but I get the following output:

disco@ip-172-31-30-166:~$ ./healthcheck.sh nodes.txt 
inspecting discomaster {"init terminating in do_boot",{{badmatch,pang},[{pinger,pingit,1,[{file,"pinger.erl"},{line,5}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}

Crash dump was written to: erl_crash.dump
init terminating in do_boot ()

I have had this running with the same ansible scripts on a local virtualbox vms (I'm using vagrant to spin up VMs then Ansible to provision). The problem only exhibits itself on my EC2 environment.

ghost commented 7 years ago

I had a similar issue, it was something to do with using the newer pipeline worker vs classic

possibly defining self.pipeline