oh no

[laudiacay]

> fail_node -n node-1
Traceback (most recent call last):
  File "/home/laudiacay/.virtualenvs/dsfinal/lib/python3.5/site-packages/cmd2.py", line 2468, in onecmd_plus_hooks
    stop = self.onecmd(statement)
  File "/home/laudiacay/.virtualenvs/dsfinal/lib/python3.5/site-packages/cmd2.py", line 2663, in onecmd
    stop = func(statement)
  File "/home/laudiacay/.virtualenvs/dsfinal/lib/python3.5/site-packages/cmd2.py", line 615, in new_func
    result = func(instance, arg, opts)
  File "/home/laudiacay/.virtualenvs/dsfinal/src/chistributed/src/chistributed/cli/interpreter.py", line 138, in do_fail_node
    self.ds.fail_node(node_id)
  File "/home/laudiacay/.virtualenvs/dsfinal/src/chistributed/src/chistributed/core/model.py", line 394, in fail_node
    raise ChistributedException("Node {} is already in a failed state".format(node_id))
chistributed.common.ChistributedException: Node node-1 is already in a failed state
EXCEPTION of type 'ChistributedException' occurred with message: 'Node node-1 is already in a failed state'
To enable full traceback, run the following command:  'set debug true'
> set debug true
No such node: None
> oh no

[borjasotomayor]

I cannot reproduce this issue with the information provided. Please see https://uchicago-cs.github.io/debugging-guide/#asking-for-help for tips of what kind of information we would need to be able to debug this issue (at the very least, we need the configuration file and the exact list of steps that lead to this issue)

[mtn]

@borjasotomayor I poked at this for a few minutes, and have an explanation. The error message

To enable full traceback, run the following command: 'set debug true'

seems to imply that we can enable debugging at the command line, but browsing the source I realized this wasn't actually what was happening. Rather, the command collides with our set request, creating a malformed request with no node destination, key, or value (hence the node reference error).

In fact, the error message is actually from cmd2 (example), which sort of implicitly assumes no command will be called set.

Off the top of my head I can't think of a great solution that isn't gross (like forking their project). But at least this is an explanation.

Also, a more minimal example should be just running chistributed with a barebones conf. If it helps I can make something extremely minimal, but the sample in this project is probably sufficient.

Edit: Actually, disabling the set debug message is straightforward, but cmd2 doesn't expose that option (unless I misunderstood). I'm going to open an issue in their project, since it seems reasonable that the user get to disable those messages in all cases. (relevant line)

Edit: There are some suggestions on the issue in cmd2. I guess the easiest way to just get the message to go away is overriding perror, if that's the goal.

[tleonhardt]

Hi. I'm one of the core maintainers of cmd2. If you have any questions on how to best move forward for your project which are related to cmd2, I would be happy to chat with you in regards to this.

Best wishes

-Todd