This is mostly a reminder for myself for when I have time to implement this.
Often my cluster submissions crash for some reason. Most of the time (not always) you get a stacktrace, but that's it in terms of information to diagnose the problem. If you added print statements, you can read the log files of the individual jobs, but that's also inconvenient.
My suggestion would be to add a "debug mode" that streams all output from job 0 to the terminal. Already this should allow for more convenient debugging. Bonus would be to make it possible to run the python debugger within the jobs. Potentially this could be done by forwarding the input from the terminal to the job.
This is intended for local execution, not for execution on the cluster.
This is mostly a reminder for myself for when I have time to implement this.
Often my cluster submissions crash for some reason. Most of the time (not always) you get a stacktrace, but that's it in terms of information to diagnose the problem. If you added print statements, you can read the log files of the individual jobs, but that's also inconvenient.
My suggestion would be to add a "debug mode" that streams all output from job 0 to the terminal. Already this should allow for more convenient debugging. Bonus would be to make it possible to run the python debugger within the jobs. Potentially this could be done by forwarding the input from the terminal to the job.
This is intended for local execution, not for execution on the cluster.
By @mseitzer (imported from GitLab)