Local solve with solver line updates

[APMonitor]

I was running into a problem with the buffer filling up for the local solve, especially when there is a lot of screen output, and blocking the execution. I changed the local solve code to remove the problem with readline of the piped output and switched to a communicate method from Popen. However, this removed the nice feature to see the solver progress as the solver outputs code. The line-by-line still works for remote solves. I'm opening this as an issue to see if someone has an idea on how to restore the line-by-line output without a block caused by the buffer filling.

Here is something that I found on StackOverflow that is related:

import subprocess, time, tempfile, re

pipe_output, file_name = tempfile.TemporaryFile()
cmd = ["rsync", "-vaz", "-P", "/src/" ,"/dest"]

p = subprocess.Popen(cmd, stdout=pipe_output, 
                     stderr=subprocess.STDOUT)
while p.poll() is None:
    # p.poll() returns None while the program is still running
    # sleep for 0.1 second
    time.sleep(0.1)
    last_line =  open(file_name).readlines()
    # it's possible that it hasn't output yet, so continue
    if len(last_line) == 0: continue
    last_line = last_line[-1]
    print(last_line)

[abe-mart]

I don't love this change. I think the real-time solver output is a valuable tool for long running solves, and I use it regularly with local solves. Can you avoid the buffer problem with disp=False, or debug=0?

[nsgates]

Another option is to automatically save the line by line output to a log file which can be opened (and refreshed) to view the progress.

[abe-mart]

I'd like to experiment with this package to see if it might help:

https://sarge.readthedocs.io/en/latest/overview.html

Do you have a simple example that triggers the buffer filling problem?

[APMonitor]

Any time I set m.options.diaglevel=2 I get a blocking error where the buffer fills up and also for other problems that have fast solutions and a lot of console output. I've already tried a few things such as increase the bufsize (didn't work because it is the child buffer that is filling, not the calling program).

There is another report of this blocking error (deadlock): https://github.com/BYU-PRISM/GEKKO/issues/61#issuecomment-516650493 I really like the console output so this definitely needs some immediate development to get it back but without the deadlock issue.

Thanks for sending the link to the Sarge package. I'd like to do it first without adding another dependency to Gekko but this looks like a great option if app.communicate, temp file for output, or some other Popen option doesn't work.

[APMonitor]

I kept the old method for local solve when option debug>=2. The new app.communicate method doesn't show line-by-line output for local solve with GEKKO v0.2.3. I will leave this issue open so that it can be a priority for a future release.