Request: Option to auto-delete temporary solution folder

[dhill2522]

I am guessing this is an edge case for many, but I have had GEKKO fill up a 1 TB hard drive and then crash several times now due to all the temporary solution folders GEKKO makes.

This normally happens when running a lot (1000-10000) of models without rebooting in between. For me this happens when doing a sensitivity analysis of a model that includes a GEKKO solve. I am guessing some others may encounter this problem on real-time control systems after the control application has been running for a long time.

Right now this is solved in my scripts by importing shutil and then deleting the temporary folder immediately after the solution shutil.rmtree(m._path).

While some applications may want to leave the temporary folder for later analysis with m.open_folder(), other applications could benefit from auto-deleting these temporary folders.

It would be really useful to be able to tell GEKKO to delete the temporary folders automatically. Something like m.solve(remote=False, auto_delete=True).

[loganbeal]

This is related to #112

We could also piggy-back on the existing debug argument of the solve method -- if debug<=1: self.cleanup()

[APMonitor]

Any application that requires a resolve or warmstart would act differently or fail if the cleanup() function is used automatically. This includes any MHE or MPC application. For most users, this isn't an issue because the /tmp folder is cleaned up on reboot or after a certain period of time for Linux and MacOS.

• Linux: By default, all the files and data that gets stored in /var/tmp live for up to 30 days. Whereas in /tmp, the data gets automatically deleted after ten days. Furthermore, any temporary files that are stored in the /tmp directory get removed immediately on system reboot.
• Windows: %TEMP% files are never deleted by default. A user can select an option to delete them automatically.
• MacOS: \tmp files are deleted every 3 days: https://superuser.com/questions/187071/in-macos-how-often-is-tmp-deleted

Maybe we make another optional argument for m.solve() to automatically cleanup after the solve? Current method:

m.solve()
m.cleanup()

Proposed method:

m.solve(cleanup=True)

Thoughts?

[dhill2522]

This is related to #112

We could also piggy-back on the existing debug argument of the solve method -- if debug<=1: self.cleanup()

Thanks, for catching this. I didn't realize there was already an open issue for it.

I like the idea of m.solve(cleanup=True).

It seems that we could embed some logic that would only delete the files after the end of the entire problem and so avoid messing up MHE/MPC runs.

[APMonitor]

There is an optional finally part to try...except:

try:
   for i in range(100):
      m.solve()
except:
   print('solution failed')
finally:
   m.cleanup()

This could be one way to visit the m.cleanup() function even if the application successfully completes or fails with an exception.

[loganbeal]

I just read a little more about context managers and I think the solution proposed in #112 looks the cleanest.

[APMonitor]

I've used a context manager with numba and other jit compilers such as:

from numba import jit
import numpy as np
x = np.arange(100).reshape(10, 10)
@jit(nopython=True) # Set "nopython" mode for best performance, equivalent to @njit
def go_fast(a): # Function is compiled to machine code when called the first time
    trace = 0.0
    for i in range(a.shape[0]):   # Numba likes loops
        trace += np.tanh(a[i, i]) # Numba likes NumPy functions
    return a + trace              # Numba likes NumPy broadcasting
print(go_fast(x))

What would the proposed context manager for gekko look like? Something like this to cleanup after every solve?

@cleanup
for i in range(100):
   m = GEKKO()
   x = m.Var()
   m.Equation(x==1)
   m.solve()

I'm in favor if it doesn't break existing applications.

[loganbeal]

Could we use atexit to automatically cleanup files when the script ends? https://docs.python.org/3/library/atexit.html We should register the cleanup method upon instantiation to make it transparent to the user.

[APMonitor]

@dhill2522 would atextit fix the problem with your script? The hard drive fills up before the script finishes?

[dhill2522]

@loganbeal atexit wouldn't work with my problem as the interpreter does not close between each model run.

A context manager may be the best way to go.

To be honest I had no idea m.cleanup was an available method. Since implementing that, my issues have been resolved. Maybe we could add this method to the documentation?

[APMonitor]

Added additional documentation and example in the Quick Start guide on the cleanup method.