search

NOAA-OWP / wres

Code and scripts for the Water Resources Evaluation Service
Other
2 stars 1 forks source link

As a system admin, I want WRES to clean up outputs when they are no longer needed #198

Open epag opened 3 months ago

epag commented 3 months ago

Author Name: Jesse (Jesse) Original Redmine Issue: 56097, https://vlab.noaa.gov/redmine/issues/56097 Original Date: 2018-10-10

Given several past evaluation projects that have been run through a version of WRES When I run another evaluation Then WRES should clean up unneeded outputs

The reason for this is WRES currently stores outputs in files and at the moment this can mean /tmp filling up. See #50642-48 #50642-54 #50642-81 #50642-82 #50642-83 #50642-84

Redmine related issue(s): 50642

epag commented 3 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2018-10-10T18:12:42Z

I created this ticket in response to the comments in #50642, I don't think it's necessary to implement except in server mode, others may disagree. I may have even gotten the gist of the issue incorrect. James or Chris correct me here as needed.

epag commented 3 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2018-10-10T18:25:16Z

I'm not sure there's any "correcting" to be done, but perhaps some discussion.

The following is what I envisaged. WRES-the-application allows for outputs to be expired and it implements a default expiry policy. This is part of core WRES-the-application. By implication, WRES would need to track output directories (whether chosen by a user or by the system) in the database and expire them according to some criteria that has a default, which can be overridden.

Basically, as a laptop user, I don't want WRES to delegate the expiry of WRES outputs to some other process. I want it to play nicely and not fill-up my temp directory.

I wouldn't advocate that the expiry policy is "next execution", as I believe is written in the OP, although it could be that (but then I'm not sure how that sits with the precondition on "several"). However, it makes sense that the check-then-act for expiry is triggered by an execution - indeed, I'm not sure how else we'd do it. In that context, the default expiry policy is constrained by the triggering mechanism (execution).

Probably just needs some further discussion here to decide whether my expectations are realistic/agreeable and how we should implement anything along these lines.

epag commented 3 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2018-10-22T19:32:13Z

James wrote:

I'm not sure there's any "correcting" to be done, but perhaps some discussion.

The following is what I envisaged. WRES-the-application allows for outputs to be expired

Yes, agree.

and it implements a default expiry policy.

Yes, agree.

This is part of core WRES-the-application.

Yes, agree.

By implication, WRES would need to track output directories (whether chosen by a user or by the system) in the database

That is an option, I would disagree. An alternative from the WRES software perspective is "I look wherever I was told to place my outputs for old outputs and clean them up if the expiration criteria are met."

and expire them according to some criteria that has a default, which can be overridden.

Yes, agree.

Basically, as a laptop user, I don't want WRES to delegate the expiry of WRES outputs to some other process. I want it to play nicely and not fill-up my temp directory.

Yes, agree.

I wouldn't advocate that the expiry policy is "next execution",

Yes, agree.

as I believe is written in the OP,

No, but the OP got into implementation details, where I was thinking that the software would check each time it ran if it needed to clean up. It means that there is more frequent checking than cleaning, and a natural time to check. The reason is to have no out-of-band intervention required beyond running an evaluation.

although it could be that (but then I'm not sure how that sits with the precondition on "several"). However, it makes sense that the check-then-act for expiry is triggered by an execution - indeed, I'm not sure how else we'd do it. In that context, the default expiry policy is constrained by the triggering mechanism (execution).

Right.

Probably just needs some further discussion here to decide whether my expectations are realistic/agreeable and how we should implement anything along these lines.

Hope this comment helps.

epag commented 3 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2018-10-22T19:44:56Z

Yes, that helps. No out-of-band intervention, I absolutely agree with that.

Only comment:

Jesse wrote:

That is an option, I would disagree. An alternative from the WRES software perspective is "I look wherever I was told to place my outputs for old outputs and clean them up if the expiration criteria are met.

I didn't quite follow that. Did you mean "wouldn't disagree" or...?

Either way, I think the "I look wherever I was told" implies some cache of "wherever I was told", right? To me, it seems like the natural place would be the database and, more specifically, somewhere near to the execution log within the database, since that already records the project declaration (I believe. EDIT: and the execution sequence, such as the setting of the temp dir, and perhaps some status info, such as what was written? I'll need to remind myself...), although it could be 'not the database' and/or 'not somewhere near to the execution log'. Overall, I'm not too worried about how this is implemented, and more worried about the desired behavior. I believe we have a similar (perhaps identical) opinion about the desired behavior having read what you wrote above.

epag commented 3 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2018-10-22T19:50:39Z

I will also add that I am filled with some level of fear about a mutation capability that erases data, even though I think we need it. We want to avoid a situation whereby we could write buggy code that could lead to data being destroyed that was NOT produced by WRES. To me, avoiding that possibility comes above everything else.

epag commented 3 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2018-10-22T19:53:58Z

James wrote:

Yes, that helps. No out-of-band intervention, I absolutely agree with that.

Only comment:

Jesse wrote:

That is an option, I would disagree. An alternative from the WRES software perspective is "I look wherever I was told to place my outputs for old outputs and clean them up if the expiration criteria are met.

I didn't quite follow that. Did you mean "wouldn't disagree" or...?

Whoops. I meant to say "That is an option, but I would prefer a different option." Or "Disagree, we can do it an alternative way."

Either way, I think the "I look wherever I was told" implies some cache of "wherever I was told", right?

Not exactly. Every time the software is run, either the default java.io.tmpdir is used or a specified-at-runtime java.io.tmpdir is used for the purpose of outputs. Ideally, by default, humans need not be aware of this setting for the software to do its work. Because humans need not be aware of this setting, and since it will therefore not change from one run to the next, and since the software is aware of its own conventions of directory creation within this specified directory, the software can infer that old outputs may be existing in the java.io.tmpdir and therefore check the criteria for deletion.

To me, it seems like the natural place would be the database and, more specifically, somewhere near to the execution log within the database, since that already records the project declaration (I believe), although it could be 'not the database' and/or 'not somewhere near to the execution log'. Overall, I'm not too worried about how this is implemented, and more worried about the desired behavior. I believe we have a similar (perhaps identical) opinion about the desired behavior having read what you wrote above.

Right. Many ways to implement, database could be among them, I prefer the above solution, because the database knowing about files on it's client's system seems like the tail wagging the dog to me. But that's maybe just me.

I will also add that I am filled with some level of fear about a mutation capability that erases data, even though I think we need it. We want to avoid a situation whereby we could write buggy code that could lead to data being destroyed that was NOT produced by WRES. To me, avoiding that possibility comes above everything else.

Yes, of course. It (the WRES software) should be careful to only remove things it is confident it created.

epag commented 3 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2018-10-22T19:56:32Z

Sorry, one more thing. Java 9 has a mechanism whereby it's possible to ask whether the runtime environment has a concept of "trash" and to move the redundant data there by preference:

https://docs.oracle.com/javase/9/docs/api/java/awt/Desktop.Action.html

In other words, a second layer of "eligible for deletion" at the OS-level. Prior to Java 9, this was rather difficult to achieve (JNI territory).

epag commented 3 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2018-10-22T20:04:20Z

Jesse wrote:

Not exactly. Every time the software is run, either the default java.io.tmpdir is used or a specified-at-runtime java.io.tmpdir is used for the purpose of outputs. Ideally, by default, humans need not be aware of this setting for the software to do its work. Because humans need not be aware of this setting, and since it will therefore not change from one run to the next, and since the software is aware of its own conventions of directory creation within this specified directory, the software can infer that old outputs may be existing in the java.io.tmpdir and therefore check the criteria for deletion.

I guess I'll put this in the category of "watching with interest" for now. I don't want to hold you up. As a laptop user, I am going to regularly specify an output directory that overrides the Java temp directory, so it will change from one run to the next. I would like WRES to clean-up after me. I would like WRES not to delete data that it didn't create. All of these things imply other things. As I say, watching with interest for now as a developer, but my first concern as a user is the desired behavior.

EDIT: I will add that solving the problem of cleaning up the default Java temp directory of files that were written by WRES is the first priority.

epag commented 3 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2018-10-22T20:11:25Z

James wrote:

Jesse wrote:

Not exactly. Every time the software is run, either the default java.io.tmpdir is used or a specified-at-runtime java.io.tmpdir is used for the purpose of outputs. Ideally, by default, humans need not be aware of this setting for the software to do its work. Because humans need not be aware of this setting, and since it will therefore not change from one run to the next, and since the software is aware of its own conventions of directory creation within this specified directory, the software can infer that old outputs may be existing in the java.io.tmpdir and therefore check the criteria for deletion.

I guess I'll put this in the category of "watching with interest" for now. I don't want to hold you up. As a laptop user, I am going to regularly specify an output directory that overrides the Java temp directory, so it will change from one run to the next. I would like WRES to clean-up after me. I would like WRES not to delete data that it didn't create. All of these things imply other things. As I say, watching with interest for now as a developer, but my first concern as a user is the desired behavior.

EDIT: I will add that solving the problem of cleaning up the default Java temp directory of files that were written by WRES is the first priority.

Good feedback. I failed to say the next part which is "if you're sophisticated enough of a user to specify a -D parameter for java.io.tmpdir, you're sophisticated enough to come up with your own cleanup strategy", but I am going to take a step back from that, perhaps. In other words, if you're not conscious of the setting, it will be the same directory and the cleanup strategy succeeds. If you're a machine it is likely the same setting and the same directory and the cleanup strategy succeeds. If you're a human who runs with a different directory frequently, I would have to sleep on that one before coming up with a strategy.

epag commented 3 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2018-10-22T20:15:13Z

Jesse wrote:

James wrote:

Jesse wrote:

Not exactly. Every time the software is run, either the default java.io.tmpdir is used or a specified-at-runtime java.io.tmpdir is used for the purpose of outputs. Ideally, by default, humans need not be aware of this setting for the software to do its work. Because humans need not be aware of this setting, and since it will therefore not change from one run to the next, and since the software is aware of its own conventions of directory creation within this specified directory, the software can infer that old outputs may be existing in the java.io.tmpdir and therefore check the criteria for deletion.

I guess I'll put this in the category of "watching with interest" for now. I don't want to hold you up. As a laptop user, I am going to regularly specify an output directory that overrides the Java temp directory, so it will change from one run to the next. I would like WRES to clean-up after me. I would like WRES not to delete data that it didn't create. All of these things imply other things. As I say, watching with interest for now as a developer, but my first concern as a user is the desired behavior.

EDIT: I will add that solving the problem of cleaning up the default Java temp directory of files that were written by WRES is the first priority.

Good feedback. I failed to say the next part which is "if you're sophisticated enough of a user to specify a -D parameter for java.io.tmpdir, you're sophisticated enough to come up with your own cleanup strategy", but I am going to take a step back from that, perhaps. In other words, if you're not conscious of the setting, it will be the same directory and the cleanup strategy succeeds. If you're a machine it is likely the same setting and the same directory and the cleanup strategy succeeds. If you're a human who runs with a different directory frequently, I would have to sleep on that one before coming up with a strategy.

Actually, I'm going to step back from what I said.

If I'm overriding the temp directory, then perhaps I'm signalling to the software that I am overriding the default behavior for cleaning up. Afterall, the main reason for cleaning up is that WRES decided where to put the output, so it should also clean up after itself (good citizen). If the user decided where to put the output, there is a reasonable argument to say that the user does not want the WRES to apply the good citizen strategy, because the user wants to keep that data long-term in the place they have specified.

So, actually, I do think we should be focusing on cleaning up the default Java temp directory only (or, rather, the scenario where WRES decides).

epag commented 3 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2020-12-14T21:46:53Z

Complicated by external processes in charge of writing outputs. Those external processes should be the only ones with write access to the locations they write to. So cleanup would need to either ask those processes to do the cleanup or violate that boundary and clean up.