Have output returned as it is written instead of at the end - Githubissues

epag commented 3 weeks ago

Author Name: Evan (Evan) Original Redmine Issue: 122191, https://vlab.noaa.gov/redmine/issues/122191 Original Date: 2023-11-02 Original Assignee: Evan

Current the evaluation is synchronous and all output is returned at the end. Return output as it is written to remove this all or nothing approach

epag commented 3 weeks ago

Original Redmine Comment Author Name: Evan (Evan) Original Date: 2023-11-02T19:16:31Z

Going to start looking into the best way to do this and tackle this ticket next

epag commented 3 weeks ago

Original Redmine Comment Author Name: Evan (Evan) Original Date: 2023-11-03T13:57:13Z

Note to self:

Also remove or repurpose wres_job directories in /mnt/wres_share/evaluations

epag commented 3 weeks ago

Original Redmine Comment Author Name: Evan (Evan) Original Date: 2024-02-07T13:36:02Z

Working on this while I wait for James to do some work on #89868

epag commented 3 weeks ago

Original Redmine Comment Author Name: Evan (Evan) Original Date: 2024-02-07T14:39:19Z

Hm, so have been thinking about this. The @Evaluator@ is still creating a directory and it should be writing output as it is working. So this is really a question of how do we become aware of the evaluation id (And therefore the output path) at the start of an evaluation. If we can become aware of that BEFORE any of the actual evaluation starts, then we could just leverage the same @JobOutputWatcher@ that we were prior.

Let me know if yall have any thoughts, otherwise I will proceed with option 3 I currently see a few options (Assuming having the output presented to users as it comes in/have output if there is a failure mid way through is viewed as necessary): Option 1, Change flow of Evaluations (Least prefered):

We basically ask the Evaluator to prepare an evaluation
We get the output path we will be working with
We establish a watcher
We start the evaluation

Option 2, The Evaluator talks to the broker (Not sure if this is possible/something we even want):

The Evaluator knows the directory it is talking to
The Evaluator starts a job output watcher

Option 3, Ignore real time, just get all output at the end (What I think is best):

Ignore the real time returning of evaluation output
If an evaluation fails, we still get a return with the Eval ID
Check the output directory of that evaluation and return the output present in that folder at the end

epag commented 3 weeks ago

Original Redmine Comment Author Name: Evan (Evan) Original Date: 2024-02-07T19:06:36Z

Current thread I am pulling on (Which will rely on the fact that there is only ever 1 evaluation per server at a time) is:

In the @WresEvaluationProcessor@ on the shim we will manually set the System Property "wres.jobId", this doesn't seem to actually be used anymore since we moved to the server architecture
This should create the new directory like we used to have
We can attach a watcher to that directory just like before

The only side effect is that now evaluation folders are going to be inside in a job folder. This can tie the two together honestly and all of the helper scripts should still work. Doing some testing now

epag commented 3 weeks ago

Original Redmine Comment Author Name: James (James) Original Date: 2024-02-07T19:13:07Z

Not exactly sure about the purpose of this ticket. I think it's to stream paths as they are produced or something? I'm not sure that is a great objective. Streaming the actual evaluation outputs themselves as they are produced (i.e., not files, but blobs of statistics/protobufs), might be a useful thing to do, but is in the distance future, I think. This would require exposure (to external clients, not just internal ones) of the amqp messaging api or something fronting that, which is documented in other tickets.

Otherwise, there should be no inspecting of output directories or anything like that, which is the horrible solution we just ditched (or I thought we had, but I guess not). The @Evaluator@ already returns the paths written in the @ExecutionResult@ and that is what should be used by the caller, not an indirect inspection of an output directory. But that necessarily happens at the end of an evaluation. Again, streaming of the outputs/statistics (not files) could be something to expose in the future - we already do it internally - this is our statistics messaging api.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Evan (Evan) Original Date: 2024-02-07T19:16:56Z

James wrote:

Not exactly sure about the purpose of this ticket. I think it's to stream paths as they are produced or something? I'm not sure that is a great objective. Streaming the actual evaluation outputs themselves as they are produced (i.e., not files, but blobs of statistics/protobufs), might be a useful thing to do, but is in the distance future, I think. This would require exposure (to external clients, not just internal ones) of the amqp messaging api or something fronting that, which is documented in other tickets.

Otherwise, there should be no inspecting of output directories or anything like that, which is the horrible solution we just ditched (or I thought we had, but I guess not). The @Evaluator@ already returns the paths written in the @ExecutionResult@ and that is what should be used by the caller, not an indirect inspection of an output directory. But that necessarily happens at the end of an evaluation. Again, streaming of the outputs/statistics (not files) could be something to expose in the future - we already do it internally - this is our statistics messaging api.

Gotcha, yeah, I was leaning toward just being fine with the output just be returned at the end, but this was a "gap" in functionality to what we had. If we agree that this is unneeded then I will just reject this ticket

epag commented 3 weeks ago

Original Redmine Comment Author Name: James (James) Original Date: 2024-02-07T19:41:15Z

I think so. The main thing would be to get the statistics back as they are produced but the only solution to that is exposing the "advanced" async amqp messaging api, which is what our internal format writers listen to. Until then, it's necessarily sync without "looking into the kitchen" via some ugly and unreliable tricks, like watching an expected dir.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Evan (Evan) Original Date: 2024-02-07T19:43:22Z

James wrote:

I think so. The main thing would be to get the statistics back as they are produced but the only solution to that is exposing the "advanced" async amqp messaging api, which is what our internal format writers listen to. Until then, it's necessarily sync without "looking into the kitchen" via some ugly and unreliable tricks, like watching an expected dir.

So am I understanding you correctly that you believe that this ticket should be rejected in favor of a more wholistic solution like you are mentioning?

epag commented 3 weeks ago

Original Redmine Comment Author Name: James (James) Original Date: 2024-02-07T19:55:40Z

I think so. Afterall, any solution to this is necessarily built on tribal knowledge about what the core app is doing internally, namely writing to a particular dir that can be "watched". It's also risky because many of these paths are pointing to files that are written incrementally, so there is nothing that can/should reasonably be done with them until they are finished, which is not until the evaluation is complete. Again, the only good solution is to expose the amqp messaging api, which broadcasts individual (completed) blobs of statistics as they are produced. For example, imagine someone looking at a GIS tool like google earth and waiting for some evaluation statistics to come in location-by-location and be plotted on a map as they are produced. That would be possible with the advanced messaging api and is a reasonable thing to aim for. Exposing paths early doesn't have many upsides I can think of and quite a few downsides.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Evan (Evan) Original Date: 2024-02-07T19:57:07Z

James wrote:

I think so. Afterall, any solution to this is necessarily built on tribal knowledge about what the core app is doing internally, namely writing to a particular dir that can be "watched". It's also risky because many of these paths are pointing to files that are written incrementally, so there is nothing that can/should reasonably be done with them until they are finished, which is not until the evaluation is complete. Again, the only good solution is to expose the amqp messaging api, which broadcasts individual (completed) blobs of statistics as they are produced. For example, imagine someone looking at a GIS tool like google earth and waiting for some evaluation statistics to come in location-by-location and be plotted on a map as they are produced. That would be possible with the advanced messaging api and is a reasonable thing to aim for. Exposing paths early doesn't have many upsides I can think of and quite a few downsides.

Sounds good, ill retire this ticket and make a new one

NOAA-OWP / wres

Have output returned as it is written instead of at the end #140