Optimize Time Query Curve Plot

[aowen87]

Is your feature request related to a problem?

This is related to #3577

Impact (Please click check boxes AFTER submitting ticket)

• [x] High - User productivity significantly improved
• [ ] Medium - User productivity partially improved
• [ ] Low - User productivity trivially improved

Is your feature request specific to a data set?

This could potentially be utilized in any dataset that contains time spans.

Describe the solution you'd like.

We'd like to be able to quickly create a curve plot of an element's variable(s) through time. Currently, this is a very slow process, especially for datasets with large time spans.

One possible solution is to directly query the database for the desired variables over the desired time span. Currently, when performing these curve plots, the entire mesh is loaded for each time step, the desired element is queried at each time step, and the query results are used to create the curve plot. By querying the database directly, we would bypass loading the mesh, adding the operators, etc. for each time step. Instead, it would be a single query.

One potential downside to this is that bypassing loading the mesh and adding operators means that the query would only be able to use the original data. I'm not sure how many situations this would complicate... I can think of one off the top of my head: creating a curve plot of node position with a displacement operator.

[markcmiller86]

Another advantage of delegating the time-looping logic all the way down to the plugin is that each timestep query does not involve round-trip messaging between client and server. I may be wrong but I thought that also comes into play.

That said, I am not sure delegating all the way down to an individual plugin is the right thing to do. I think the right thing involves a combination of adding to plugin API but also adding time-looping logic just above all plugins, maybe in avtGenericDatabase.

If we added to the plugin interface just the ability query a set of variables at a specific set of nodes or zones...something like...

GetMeshVariablesAt(vector<string> varnames, vector<int> mesh_elem_ids, int dom)

where encoded into mesh_elem_ids is whether the ids specify a node or zone (or maybe an extra arg for that) and we get back a vector<double> (regardless of database's native types for the queried variables).

Then, some formats can choose to implement this whereas the default implementation, in avtGenericDatabase, is to ask a plugin to read the whole mesh, read each whole variable, and then extract the desired values -- which is basically what is happening now anyways.

Silo format could, for example, use low-level Silo calls to perform these queries without reading the whole objects. Mili plugin could do same.

The desireability of what I am describing here is that plugin developers don't have to manage all the looping logic, etc. down in the plugin. Only the logic to smartly read only the bits and pieces of data necessary to service the query is handled there.

If this is already what you are planning, my apologies. Otherwise, I hope this comment inspires some further design discussion from everyone.

[aowen87]

I definitely agree with the round-trip messaging likely having a significant impact on time as well.

I hand't really come to any decisions about how to handle the time looping yet, but what you're suggesting makes a lot of sense to me if we were to add a timestep variable to this "GetMeshVariablesAt" method.

[markcmiller86]

if we were to add a timestep variable to this "GetMeshVariablesAt" method.

Doh! yes, you are right.

And, it also occurs to me that we might need to think a bit harder about best way to handle MTxD type databases as my thinking cap, above, was only for STxD databases.

[aowen87]

So, one problem with moving the time iteration outside of the database plugin is that we loose some efficiency... This is because we need the data to be organized as

curves = [ [e1_v1_t1, e1_v1_t2, ..., e1_v1_tN] , [e1_v2_t1, ..., e1_v2_tN], ...]

where "eI_vJ_tK" is the Jth variable of the Ith element at time step K, where there are a total of N time steps. The order of the element/variable combinations doesn't matter, but we do need to preserve that order of time steps, which means our inner most loop is time:

timeSpans = [ ]
for e in elements do:
    for v in variables do:
        singleSpan = [ ]
        for t in times do:
            e_v_t = value at e, v, t
            singleSpan.append(e_v_t)
        timeSpans.append(singleSpan)

(e and v loops can be rearranged, but t must be innermost for structure efficiency)

If the time looping is controlled by one of the higher-up levels, then this forces our looping to look something like this:

for t in times do: (controlled by database class)
    for e in elements do: (now controlled by plugin)
        allVarsSingleTime = [ ]
        for v in variables do:
            t_e_v = value at t, e, v
            allVarsSingleTime.append(t_e_v)

(we would then need to restructure the data to be in the appropriate format)

This isn't terrible, but I'm guessing we might take some degree of a performance hit. I can run some tests to see just how much of a hit that would be.

[aowen87]

I have a very initial draft of the new query over time that I'm testing (I'll call it Direct DB QOT for now), so I thought I'd post some initial performance comparisons. First, here are some things to keep in mind:

1. I have not yet optimized the plugin to retrieve these values efficiently.
2. I'm currently letting the plugin control all looping (including time).
3. I'm only tracking the amount of time spent retrieving the curves (does not include plot time).

These results come from query over 1600 time steps. Original QOT: 4.14 seconds Direct DB QOT: .01 seconds

Kevin is working on getting me a dataset with an even larger time span, so I'll post some more results when I get these larger time span results.

[brugger1]

Wow, that’s an impressive speed up!!!

From: Alister notifications@github.com Sent: Tuesday, August 6, 2019 12:08 PM To: visit-dav/visit visit@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [visit-dav/visit] Optimize Time Query Curve Plot (#3718)

I have a very initial draft of the new query over time that I'm testing (I'll call it Direct DB QOT for now), so I thought I'd post some initial performance comparisons. First, here are some things to keep in mind:

1. I have not yet optimized the plugin to retrieve these values efficiently.
2. I'm currently letting the plugin control all looping (including time).
3. I'm only tracking the amount of time spent retrieving the curves (does not include plot time).

These results come from query over 1600 time steps. Original QOT: 4.14 seconds Direct DB QOT: .01 seconds

Kevin is working on getting me a dataset with an even larger time span, so I'll post some more results when I get these larger time span results.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/visit-dav/visit/issues/3718?email_source=notifications&email_token=AAING7RR4CHTUV662472CXDQDHDY5A5CNFSM4II77YZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3WE5KQ#issuecomment-518803114, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAING7VV56ZTPQ2W4TMENHDQDHDY5ANCNFSM4II77YZA.

[aowen87]

More timing results. I ran each of these 2-3 times, so I'm putting the min and max times from those runs. Everything is still in seconds, and the same points to keep in mind from above are still in play.

10,000 time steps Original QOT: 9.57 -> 10.39 Direct DB QOT: .15 -> .18

100,000 time steps Original QOT: 117.96 -> 128.79 Direct DB QOT: 1.62 -> 1.62

[brugger1]

Awesome results! 2 minutes to 2 seconds is a huge improvement and makes it usable.

From: Alister notifications@github.com Sent: Tuesday, August 6, 2019 4:15 PM To: visit-dav/visit visit@noreply.github.com Cc: Brugger, Eric brugger1@llnl.gov; Comment comment@noreply.github.com Subject: Re: [visit-dav/visit] Optimize Time Query Curve Plot (#3718)

More timing results. I ran each of these 2-3 times, so I'm putting the min and max times from those runs. Everything is still in seconds, and the same points to keep in mind from above are still in play.

10,000 time steps Original QOT: 9.57 -> 10.39 Direct DB QOT: .15 -> .18

100,000 time steps Original QOT: 117.96 -> 128.79 Direct DB QOT: 1.62 -> 1.62

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/visit-dav/visit/issues/3718?email_source=notifications&email_token=AAING7R5L4ENIWBZA36JEWDQDIAYXA5CNFSM4II77YZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3WXEGY#issuecomment-518877723, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAING7TCGJ3SH7M5UQI3VHDQDIAYXANCNFSM4II77YZA.

[aowen87]

I've implemented a "default" collection method that exists outside of the database plugin and can be used for any reader. To test its performance, I disabled the mili plugin collection method and just used the default one on the same datasets that I used in my previous timings. Surprisingly, it performs at least as well as using the method that's down in the plugin (in some cases, it was even slightly faster):

10,000 time steps Plugin Version: .15 -> .18 Default Version: .16

100,000 time steps Plugin Version: 1.62 Default Version: 1.57

This might be due to not fully optimizing the plugin collection method. Doing so could still give us a slight advantage over the default method, but I think this default method should be more than good enough for any realistic use cases. This also simplifies the process of integrating this into use with multiple readers.

[markcmiller86]

@aowen87 this is awesome!

[aowen87]

On second thought, I think I may need to allow the possibility of implementing this at the plugin level as well.

For Mili datasets, a large number of the available "scalar" variables are actually components of vectors, arrays, and tensors that are exposed through expressions. In order to plot this kind of expression variable through the "default" route, I would need to know

1. that the requested variable is a component (could determine through try/catch).
2. the index of the component (not sure if this is available here...).

This is handled in the original QOT by offsetting this work to the pick query, which queries the vtkDataset that has the components defined as scalars. I'm not sure of any way to get at this info from the FileFormatInterface. Determining the above two conditions in the Mili plugin is pretty simple, but this could cause issues in other database plugins that render individual components of vectors/scalars/tensors through expressions. I'm not sure how common this is outside of Mili, though...

[markcmiller86]

On second thought, I think I may need to allow the possibility of implementing this at the plugin level as well.

I think this is true in any case for MTXX plugins, right? I mean for an MTXX plugin (multi-time), there is no abstraction above the plugin that represents the loop over timesteps. For MTXX plugins, the whole query essentially needs to be delegated to the plugin. For STXX plugins, that is not the case.

We should probably have a larger discussion about this...the impact of expressions is a bit of a challenge because expressions are evaluated over a whole mesh at one time instance above a plugin whereas what we want is that expression evaluated for some mesh elements, over all time. It smells like we have to evaluate a whole-mesh expression every time instance in order to get the single (or teeny tiny subset) of mesh elements we need that expression evaluated on in order to service the QoT.

[aowen87]

For the default version, MTXX are handled the same as STXX; I iterate over the requested time steps (if valid), activate them, and then call "GetVar" to retrieve the variable and parse out the requested elements. This is all done in the base FileFormatInterface class.

[markcmiller86]

So, having not thought about this anywhere near as much, I am all of the sudden wondering about a few things...

1. What happens if the expression is an expression function like MinMax or something else that involves the whole mesh variable? In that case, each timestep involves a full read and there is no savings to be gained by possibly delegating that read down to the plugin where a partial read could effectively be faster, if available.
2. Are you indeed doing (or planning) partial read in the Mili plugin?
3. Is the basic approach here to overload GetVar (or GetVectorVar) with an optional argument that lists the mesh elements of interest?

[aowen87]

1. At this point, the savings aren't coming from only reading requested elements; I'm reading in all elements and then selecting which ones to keep. I'm avoiding loading in the mesh position at each time step, which might help some with performance, but it seems like largest benefits are coming from avoiding all of the overhead that comes with performing a full pick at each time step. The expressions are still problematic for other reasons, though (how would the FormatInterface know if a variable is an expression and how to calculate it?).
2. I was planning on doing this, but, after talking with Kevin, it looks like this won't be possible with the current implementation of Mili; Mili will only serve up all elements defined for a variable, and it is then up to the user to parse out which elements are needed.
3. Not exactly... I would have done something like this as an intermediate step to accomplishing step 2, but the primary retrieval method is currently implemented as something like this:

   
   GetTimeSpanCurves(domain, variables, elementIds, tsRange, stride)
   {
   Do a bunch of stuff
   ...
   Loop over time, variables, and elements
       Call GetVar or GetVectorVar
       Add to collected curves
   ...

return curves }

[aowen87]

One potential route for handling expressions would be to just use the original QOT when expressions are encountered and use the new Direct DB QOT for all other variables. Downsides:

1. Expression variables would not see the benefits of this optimized time query.
2. I'd need to do some refactoring in Mili. Currently, most of the scalars are actually vector/tensor/array components that are rendered through expressions. I would need to convert these from expressions to true component arrays. I believe most of the infrastructure needed for this is already there.

I'm not really sure what other options there are... It would be nice if I could apply an expression operator at the database level, but that seems unrealistic.

[markcmiller86]

So, correct me if wrong but the basic issue with Expressions is avoiding evaluating the expression everywhere to return the expression for only the elements it was requested?

• Expression variables would not see the benefits of this optimized time query.

So, for complex expressions that seems totally reasonable. What about expressions that are just aliases for database primals? Seems like those could be handled by doing the indirection up front. I guess that is a slippery slope though too.

Seems like broader brainstorming with the team might help shake out some other ideas.

[aowen87]

Yeah, that is the main issue at hand. And evaluating the more complex expressions through their current implementations (I think...) would require attaching the retrieved values to a dataset and basically doing what the original QOT is doing.

So, for complex expressions that seems totally reasonable. What about expressions that are just aliases for database primals? Seems like those could be handled by doing the indirection up front. I guess that is a slippery slope though too.

This is where I'm struggling. For mili types, I'm currently handling these simpler expressions in the plugin implementation because it's just easy to do that down there. It would be nice to handle this more generally for other DB types as well, but yes... slippery slope.

I think a broader brainstorming with the team would be very useful at this point. I'll be on site next week, so I'll make sure to sit down with some more of the team to hear their thoughts/ideas.

[markcmiller86]

3. Not exactly... I would have done something like this as an intermediate step to accomplishing step 2, but the primary retrieval method is currently implemented as something like this:

GetTimeSpanCurves(domain, variables, elementIds, tsRange, stride)
{
    Do a bunch of stuff
...
    Loop over time, variables, and elements
        Call GetVar or GetVectorVar
        Add to collected curves
...

return curves
}

I did have one though on the above.

1. Define additional methods in database interface for the specific list of elements versions of GetMesh, GetVar, GetVectorVar
2. Have a default implementation for these at the FormatInterface classes (or there abouts) that just turn around and call the normal GetMesh, GetVar, GetVector var method (e.g. the whole mesh method) and then cull out the parts that aren't needed.
3. Allow the default implementations of the above to be overridden in a database plugin. So, those that can do partial I/O can, optionally and transparently, choose to implement it and enjoy the savings from it.

[brugger1]

I agree, that is a great solution. The expressions would then operate on the reduced data set and then the query over time would pull the data out of the reduced data set.

From: Mark C. Miller notifications@github.com Sent: Wednesday, September 4, 2019 4:26 PM To: visit-dav/visit visit@noreply.github.com Cc: Brugger, Eric brugger1@llnl.gov; Comment comment@noreply.github.com Subject: Re: [visit-dav/visit] Optimize Time Query Curve Plot (#3718)

1. Not exactly... I would have done something like this as an intermediate step to accomplishing step 2, but the primary retrieval method is currently implemented as something like this:

GetTimeSpanCurves(domain, variables, elementIds, tsRange, stride)

{

Do a bunch of stuff

...

Loop over time, variables, and elements

    Call GetVar or GetVectorVar

    Add to collected curves

...

return curves

}

I did have one though on the above.

1. Define additional methods in database interface for the specific list of elements versions of GetMesh, GetVar, GetVectorVar
2. Have a default implementation for these at the FormatInterface classes (or there abouts) that just turn around and call the normal GetMesh, GetVar, GetVector var method (e.g. the whole mesh method) and then cull out the parts that aren't needed.
3. Allow the default implementations of the above to be overridden in a database plugin. So, those that can do partial I/O can, optionally and transparently, choose to implement it and enjoy the savings from it.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/visit-dav/visit/issues/3718?email_source=notifications&email_token=AAING7QPU2345KME3C67T2DQIA7YRA5CNFSM4II77YZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD55KCAQ#issuecomment-528130306, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAING7SUHQWFWDFGMPGF2HTQIA7YRANCNFSM4II77YZA.

[aowen87]

1. Define additional methods in database interface for the specific list of elements versions of GetMesh, GetVar, GetVectorVar
2. Have a default implementation for these at the FormatInterface classes (or there abouts) that just turn around and call the normal GetMesh, GetVar, GetVector var method (e.g. the whole mesh method) and then cull out the parts that aren't needed.
3. Allow the default implementations of the above to be overridden in a database plugin. So, those that can do partial I/O can, optionally and transparently, choose to implement it and enjoy the savings from it.

So, are you suggesting to create an entirely new dataset that consists of a reduced mesh only containing the requested elements and then attach the requested variable time spans to this dataset?

[brugger1]

That’s my suggestion.

From: Alister notifications@github.com Sent: Thursday, September 5, 2019 7:37 AM To: visit-dav/visit visit@noreply.github.com Cc: Brugger, Eric brugger1@llnl.gov; Comment comment@noreply.github.com Subject: Re: [visit-dav/visit] Optimize Time Query Curve Plot (#3718)

1. Define additional methods in database interface for the specific list of elements versions of GetMesh, GetVar, GetVectorVar
2. Have a default implementation for these at the FormatInterface classes (or there abouts) that just turn around and call the normal GetMesh, GetVar, GetVector var method (e.g. the whole mesh method) and then cull out the parts that aren't needed.
3. Allow the default implementations of the above to be overridden in a database plugin. So, those that can do partial I/O can, optionally and transparently, choose to implement it and enjoy the savings from it.

So, are you suggesting to create an entirely new dataset that consists of a reduced mesh only containing the requested elements and then attach the requested variable time spans to this dataset?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/visit-dav/visit/issues/3718?email_source=notifications&email_token=AAING7XRFC44F7PTYBPP5J3QIEKPPA5CNFSM4II77YZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD57KQXI#issuecomment-528394333, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAING7TTRXKOV6FK7KSJ2QTQIEKPPANCNFSM4II77YZA.

[aowen87]

That’s my suggestion.

Okay, I think that would work for some of the expressions but not all, correct? If an expression needs information about the entire mesh or all time steps, then the result would be incorrect on this reduced mesh.

[brugger1]

Yes, presumably we would only be doing this in the case where we would have been asking for a subset of nodes/zones anyway.

From: Alister notifications@github.com Sent: Thursday, September 5, 2019 9:36 AM To: visit-dav/visit visit@noreply.github.com Cc: Brugger, Eric brugger1@llnl.gov; Comment comment@noreply.github.com Subject: Re: [visit-dav/visit] Optimize Time Query Curve Plot (#3718)

That’s my suggestion.

Okay, I think that would work for some of the expressions but not all, correct? If an expression needs information about the entire mesh or all time steps, then the result would be incorrect on this reduced mesh.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/visit-dav/visit/issues/3718?email_source=notifications&email_token=AAING7TJXU6STQXKIGFRXXDQIEYOXA5CNFSM4II77YZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD57YXEA#issuecomment-528452496, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAING7VITB6MEQXWU2XH4RLQIEYOXANCNFSM4II77YZA.

[markcmiller86]

So, are you suggesting to create an entirely new dataset that consists of a reduced mesh only containing the requested elements and then attach the requested variable time spans to this dataset?

I guess I am saying that. I didn't think that represents too much of a perturbation from what you are already doing. Perhaps it also addresses the expression integration part of things too or at least more than was is possible with your current approach.

To some extent, this might kinda sorta behave similar to something after having threshold operator applied and the result of that operation being some (small) collection of disconnected elements of the mesh. I think that might be a good metaphor for understanding this new case.

That said, because expression evaluation happens asap after read from database (and before any operators are applied), the thing that is new here and potentially problematic is that we'd be marshalling a smaller dataset (subset formed by explicit list of elements of the whole mesh) through the expression system. Expressions involving neighbors would be problematic unless we agreed to serve up not only the requested elements but 1 or 2 layers of neighbors to ensure neighbor based expressions can complete correctly. But, the final digestion of that in QoT where the curve(s) are assembled might also need to be cognizant of this new thingy it is being served to deal with it correctly.

This is all just open dialog and brainstorming on my part to help stimulate convergence on best solution.

[markcmiller86]

@aowen87 it occurs to me I won't be in on Tuesday next week but can arrange to be either Wed or Thu for a discussion.

[aowen87]

@markcmiller86 Either of those days works for me.

[aowen87]

I had a brain storming session with Eric last week, and here's what we came up with.

We can see two ways of handling the expression system for this optimized QOT:

1. Continue the "array" approach where we query the database directly for the time series arrays, and then apply the expression filters directly to these arrays when appropriate. Pros: a. We don't need to create a dataset to hold these values. b. Don't need to worry so much about setting up a pipeline. c. Fairly simple infrastructure requirements. Cons: a. The expression system's ability to determine which variables are needed for which expressions before the database is spoken to is one of the key abilities we'd like to leverage, and this bypasses that ability. b. Some of the most important expressions for us to have access to, such as array decomposition, require datasets as input, which we would not have using this route.
2. Create an entirely new database retrieval pathway that generates a mesh containing the requested time span. Pros: a. This would allow us to leverage the expression system for simpler expressions. b. This would allow the expression system to handle all expression variable requirements (such as array decomposition). Cons: a. Last week, at least, it was unclear how these datasets would look (or need to look). b. Need to set up a pipeline to retrieve the query mesh. c. More complex infrastructure needed.

Because the cons of the first approach are pretty serious, I decided to implement the second approach and compare the timing results.

Basic Approach: In a nutshell, I implemented new GetMesh, GetVar, GetVectorVar methods as well as supporting infrastructure within GenericDatabase to create a pathway for generating a query over time specific dataset.

The Mesh: The mesh is a polydata point mesh where each point is located at the coordinate (x, 0, 0), where x is either the simulation time, cycle, or time step for the query over time. The dataset also contains a scalar array whose size is equal to the number of time steps. Each scalar entry "i" is the value of a given variable at a given zone or node at the "ith" time step.

Timing: Isolating this method for timing is a bit difficult, so I'm taking a different approach; instead of isolating the retrieval of the time steps, I'm timing from the moment the QOT is initiated to the moment the plot is realized. The times will be slightly slower than my original timings because of this, but they should also be a bit more representative of what a user will experience.

All timings are over 100,000 time steps. "Array version" is my first implementation, which just grabs the arrays from the database. "Mesh version" is the newly implemented version that retrieves a mesh.

Scalar Variable Timing: Original QOT: 108.0 s Direct DB QOT (Array version): 3.29 s Direct DB QOT (Mesh version): 1.7 s

Scalar Variable Decomposed From Tensor Array: Original QOT: 382.5 s Direct DB QOT (Array version): 6.7 s Direct DB QOT (Mesh version): 3.5 s

Drawbacks Currently, I'm only able to retrieve one variable/element pair at a time with the new "QOT Mesh" approach. Getting multiple variable/element pairs to work with a single mesh retrieval seems a bit tricky, but I'm planning on looking into that soon. Also, this will only give valid results for certain types of expressions, so I'll need to find a strategy of determining if a requested expression is valid or not. For invalid expressions, we can always rely on the original QOT.