visit-dav / visit

VisIt - Visualization and Data Analysis for Mesh-based Scientific Data
https://visit.llnl.gov
BSD 3-Clause "New" or "Revised" License
427 stars 109 forks source link

Consolidate ASCII input to uber plugin #34

Open markcmiller86 opened 5 years ago

markcmiller86 commented 5 years ago

We have several plugins supporting ASCII input; Curve2D, XYZ, PlainText probably others. They all have to deal with similar issues, bad chars, overflow, creation of mesh or curve objects, read options, etc.

We should consoidate all this functionality to a single plugin with various options

We don't really handle big curve data too well. We take forever to read it. We don't scale with it because it is treated as single block (though multi-block curves are now supported), we store it in memory of mdserver and engine, and viewer (as its renderable form). It can be several minutes to get an image for a curve of 100 million points. We ought to be able to handle that case ok.

markcmiller86 commented 1 month ago

@JustinPrivitera there is a note in the opening comment of this issue regarding performance for Curve plugin (ascii data) for curves with millions of points. Its slow for sure. That just seems like bizarre behavior given VisIt supposedly "scales well".

I do think there are a large number of ascii formats that are trivially decomposeable (e.g. that don't require the hammer of Conduit/Blueprint export) into a .visit file to handle correctly. But, we could probably just "do the right thing" reading such a file into a parallel VisIt.

I am in the planning stages of that exact kind of behavior for very large STL files (on both read and write). It might be worth comparing notes and/or combining efforts to identify all the common/shared functionality for such cases.

JustinPrivitera commented 1 month ago

@markcmiller86 Sure I'm happy to chat. Turning it into Blueprint and partitioning into multidomain is fairly simple.