X Ray Image Query continued iteration

The X Ray Image Query requires continued iteration. I will work with @cyrush and Steve Langer on adding features that Steve and others need. These include but are not limited to:

Features for 3.3.1:

[x] simulation time (metadata)
[x] total spatial extent in each direction (metadata) - using near plane projection - #18063
[x] move have conduit guards for setting result message to be cleaner - maybe

3.3.1 deliverables have been completed so I am changing the milestone to 3.3.2.

Features for 3.3.2

Docs:

[x] documentation for the blueprint metadata
[x] docs for imaging plane topos - #18327
[x] docs for energy group bounds - #18327
[x] docs for units
[x] docs for rays - #18398
[x] docs reorg - #18327
[x] document spatial extents coordset and why there are duplicates of the fields - #18398
[x] Make x ray query docs their own page - #18398
[x] document the fact that a lot of what is seen in docs is only true for the newest version of visit - #18398
[x] path length units change - #18398
[x] document spatial extents error messaging - #18398
Metadata:
[x] number of pixels in each direction (metadata) - #18290
[x] pass thru detector height and width metadata? - #18290
[x] palette max/min in bov output and bp output (useful for seeing if arrays are nonzero) - just max and min in blueprint output - #18290
Arguments:
[x] energy group bounds (?) **** - have users pipe thru the z vals - have users send them thru as an argument - #18315
[x] Change the name from energy_group_bins to energy_group_bounds - #18326
[x] units of the fields - have users pipe these thru - #18333
Visualize X Ray Detector:
[x] add imaging plane topology (might as well add near, view, and far plane topos) - #18290
[x] add image plane center as part of metadata (in addition to imaging plane topology) - this is focal point; already comes out
[x] save out the rays - "Blueprint with Imaging plane in 3D space with rays that represent the traced lines -- so you could plot both the dataset and the xray in the same space" - #18362
Other:
[x] promote spatial extents to be a proper coordset; duplicate fields and topo to use spatial extents - #18373
[x] use vector math in avt/Math/avtVector - #18373
[x] maybe all vectors should be represented by avtVectors - #18373
[x] bmp output type warning - #18166

3.3.2 deliverables (aside from docs) have been completed so I am changing the milestone to 3.3.3.

Features for 3.3.3

Docs:

[x] fix the wording for the energy group bounds parameter, it should say that it can be found in the spatial extents mesh, NOT the metadata as it says now. - #18520
[x] blueprint access info - see my notes with the tag BP_ACCESS_INFO - #18522
[x] blueprint array reshape info - see my notes with the tag BP_RESHAPE_INFO - #18528
[x] docs for spectra topo and fields - brand new section + update all examples - #18673

Metadata:

[x] metadata - image_topo_order_of_domain_variables = ["x", "y", "z"] for Tanim, be explicit about x varies fastest, then y, then z - #18498
[x] change metadata to pot_hole_case - #18498

Meshes:

[x] 1D output spectrum curve - collapse x and y dims of spatial topo into 1d, sum intensity and path length - do we scale? probably not - or just write both, so 4 fields - #18576
[x] add a new topo - "spatial energy reduced" - 2d coordset where x and y are in physical space, the field values are the sum of every intensity or path_length value scaled by the bin widths (added together) - #18557 - #18564

Other:

[x] Add "Error" to all the error messages to make it consistent and easier to find out - #18576
[x] audit failures to either throw an exception or return
[x] tests that have array variables - #18576

3.3.3 deliverables (aside from docs) have been completed so I am changing the milestone to 3.4.

Features for 3.4

Docs:

[x] Add my docs gen script to the visit repo - #18636
[x] make the various numbered sections underneath troubleshooting, introspecting w/ python, and visualizing w/ visit into proper subsections for easy access - #18650
[x] use doecgf talk info to enhance the docs - #18718
[x] make a new section "Quick Results" that has info about the spectra curves and spatial energy reduced mesh; will discuss quick views of data - #18707
[x] examples section has some incorrect info about full versus old view spec. Search for "Note that we have created the dictionary from scratch, rather than starting with the default ones. This is necessary to use the full view specification." - #18971
~~[ ] discuss how to get transmission~~
[x] put my doecgf talk into the docs - #18718
[x] make all the output pieces sections link to the correct visualization and python introspection sections - #18693
[x] reorganize docs so that output pieces are down a level under a larger "output pieces" section - #18698
[x] #18860

Bugs/investigations:

[x] From Steve: "In the past I have run into cases where Visit would decide that square pixels are required and adjusted things to meet that requirement. I want to be able to set the detector height, the detector width, the number of pixels in x, the number of pixels in y, and have Visit honor all 4 choices. You could add an option pixels_square. If set to zero (default), Visit does what I requested above. If it is one, Visit adjust things to get square pixels. I don’t see the need for an option to force square pixels, but someone might be relying on that behavior." - #18781
[x] strange intensity results with up to 3 energy group bins, investigate using script I made called justins_sad_adventure.py - I have solved this; the results were due to the fact that absorbtivity and emissivity coefficients that are the same cancel each other out when divide emis by absorb is set.
[x] new conduit bugfix reads in yaml and json; update tests to take advantage of this - #18667
[x] tests are failing because os.listdir is not consistent with results. Sort the filename arrays. - #18895

Other:

[x] filename_scheme #18030
[x] remove bmp output type - #18193
~~[ ] add a parameter image energy limits: 2 numbers - for example: 8 kev and 50 kev - energy range; only interested in doing the bins that fall between those values; inclusive range~~
~~[ ] also would be nice to limit the times - get a subset of the times; if times are too close together then take a subset~~
[x] #18549 - #19011 (for the x ray query)
[x] log when you are hitting the pitfalls case in the docs - #18748
[x] #18764
[x] warning when you do not give the right number of energy values - #18871
[x] put the filenames into a mapnode that is turned into xml so that users can access a list of the filenames - #19011

~~Changing the milestone back to 3.3.3 since I am adding in a few more features for the upcoming release.~~

3.4 deliverables (aside from docs) have been completed so I am changing the milestone to 3.4.1.

Features for 3.4.1

Docs:

[x] all of the examples might be wrong b/c of my changes to default camera params - #19350

Bugs/investigations:

[x] #19087
[x] #19088

Other:

3.4.1 deliverables have been completed so I am changing the milestone to 4.0, as we are ramping down work on the X Ray Image Query.

Future Deliverables:

Docs:

[ ] add a quickstart guide - maybe once steve has his dante quickstart guide up and running
[ ] add visit gui instructions for the visualizing with visit section
[ ] Steve: "We have an asymmetry between the coordinate axes (x and y) and the energy axis (z). A coordinate axis can be specified by (xmin, xmax, nxpix) while the energy axis needs to have a vector of energy bounds because the points are not equally spaced. There is nothing wrong with this approach, but it means it will be easy for users to become confused unless we have clear documentation. I refer to “the energy axis” or “the third array dimension” when I talk about this to another physicist."
[ ] use the teacup /usr/gapps/visit/data/tcup_imgvol/ to make cool docs pictures
[ ] code example in the docs that shows how to perform per-energy group slices on the query output with visit given energy group bounds
[ ] how to ray trace hydra - python example of the script I am working on for steve
[ ] we want to explain how to limit the time steps with a python code example - have an example in the x ray query docs for running across a time range - explain to change the indices to modify the time range
[ ] make an image energy limits tutorial - for example: 8 kev and 50 kev - energy range; only interested in doing the bins that fall between those values; inclusive range
[ ] steve preprocessing black body function applied to emissivities - use visit expression language to change the emissivities instead of changing the query - No! Instead I should have a paragraph about how different sim codes produce different emissivity results. Code needs to dump emissivity or code team needs to provide visit team with correct expr to compute it. A "status of supported codes" section
[ ] document that the query returns a python object with information, how to use this to check if the query failed

Bugs/investigations:

[ ] bugfix: "...crash if you use many more MPI tasks than there are mesh domains. In his case, he was using 360 MPI tasks on a 144 domain problem. It may actually be related to domains x groups, b/c it doesn’t fall over right after 144 MPI tasks. I was able to use 180 just fine where 360 was clearly not working. Just wanted to let you know that you could face a crash if you run with many more MPI tasks than domains." - #18430
[ ] what is the ratio of time spent on i/o versus ray tracing? - is it best to run with lots of cores? what is the best way to run? - timing would be nice - for every timestep, record time when you start query, record time for i/o finishing, record for ray trace start and end, time for end of timestep (writing the output file) - mpi wtime?
[ ] importance sampling pre-processing

Other:

[ ] feature parity with https://github.com/Alpine-DAV/ascent/pull/1354 - add up vector, look at vector, line mesh that has the whole frustrum.
[ ] #18422
[ ] add the docs examples to the tests and have the docs reference the test code

Ideas:

[ ] take advantage of modern cpp language features
[ ] the image writer calculates data max and data min, which we already do for blueprint. Maybe they can share this info instead of doing it twice?
[ ] The documentation should mention that the “path length” is an optical depth (a pure number), not a distance (units of centimeters, or some other distance unit).
[ ] Explore making the output type (and the new filename scheme) into enums
[ ] Address the problem of falling off the end... we go to 9999 for family files and to 100 for bov/bof files. It would be good if there were either error messages or just more digits used.
[ ] transmission (new field output) - the ratio of the intensity at the current point to the intensity at the starting point (exp(-tau)) - see email/notes line 800 - investigate how the ray tracing works and where the field vals come from - avtXRayFilter.C
~~[ ] optical depth (tau) (new field output) - the negative of the log of the transmission - this is probably the same as the path_length - need to verify~~
~~[ ] direction in which rays travel (metadata) - does Steve want us to save out all the rays (in another field?)~~
~~[ ] is there a better name for viewAngle~~
[ ] Instead of int filenameScheme with 3 magic numbers (currently)... why not std::string filenameScheme where "" means what NONE means and "$Q%F means to render datum Q according to (optional) printf-spec F. "$c%06d" means what CYCLEFILES means here and "$s%04d" means what FAMILYFILES (or maybe "$i%04d" where i=index) means here but where but where users can then easily control the number of digits as well as some of the way it gets formatted into the filename including "$t%8.4g" could render a time value instead of cycle or index or (borrowing from this stuff to some extent.
[ ] for cases where a sim code provides energy group bins (usually via array_compose with bins) recipe to propagate this via blueprint outputs - 1) if they are in the data, we can perhaps gain access to them in python and do everything behind the scenes, 2) ditto for c++ - "In the future, you could modify the query to pick up the energy group array from the opacity dump files. Visit already recognizes dumps from different rad-hydro codes and that could be used to select a code-specific function to read the energy groups. If a code does not save the groups (I claim it should), users would need to rely on the current approach. I know that HYDRA and Lasnex save the energy groups because I have used them a lot in the past."
[ ] upgrade x ray filter, use avtVector there too
[ ] The ability to pass custom info through the query, perhaps extra dict values that get sent to blueprint output. See notes.
[ ] consider deprecating more of the non-conduit output options

visit-dav / visit