search

HEXRD / hexrdgui

Qt6 PySide6 based GUI for the HEXRD library.
Other
29 stars 13 forks source link

Precision limitation leading to rounding error in omega specs in image loader #714

Closed joelvbernier closed 3 years ago

joelvbernier commented 3 years ago

I just noticed this one:

I have an imageseries set from chess with 1441 frames spanning (0, 360). Thus the delta omega reflected in the metadata on the imageseries object is 360./1441 ~ 0.249826509. However, when I load these imageseries series via the HEDM-style "Load Data" widget, I see the following:

image

The precision limit of 2 decimal places on the delta omega is leading to an erroneous omega max value of 360.25; now, I am not entirely sure how these numbers are getting used, since the imageseries already has metadata attached (@psavery or @bnmajor could you tell me how/where these values get used? Obviously, if you are loading raw images, this is the only way to input the omega metadata, so I imagine these values are indeed important)

This bad omega information would cause find-orientations and fit-grains to fail, the range cannot exceed 360. In this case, I cannot enter the accurate delta omega manually, and the omega max seems to be slaved to the delta omega value. We will at the very least require a higher degree of precision in these fields to avoid this rounding error (single precision should suffice). We should also raise a warning if the range exceeds 360.

joelvbernier commented 3 years ago

Looks like I'm getting a TypeError from this PR; reopening

Traceback (most recent call last):
  File "/Users/joel/Documents/GitHub/hexrdgui/hexrd/ui/load_panel.py", line 202, in select_images
    self.create_table()
  File "/Users/joel/Documents/GitHub/hexrdgui/hexrd/ui/load_panel.py", line 399, in create_table
    str(round(self.delta[i], 6)))
TypeError: type str doesn't define __round__ method

is self.delta coming in as a string?