Proper Handling of Background Term in Invariant Analysis (Trac #958)

[butlerpd]

We have recently seen a couple of issues with the invariant failing. These have been traced back to using too large a background term which is a user input field. The traceback error is not terribly helpful to the user and in fact this should not cause a failure unless perhaps if massively overestimated.

Lionel Porcar is now asking how the Invariant calculation handles negatives at high Q when subtracting background. This needs to be checked but could be the source of the above problem. If the data set is noisy and reaches background the subtraction will always produce some negative values. Decreasing the background till all points remain positive after subtraction is in fact '''incorrect'''!!! It will provide an overestimate of the invariant which could be substantial since q is large at this point.

Aside from ensuring a proper calculation, providing an automated background estimate as for Pr could be helpful. Also it may be useful for both invariant and Pr to allow for an array to represent the background for instruments (such as at pulsed sources) where the background may not be flat?

Migrated from http://trac.sasview.org/ticket/958

{
    "status": "new",
    "changetime": "2018-02-13T02:55:21",
    "_ts": "2018-02-13 02:55:21.540988+00:00",
    "description": "We have recently seen a couple of issues with the invariant failing. These have been traced back to using too large a background term which is a user input field.  The traceback error is not terribly helpful to the user and in fact this should not cause a failure unless perhaps if massively overestimated.\n\nLionel Porcar is now asking how the Invariant calculation handles negatives at high Q when subtracting background.  This needs to be checked but could be the source of the above problem.  If the data set is noisy and reaches background the subtraction will always produce some negative values.  Decreasing the background till all points remain positive after subtraction is in fact '''incorrect'''!!! It will provide an overestimate of the invariant which could be substantial since q is large at this point.\n\nAside from ensuring a proper calculation, providing an automated background estimate as for Pr could be helpful. Also it may be useful for both invariant and Pr to allow for an array to represent the background for instruments (such as at pulsed sources) where the background may not be flat?",
    "reporter": "butler",
    "cc": "",
    "resolution": "",
    "workpackage": "SasView Bug Fixing",
    "time": "2017-04-30T16:26:55",
    "component": "SasView",
    "summary": "Proper Handling of Background Term in Invariant Analysis",
    "priority": "major",
    "keywords": "",
    "milestone": "SasView 4.3.0",
    "owner": "",
    "type": "defect"
}

[butlerpd]

Trac update at 2017/10/27 08:07:58: butler commented:

Conversations with Richard Heenan and Andrew Jackson leads to the conclusion that we in fact should not worry about a Q dependent background and instead ensure the documentation makes clear that the data must be background subtracted to the point that any residual background must be Q independent.

Need to modify docs besides ensuring the code behaves correctly. These are computational and should be done sooner than later so part of 4.2 or 4.3 Adding an option to auto find the background should be a 5.0 task.

This ticket should therefore only refer to the first part. A new ticket to be created for 5.0 will address the auto background estimate.

[butlerpd]

Trac update at 2018/02/13 02:55:21: butler changed milestone from "SasView 4.2.0" to "SasView 4.3.0"

[ajj]

Not really sure what is actually left to fix here ... is it the dealing with negative values issue?

[butlerpd]

Given #1087 I guess it is all about the documentation? In other words, reading the issue I believe the issue is that new people (and sometimes older folks like me who forget to engage their brains) don't always realize the importance of proper background subtraction?