CIRDLES / Squid

Squid3 is being developed by the Cyber Infrastructure Research and Development Lab for the Earth Sciences (CIRDLES.org) at the College of Charleston, Charleston, SC and Geoscience Australia as a re-implementation in Java of Ken Ludwig's Squid 2.5. - please contribute your expertise!
http://cirdles.org/projects/squid/
Apache License 2.0
12 stars 24 forks source link

Column-swapping in Tasks: a simple zircon-based example #613

Closed sbodorkos closed 1 year ago

sbodorkos commented 3 years ago

Jim @bowring, further to multiple previous discussions and long-promised actions, I have attached a SQUID 2.50 Task that 'post-processes' measured values of 204/206 in order to exclude negative values. These occur when count-rates at background (mass 204.1) exceed those at 204Pb (mass 204), which on the GA instrument happens a bit more often than we'd like.

SquidTask_ILC Zircon 10pk exp=2 AllPos204.GA.xls

The Task replaces values of 204/206 below +1e-7 (both positive and negative) to a revised value of +1e-7. When that revision happens, the associated %err is set to 9999. Spots with 204/206 exceeding +1e-7 retain their original value and %err,

Unfortunately, the SQUID 2.50 process is quite convoluted, because all FO-switched expressions get re-evaluated as the expressions are sequentially executed, so it is important for the user to recognise and avoid potential corruption of the data processing as a consequence of circular references. The absence of an explicit definition of 'original' and 'revised' isotopic ratios means Task-writing is tedious. For example, equation 10 defines ["Orig204/206"] as:

if(["Total 204 cts/sec"]=["Bkrd cts/sec"],1e-32,(["Total 204 cts/sec"]-["Bkrd cts/sec"])/(["Total 206 cts/sec"]-["Bkrd cts/sec"]))

which is the exact implementation of the 204/206 ratio-calculation (for very low count-rates) coded in SQUID 2.50 and Squid3 "under the hood". But in SQUID 2.50, we can't write the much simpler (and equivalent-looking Task equation for ["Orig204/206"]:

["204/206"]

because ["204/206"] is going to be modified by the column-swap, and re-evaluation of ["Orig204/206"] as a consequence of the column-swap will mean that ["Orig204/206"] will end up not being the original measurement of 204/206 after all.

Equations 11-13 evaluate the associated absolute uncertainty of 204/206 at low count-rates, again following the under-the-hood calculations. Equation 14 simply chooses the larger of ["Orig204/206"] and +1e-7 as ["New204/206"], and Equation 15 correspondingly sets the percentage error of ["New204/206"] to either 9999 or the %err associated with ["Orig204/206"].

Equations 16 and 17 showcase the column-swap syntax developed by Ludwig. The general form is ["CalculatedValue"]<=>["OriginalValue"].

The effect of these equations is to force ["New204/206"] into the data-processing space formerly occupied by ["204/206"], and likewise ["New204/206%err"] instead of [+/-"204/206"], and then refresh all calculations (both under the hood and throughout the Task). This is why equation 10 is written the way it is: there is only one ["Orig204/206"], and I do NOT want it re-evaluated!

A by-product of equations 16 and 17 are columns labelled "RAW 204/206" and "RAW 204/206 %err", which are supposed to contain ["OriginalValue"] data as it stood prior to the column-swap i.e. these data should be directly comparable to the outputs of my equations 10 and 13. The idea is that whenever you modify an isotopic ratio using a column-swapping Task, you retain a record of the original values, so you can find your way back if you need to.

As discussed in previous Skype-calls, can Squid3 potentially perform these tasks more easily? Perhaps by reliably identifying "original" and "modified" versions of isotopic ratios for the purpose of expression evaluation, thereby sparing us the need to respecify important calculations from scratch so we don't end up with inaccurate re-evaluations after the column-swap?

Essentially, I would like to be able to write equation 10 as:

["204/206"]

and have Squid3 "know" that I intend that expression to relate solely to the original value of 204/206, and not be re-evaluated in the event of a column-swap. But it's trickier than it looks, because in most situations (and particularly the downstream processing that calculates radiogenic ages), real users do want those expressions re-evaluated as a result of column-swapping.

bowring commented 3 years ago

@sbodorkos - As we have discussed, Squid3 currently models specified ratios to have two pairs of value fields: (ratioVal, ratioFractErr) and (ratioValUsed, ratioFractErrUsed), where the naming derives from the Squid2.5 math and FractErr refers to 1-sigma absolute uncertainty. The first pair - (ratioVal, ratioFractErr) - is assigned values after the "NU-switched" processing and remains immutable. Initially, the second pair - (ratioValUsed, ratioFractErrUsed) - is given the same values. Every expression referring to ["204/206"], for example, is given the values from the second pair - (ratioValUsed, ratioFractErrUsed) to use in its calculations.

The current practical use of this model in Squid3 is in implementing the 204 overcount corrections. When the user chooses the correction (from 207 or from 208), a built-in expression (see: CountCorrectionExpression204From207 and CountCorrectionExpression204From208) is used to calculate the corrected ratio and uncertainty and Squid3 modifies the ["204/206"] ratios accordingly by writing the new values to the ratio's second pair - (ratioValUsed, ratioFractErrUsed). All expressions are then re-evaluated and any reference to ["204/206"] retrieves these corrected values. This change is reflected in the Expression Manager's Peek window for "Ratios with uncertainties". When the user chooses to not use an overcount correction, Squid3 simply copies the values from the immutable first pair - (ratioVal, ratioFractErr) - to the second pair - (ratioValUsed, ratioFractErrUsed).

The general case of column-swapping for ratios is more complicated.

One possible solution is to create three new Squid3 functions to be used in writing custom expressions: 1) ORIG(["204/206"]) or ORIG([+/-"204/206"]) that would ALWAYS return the immutable original NU-switched values.

2) SWAP(["204/206"], ValueModel(value, 1-sigma abs uncertainty, true)) that overwrites the second pair - (ratioValUsed, ratioFractErrUsed) with the warning that internal use of ["204/206"] instead of ORIG(["204/206"]) is a circular reference. These new values from SWAP() would be returned by any reference to ["204/206"]. One complication here is that if overcount corrections were made first, the use of ORIG() would not return these corrected values.

3) RESTORE(["204/206"]) that copies the immutable first pair into the second pair of values so that they would be returned by any reference to ["204/206"].

Another, less-powerful and possibly more practical solution is to replicate the overcount corrections window. In this case, a table would be presented with a row for each ratio. The first column has the names of the ratios, the second column has radio buttons restoring the immutable value to the ratio in the same row, and the third column has radio buttons each selecting the named expression to its right that would be used ONCE to overwrite the second pair of values - (ratioValUsed, ratioFractErrUsed) for the ratio in that row. That expression would be written, named, and managed in the expression manager. The use of ORIG() here might be useful to avoid overcount corrections or to reference another ratio's immutable values. The main advantage of this solution is that Squid3 would guarantee that the expression is only applied once to the ratio, as in the case of overcount corrections.

Have a headache yet?

This promises to be an interesting discussion!

an interesting footnote: expression 10 can be written as: if(["204"]==["BKG"],1e-32,(["204"]-["BKG"])/(["206"]-["BKG"]))

cwmagee commented 3 years ago

So are we talking about 204 overcounts or background overcounts here?

bowring commented 3 years ago

@sbodorkos - After studying this some more and hearing from you that this really only involves 204/206, I think we should modify the 204 over-count corrections screen to allow a choice of expression beyond the two currently built-in. The user would write the expression(s) they want to use to generate new values for 204/206 and it (they) would populate a drop-down box (or similar) that included the current two expressions: CountCorrectionExpression204From207 and CountCorrectionExpression204From208 and a choice to "restore original". The behavior would be the same as present - all unknowns would have their 204/206 ratio altered for the purposes of use in other expressions.

I note that the expressions you supplied in your example file affect RMs too, so we would have to modify the logic to handle that condition. In addition, we could handle other "ratios of interest".

We would introduce a new function "Orig()" that returns the original value of any expression, but meant to be used in the context of Orig(["204/206"]). Thus, your expression "Orig204/206" would be Orig(["204/206"]) and "Orig204/206AbsErr" would be Orig([±"204/206"]).
You could also write

Orig204/206 = ValueModel(Orig(["204/206"]), Orig([±"204/206"]), true)

and, from your excel task:

an expression that could be used in the above-mentioned drop-down box:

New204/206 = 
if(Orig(["204/206"])>1e-7, ValueModel(Orig(["204/206"]), Orig([±"204/206"]), true),
                            ValueModel(1e-7, 9999, false))

Thoughts?

bowring commented 3 years ago

@sbodorkos - I have explored the example task some more and realized that Equation 10 produces the original ratio before the interpolation stage. However, Squid3 currently preserves the interpolated values of the ratios of interest as the "original value." So I need to understand whether the column swapping is to occur before the interpolation or to assume that the interpolation has already happened. For the case of the over-count corrections discussed above, it is the interpolated ratios that are considered original.

Looking forward to your thoughts.

sbodorkos commented 3 years ago

@bowring agreed that this should be restricted to modifications of 204/206 - we don't seem to need to fiddle around with any other ratio from a column-swapping perspective.

Re interpolation, real-world count-rates of 204Pb are usually so low that the SQUID 2.50 code (and our replication of it in Squid3) "steps around" interpolation when determining 204/206. See this bit, which is from near the beginning of https://github.com/CIRDLES/ET_Redux/wiki/SHRIMP:-Step-3

image

However, you are correct in detail - our Orig values are interpolated in those relatively rare instances where 204Pb counts are high enough to permit interpolation in the calculation. In this context, it is worth noting that the existing 204-overcount corrections always "de-interpolate" such analyses (i.e. the overcount-corrected 204/206 values are always calculated using the NON-interpolation arithmetic described in the above screenshot, irrespective of 204 count-rates in specific analyses).

sbodorkos commented 2 years ago

@bowring I had forgotten about most of this! It really is just about ["204/206"], and maybe we can step around the custom calculations for the moment, and just look at the overcount-corrections.

In v1.10.0, I am happy with how the overcount-corrected ["204/206"] values and uncertainties swap into position, based on the radio-button setting in the Common Pb – Unknowns window. All we need is a mechanism for preserving (and reporting, if desired) the raw 204/206 in addition to the the version that ends up occupying the ["204/206"] field.

A [conceptually] simple way to do it would be to calculate original ["204/206"] as we always have (i.e. post-interpolation), and then, as a one-off, Copy its ValueModel to a second "site", named Raw_204Pb_206Pb (this process would be different to simply equating the latter with ["204/206"]). From there, the column-swapping process could work exactly as it already does in the ":Corrected from 207" and "Corrected from 208" scenarios. For "Original 204/206", ["204/206"] and Raw_204Pb_206Pb would remain identical.

People wanting to include both in a Report (which is what I am immediately interested in) could specify ["204/206"] and Raw_204Pb_206Pb separately in their template.

NicoleRayner commented 1 year ago

Hi guys - reviving this discussion on column swapping. I want to do a custom overcount correction on my 4/6 ratio (proportional to the Th concentration of the unknown and subtracting median background cps for the RM). I have attached a squid file here with the assorted expresions involved. This is modified from a squid 2.5 file, so the expressions are inelegant (I feel shame) but before I clean them up extensively I wanted to check in on what my options are here. I want to use the expression currently named 204Pb206PbOvercts_corr as the overcount corrected 204/206 in my common Pb corrections (the expression for its error is ERR204Pb206PbOvercts_corr but I can get this whole thing into a value model if/when the time comes). So my question is: is there a way to swap this in for 204/206 Orig? Can I use the built in SWAP expression some way? 997_1_2021_Oct_04_12.12.zip Thanks for any advice!

bowring commented 1 year ago

@NicoleRayner and @bowring met online and re-ignited this issue. Solution forthcoming this summer!

sbodorkos commented 1 year ago

Very glad to see this - when Nicole raised her issue a couple of weeks ago, I didn't respond because I couldn't even work out whether I had raised any similar issue previously, but I didn't want to confess! We at GA have a similar use-case for monazite analyses (I think our data-processing is very similar to Nicole's).

bowring commented 1 year ago

@sbodorkos - v2.0.1 has been released with this update - please take it for a ride.

Release: https://github.com/CIRDLES/Squid/releases/tag/v2.0.1

Discussion: https://github.com/CIRDLES/Squid/discussions/752

Jim

NicoleRayner commented 1 year ago

This all seems to work great! Thanks so much. Two little tweaks/observations. There still remains two SWAP expressions in built-in. I don't think those need to be there anymore. And also, when trying to make a report and include the OverCtCorr expression from Metadata and Common Pb Specs, it doesn't recognize the new dialog button. It should read SWAPCustomCorrection204 image

bowring commented 1 year ago

New version v2.0.2 solves the report issue. It also moves the expression SWAPCustomCorrection204 to the Built-in expressions. These are being preserved for transparency as they are the actual expressions used to perform the corrections. Thoughts?

NicoleRayner commented 1 year ago

If SWAPCustomCorrection204 is in Built-in I can't edit it. I assume the intention is to make a copy of this expression in Custom (with the exact same name and say yes to "do you want to replace") and then make edits to that version within custom. That seems to work. When I select the SWAPCustom button in the Common Pb window the ages change so it seems to be using the Custom version. And I note that it then disappears as an expression name from Built-In. I will add this note to the discussion page on this topic and probably also make a little video to add to the collection.

bowring commented 1 year ago

@NicoleRayner - that's the idea - this way the expression example always appears in a new project - there may be a better approach. I also noticed that the CommonPb frame does not refresh the SWAPCustomCorrection204 calculations - if they were already selected - unless forced to do so. Probably ought to be any issue?

NicoleRayner commented 1 year ago

UNCLASSIFIED - NON CLASSIFIÉ

Yes - I will make an issue for that thing about the Common Pb frame.

Also more diabolically if you alternate between the Pb-U and Th-U systems in the Interpretation - RM - calibration constant WM menu the SWAP disappears from custom and goes back into Built-In. There are also issues with "stickiness" of deselected RM analyses in the calibration constant WM menu if you alternate between the Pb-U and Th-U systems (the selections don't stick). Should that be the same issue or different?

It is a bit tricky that the custom SWAP expression doesn't track with a given task. That is what I would have assumed but I can see how it might not work well that way (how do you ever get to orig). One possible way around this would be to have the task contain "misnamed" swap expression with all the desired content such as "AAA_SWAPCustomCorrection204". This way it is at the top of the custom list and if you wish to activate it all you need to do is edit out the name to make it just "SWAPCustomCorrection204" rather than have to build it out/import the xml of the modified one each time.

Nicole

From: Jim Bowring @.> Sent: Wednesday, September 20, 2023 3:54 PM To: CIRDLES/Squid @.> Cc: Rayner, Nicole @.>; Mention @.> Subject: Re: [CIRDLES/Squid] Column-swapping in Tasks: a simple zircon-based example (#613)

Caution - email originated from outside of NRCan. Read the warning below / Attention- Ce courriel provient de l'extérieur des RNCan. Voir la mise en garde ci-dessous

@NicoleRaynerhttps://github.com/NicoleRayner - that's the idea - this way the expression example always appears in a new project - there may be a better approach. I also noticed that the CommonPb frame does not refresh the SWAPCustomCorrection204 calculations - if they were already selected - unless forced to do so. Probably ought to be any issue?

- Reply to this email directly, view it on GitHubhttps://github.com/CIRDLES/Squid/issues/613#issuecomment-1728340728, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJROHKSMOCNEYG3UR5AAUO3X3NCWZANCNFSM43VFDLFA. You are receiving this because you were mentioned.Message ID: @.**@.>> This email originated from outside of NRCan. Do not click links or open attachments unless you recognize the sender and believe the content is safe. For more information, please visit How to Identify Phishinghttps://intranet.nrcan.gc.ca/services-policies/dont-get-scammed-cyber-security-101 emails on the NRCan Intranet. Ce courriel provient de l'extérieur des RNCan. Ne cliquez pas sur les liens et n'ouvrez pas les pièces jointes, à moins de connaître l'expéditeur et croire que le contenu est sécuritaire. Pour de plus amples renseignements, veuillez consulter Comment identifier des courriels d'hameçonnageshttps://intranet.nrcan.gc.ca/services-policies/dont-get-scammed-cyber-security-101 dans l'intranet des RNCan.