We just finished fixing our detrending scripts. The attachments show both
the correction function from polyfit and the final detrended data. We ended
up using a window size for detrending of 100 with a polynomial order of 7.
Let us know what you think!
Thanks!
Nancy, Chris and Tiana
You can see the issue I mentioned last week in that
the polynomial fits have discontinuities
which do not look physical. I think it
is probably too much to go back to the
technique I proposed of fitting a polynomial
to a window around every single data point.
However, a hybrid that might work would be to
fit, say, every 50th data point ('anchor') with a 100 point wide
window. Then, every data point will be fit to
two polynomials; for example the 75th data
point would be fit with the polynomials centered
on the 50th & the 100th data points. Then,
we can simply linearly interpolate between these
based on the proximity to each anchor point.
So, for the 75th point we would take 50% of the
polynomial centered at the 100th anchor point
and 50% of the polynomial at the 50th data point
(i.e. average the two). While at the 90th data point
we would take 20% of the polynomial fit centered
on the 50th data point & 80% of that centered
on the 100th data point. This should give a correction
function that will vary much more smoothly & should
not have sharp discontinuities. It also has the
advantage that it uses the code that you already
computed, and will only take twice as long to compute.
Does this make sense?
-Eric
Yes. Basically you want windows to overlap, and the detrending function to be a "weighted" average of different polynomial fits (with the weight determined by how far it is from the 'anchor' point).
In this case the first 50 and last 50 points will be fit by only 1 function. Is this okay? or could we add a 50 point window starting at the first and final points to try an additional polynomial fit at this region?
--Praveen
In this case the first 50 and last 50 points will be fit by only 1 function. Is this okay?
Yes, this is not ideal, but it is okay (this will be true at any of the gaps).
or could we add a 50 point window starting at the first and final points to try an additional polynomial fit at this region?
Yes, we need to do that for the first & last data in every
'portion'.
pkundurthy
commented
12 years ago
Hello!
We just finished fixing our detrending scripts. The attachments show both the correction function from polyfit and the final detrended data. We ended up using a window size for detrending of 100 with a polynomial order of 7. Let us know what you think!
Thanks! Nancy, Chris and Tiana
You can see the issue I mentioned last week in that the polynomial fits have discontinuities which do not look physical. I think it is probably too much to go back to the technique I proposed of fitting a polynomial to a window around every single data point.
However, a hybrid that might work would be to fit, say, every 50th data point ('anchor') with a 100 point wide window. Then, every data point will be fit to two polynomials; for example the 75th data point would be fit with the polynomials centered on the 50th & the 100th data points. Then, we can simply linearly interpolate between these based on the proximity to each anchor point.
So, for the 75th point we would take 50% of the polynomial centered at the 100th anchor point and 50% of the polynomial at the 50th data point (i.e. average the two). While at the 90th data point we would take 20% of the polynomial fit centered on the 50th data point & 80% of that centered on the 100th data point. This should give a correction function that will vary much more smoothly & should not have sharp discontinuities. It also has the advantage that it uses the code that you already computed, and will only take twice as long to compute.
Does this make sense? -Eric
Yes. Basically you want windows to overlap, and the detrending function to be a "weighted" average of different polynomial fits (with the weight determined by how far it is from the 'anchor' point).
In this case the first 50 and last 50 points will be fit by only 1 function. Is this okay? or could we add a 50 point window starting at the first and final points to try an additional polynomial fit at this region?
--Praveen
Yes, this is not ideal, but it is okay (this will be true at any of the gaps).
Yes, we need to do that for the first & last data in every 'portion'.
-Eric