fit_speck_removal cannot read from standard input

[ecbland]

BUG

fit_speck_removal is not able to read from standard input

Priority

• [X] Minor - typo, incorrect naming, fails on specific uncommon situation
• [ ] Major - misrepresentation of data, failure in a common situation
• [ ] Urgent - failure in usage, large misrepresentation

Example of the bug

make_fit -fitacf-version 3.0 20150301.0000.03.cly.rawacf | fit_speck_removal > cly.filter.fitacf3
Error reading file

Note: I do not recommend this workflow because I would encourage users to look at the fitted data straight out of FITACF3.0 before applying the filter. However, there may be an advantage for mass data processing in the future.

Attempts

What have you tried already to fix it?

I believe this bug occurs because fit_speck_removal has to read the fitacf file twice in order to perform the filtering. rewind(fp) is used to navigate back to the start of the file, but this does not seem to work when fp==stdin. I looked for clues in other routines that use rewind(fp). However, the only ones are trim_fit and trim_snd, and I couldn't even determine if the rewind(fp) line is necessary there (I got the same output file when I commented it out).

What have you tried to get around it?

Run make_fit and fit_speck_removal separately:

make_fit -fitacf-version 3.0 20150301.0000.03.cly.rawacf > cly.fitacf3
fit_speck_removal cly.fitacf3 > cly.filter.fitacf3

Data Location

Any rawacf file

Potential Bug Location

The error occurs when the fitacf data are read for the second time:

https://github.com/SuperDARN/rst/blob/97f26d65a45a83cc2f96c808682061911efa905b/codebase/superdarn/src.bin/tk/tool/fit_speck_removal.1.0/fit_speck_removal.c#L236-L242

Potential Solution(s)

The simplest solution is to remove the "read from stdin" functionality from this routine. This would encourage people to look at the fitted data before filtering it, and I think this is a good idea while the routine is new.

To properly fix this bug, we probably need a way to navigate backwards through a dmap file read from standard input. I have no idea if this is possible :smile:

[egthomas]

@ecbland I vaguely recall the rewind(fp) being necessary in some of the trimming functions so that the first record of a file could be correctly handled.

[mts299]

We can close this now? We made an error for it?