Added BiScEF to projects.yml

[knutstanley]

After my presentation at ESWW, it was suggested that I add the repo to the list at heliopython. So, here it is. It's not really a software project (the core content is the format description), but contains some python scripts.

[sapols]

Thank you for your submission. I've had a slow start to the year but I will review this in the next week or so. Just wanted to let you know I saw it. 👍

[sapols]

Hi @knutstanley, I finally got around to reviewing this. I think it's always interesting when someone comes up with a new format, and it sounds like there were some positive sentiments from ESWW, but I do have questions about including this in PyHC.

Like you said, it's not really a software project. Thanks for being upfront about that. Am I correct that the only Python functionality provided is (1) plotting the files and (2) converting them to HDF5? I would, at the least, want that functionality more clearly documented before considering merging this.

I agree with your self-evaluation badges. Projects are allowed to join even if some things "require improvement," but when pretty much everything requires improvement like this, it's a special case and becomes my discretion. We typically only let it slide if there is already a non-trivial user base (if it's something people are actually using) and if there are immediate plans to further develop the repo. Can you explain if either of these are the case for you?

[sapols]

Oh and the merge conflict is my fault. I merged another PR earlier today that added a "HERMES-Core" project to this project.yml file. I could resolve that myself later.

[knutstanley]

Projects are allowed to join even if some things "require improvement," but when pretty much everything requires improvement like this, it's a special case and becomes my digression. We typically only let it slide if there is already a non-trivial user base (if it's something people are actually using) and if there are immediate plans to further develop the repo. Can you explain if either of these are the case for you?

The user base is currently very small. We intend to expand it, of course, but can not guarantee that the rest of the world will actually agree with us and start using it. The repo will be updated, but we do not currently have concrete plans for developments that would change the badge situation. This is mainly because the software part is a minor part, while the format description is the important part.

Am I correct that the only Python functionality provided is (1) plotting the files and (2) converting them to HDF5? I would, at the least, want that functionality more clearly documented before considering merging this.

We could expand on the documentation of this, if that alone would be enough to be on the list. But from the rest of your comment, I suspect a bit more is required.

No hard feelings if you decide not to include this. I made this suggestion for a list entry just because a person which clearly was more familiar than me with PyHC thought I should do it, and the effort required to do so was not very large. A bit of a "shot in the dark" to see what would happen.

[sapols]

Understood. Thanks again for being open. We talked about your project at a telecon this morning, and a few people wondered if you wouldn't mind pitching this format to PyHC a little? As in write just a few sentences here in this PR for how the BiScEF format would benefit the community? E.g. what heliophysics use cases does this format address? What types of people might use it? That kind of thing?

[knutstanley]

The BiScEF format is designed to hold data produced by "Scintillation receivers" (One of the most widely used currently is: https://www.septentrio.com/en/products/gps/gnss-reference-receivers/polarx5s). These receive signals from GNSS (Global navigation satellite system) satellites, but at a much higher sampling rate than normal GNSS receivers, with more robust signal tracking, and with options for output of types of data that is not usually an output from normal GNSS receivers.

Their main task is to measure the strength of phase fluctuations, amplitude fluctuations, and high-resolution phase data. This characterizes the signal scintillation, which is mainly caused by small-scale density fluctuations in the ionosphere. The density fluctuations are in turn caused by the interaction of the Earth atmosphere and magnetosphere with structures in the solar wind. The structures in the solar wind are originating from events at the Sun.

Data measured by these receivers are used both operationally for monitoring purposes (e.g.: http://chain.physics.unb.ca/chain/), and by the academic community to perform research on the ionosphere, the ionosphere-magnetosphere-solar wind connections, and on the impact of the disturbances on technical systems. I know that this kind of data is commonly used in the space physics community, but I do not know to which extent the heliophysics community use them. For these data to be of use you would need to be interested in Earths' ionosphere and/or impact on trans-ionospheric signals.

There has not really been a standard format for storing and sharing these data. An attempt was made with the SCINTEX format (https://www.icao.int/APAC/Documents/edocs/SCINtintillation.pdf), which was based on the RINEX format. SCINTEX has several major short-comings and is in my opinion not very suitable. (Illustrated by the fact that I felt compelled to define a new format myself) (Also, I have never seen any SCINTEX data files in any data archive and do not know of anyone using it.) Receiver manufacturer tools output results in their own very simple files, which are practical to use but not suitable for storage and sharing (for instance, they contain no metadata at all).

[smithara]

Could projects like this go under a new "miscellaneous" category (we already have packages: core, other, and unevaluated)? The standards are really designed around the project being a Python package which is why this doesn't really fit, but still could be within the PyHC sphere of influence.

There are a lot of other things that could fit here and could use some curation (e.g. web services, cookbooks). Actually managing that broader scope and being comprehensive quickly gets out of hand, and one approach I see that deals with that problem is "awesome lists", e.g. https://github.com/jonathansick/awesome-astronomy. I could imagine one of these like "Open Heliophysics" (here's one similar attempt already), under the heliophysicsPy org. The advantage is to have a low barrier for contribution and maintenance of the list, but still make things discoverable by putting them all on the same page.

[sapols]

Thanks for the comment @smithara. I agree this would belong in an "awesome list" if PyHC made one. But after careful consideration, I'm sorry but I'm going to close this PR without merging. BiScEF in its current state is just too unrelated to the rest of our PyHC packages. I appreciate the submission though, and could reconsider in the future if BiScEF catches on more and/or if the repo's Python tooling matures. Thank you.