SuperDARN (not davitpy) processing tools

[aburrell]

In the interest of making more of our analysis code for SuperDARN accessable to the entire community, I wonder if there's a place for us to add c-code post-fitacf level analysis routines?

[asreimer]

@aburrell you may like to talk to our software engineer @kkotyk about this. He's currently working on getting the important bits of RST converted to python. In fact, he just finished getting dmap reading and writing working: https://github.com/SuperDARNCanada/pydmap

I'm working on getting pydmap implemented in davitpy as a replacement for the C code we currently have.

[ksterne]

Seems as this is an old question, but doing a bit of review of items here. @aburrell, in addition to the software USask is developing, are you looking for the RST repository at:

https://github.com/vtsuperdarn/VTRST3.5

While not as active in development or nearly as well documented, this contains the usual gridding and map potential source codes. Is this what you were thinking for post-fitacf level analysis routines?

[asreimer]

This is perhaps something we can talk about at AGU too.

[aburrell]

Thanks for the tip Kevin. I think RST is probably the right destination for what I was looking for in the end. However, I think there is a role for davitpy here, and that's something we can talk about at AGU.

On 26 Oct 2016, at 23:54, Ashton Reimer notifications@github.com wrote:

This is perhaps something we can talk about at AGU too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/216#issuecomment-256500918, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_hUaINjNALZjrbo4KIqYpGYWWshfks5q39owgaJpZM4HYDv-.

[Shirling-VT]

This is definitely something worthwhile discussing at AGU. I heard from several outside users suggesting that they just want the SuperDARN data processing tool instead of the whole davitpy in the past two GEM workshop.

On Fri, Oct 28, 2016 at 11:53 AM, Angeline Burrell <notifications@github.com

wrote:

Thanks for the tip Kevin. I think RST is probably the right destination for what I was looking for in the end. However, I think there is a role for davitpy here, and that's something we can talk about at AGU.

On 26 Oct 2016, at 23:54, Ashton Reimer notifications@github.com wrote:

This is perhaps something we can talk about at AGU too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ vtsuperdarn/davitpy/issues/216#issuecomment-256500918, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_ hUaINjNALZjrbo4KIqYpGYWWshfks5q39owgaJpZM4HYDv-.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/216#issuecomment-256957724, or mute the thread https://github.com/notifications/unsubscribe-auth/AIEZK78cOvYBqZ2RJjpDPATUU-sU8-UXks5q4hqBgaJpZM4HYDv- .

Best wishes, Xueling

[aburrell]

Here’s something that might be useful. How about everyone make a note of which davitpy modules they use and how often? Then we might have a better idea of what is actually the most useful and what is more peripheral. Xueling, if you remember what bits people at GEM said they liked, that could be added as well.

On 28 Oct 2016, at 17:17, Xueling Shi notifications@github.com wrote:

This is definitely something worthwhile discussing at AGU. I heard from several outside users suggesting that they just want the SuperDARN data processing tool instead of the whole davitpy in the past two GEM workshop.

On Fri, Oct 28, 2016 at 11:53 AM, Angeline Burrell <notifications@github.com

wrote:

Thanks for the tip Kevin. I think RST is probably the right destination for what I was looking for in the end. However, I think there is a role for davitpy here, and that's something we can talk about at AGU.

On 26 Oct 2016, at 23:54, Ashton Reimer notifications@github.com wrote:

This is perhaps something we can talk about at AGU too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ vtsuperdarn/davitpy/issues/216#issuecomment-256500918, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_ hUaINjNALZjrbo4KIqYpGYWWshfks5q39owgaJpZM4HYDv-.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/216#issuecomment-256957724, or mute the thread https://github.com/notifications/unsubscribe-auth/AIEZK78cOvYBqZ2RJjpDPATUU-sU8-UXks5q4hqBgaJpZM4HYDv- .

Best wishes, Xueling — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/216#issuecomment-256963371, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_uJovzBHu5OGrCfgRx3B_9Liawawks5q4h_9gaJpZM4HYDv-.

[MuhammadVT]

Hi all,

I had the same impression as Xueling that outside users are mostly interested in reading and plotting the SuperDARN data. For me I use pydarn module the most, specifically pydarn/sdio, pydarn/radar, pydarn/plotting, and pydarn/sdio. I use utils and gme/ind as well and started to use models/hwm.

Thanks, Muhammad.

On Fri, Oct 28, 2016 at 12:29 PM, Angeline Burrell <notifications@github.com

wrote:

Here’s something that might be useful. How about everyone make a note of which davitpy modules they use and how often? Then we might have a better idea of what is actually the most useful and what is more peripheral. Xueling, if you remember what bits people at GEM said they liked, that could be added as well.

On 28 Oct 2016, at 17:17, Xueling Shi notifications@github.com wrote:

This is definitely something worthwhile discussing at AGU. I heard from several outside users suggesting that they just want the SuperDARN data processing tool instead of the whole davitpy in the past two GEM workshop.

On Fri, Oct 28, 2016 at 11:53 AM, Angeline Burrell < notifications@github.com

wrote:

Thanks for the tip Kevin. I think RST is probably the right destination for what I was looking for in the end. However, I think there is a role for davitpy here, and that's something we can talk about at AGU.

On 26 Oct 2016, at 23:54, Ashton Reimer notifications@github.com wrote:

This is perhaps something we can talk about at AGU too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ vtsuperdarn/davitpy/issues/216#issuecomment-256500918, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_ hUaINjNALZjrbo4KIqYpGYWWshfks5q39owgaJpZM4HYDv-.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/ 216#issuecomment-256957724, or mute the thread https://github.com/notifications/unsubscribe-auth/ AIEZK78cOvYBqZ2RJjpDPATUU-sU8-UXks5q4hqBgaJpZM4HYDv- .

Best wishes, Xueling — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ vtsuperdarn/davitpy/issues/216#issuecomment-256963371, or mute the thread https://github.com/notifications/unsubscribe- auth/AGuC_uJovzBHu5OGrCfgRx3B_9Liawawks5q4h_9gaJpZM4HYDv-.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/216#issuecomment-256966148, or mute the thread https://github.com/notifications/unsubscribe-auth/AG1iW7p1ltQ2M8dnzxcDC2P-yortikBEks5q4iLPgaJpZM4HYDv- .

[asreimer]

This is an important discussion to be having. @aburrell has a great idea about taking note of which modules people use. This is what I envisioned when I first brought up the idea of separating davitpy into a core and an extra repository.

On a related note, check out backscatter, a data analysis software package that @kkotyk at SuperDARN Canada has been preparing.

Will we have lots to discuss at AGU!

[aburrell]

It's a bit of a shame to see code being duplicated already, but I can see the appeal of starting without any baggage.

To make our discussion about modules easier, I've created a survey to make this easier to quantify:

http://www.surveygizmo.com/s3/3149166/davitpy-modules

[asreimer]

Sorry, what code is being duplicated? Backscatter includes a python version of the dmap file reading and writing software that is currently only in RST. davitpy 'borrowed' the C dmap stuff from RST and wrapped it in python. Backscatter also includes a python version of make_fit, which is otherwise only available as a C version in RST. I don't think there is any code being duplicated here.

Good idea with the survey!

[aburrell]

To an outsider, it looked like dmap was being duplicated, as well as c code from RST. Whilst there is certainly value in having analysis kits in different languages (go, davitpy, rst), I worry when development drifts away from community input. That’s one of the reasons I decided to use davitpy instead of starting from scratch, and it appears to have happened aeons ago with RST. If ‘backscatter’ improves general community use of SuperDARN, though, I’m all for it.

I appear to have lost the email address for the davitpy google list. I’d like to send the survey link out there. Does anyone have it on hand?

On 31 Oct 2016, at 21:44, Ashton Reimer notifications@github.com wrote:

Sorry, what code is being duplicated? Backscatter includes a python version of the dmap file reading and writing software that is currently only in RST. davitpy 'borrowed' the C dmap stuff from RST and wrapped it in python. Backscatter also includes a python version of make_fit, which is otherwise only available as a C version in RST. I don't think there is any code being duplicated here.

Good idea with the survey!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/216#issuecomment-257429817, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_mxyliY6iYItv4XSwOm3chKo4uM0ks5q5mFKgaJpZM4HYDv-.

[Shirling-VT]

I have it and already did the survey.

On Monday, October 31, 2016, Angeline Burrell notifications@github.com wrote:

To an outsider, it looked like dmap was being duplicated, as well as c code from RST. Whilst there is certainly value in having analysis kits in different languages (go, davitpy, rst), I worry when development drifts away from community input. That’s one of the reasons I decided to use davitpy instead of starting from scratch, and it appears to have happened aeons ago with RST. If ‘backscatter’ improves general community use of SuperDARN, though, I’m all for it.

I appear to have lost the email address for the davitpy google list. I’d like to send the survey link out there. Does anyone have it on hand?

On 31 Oct 2016, at 21:44, Ashton Reimer <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Sorry, what code is being duplicated? Backscatter includes a python version of the dmap file reading and writing software that is currently only in RST. davitpy 'borrowed' the C dmap stuff from RST and wrapped it in python. Backscatter also includes a python version of make_fit, which is otherwise only available as a C version in RST. I don't think there is any code being duplicated here.

Good idea with the survey!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ vtsuperdarn/davitpy/issues/216#issuecomment-257429817, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_ mxyliY6iYItv4XSwOm3chKo4uM0ks5q5mFKgaJpZM4HYDv-.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/216#issuecomment-257436793, or mute the thread https://github.com/notifications/unsubscribe-auth/AIEZK1WhFoYTsAY3KlqgeColpgG_C1dWks5q5miTgaJpZM4HYDv- .

Best wishes, Xueling

[MuhammadVT]

Survey is done.

On Oct 31, 2016 9:51 PM, "Xueling Shi" notifications@github.com wrote:

I have it and already did the survey.

On Monday, October 31, 2016, Angeline Burrell notifications@github.com wrote:

To an outsider, it looked like dmap was being duplicated, as well as c code from RST. Whilst there is certainly value in having analysis kits in different languages (go, davitpy, rst), I worry when development drifts away from community input. That’s one of the reasons I decided to use davitpy instead of starting from scratch, and it appears to have happened aeons ago with RST. If ‘backscatter’ improves general community use of SuperDARN, though, I’m all for it.

I appear to have lost the email address for the davitpy google list. I’d like to send the survey link out there. Does anyone have it on hand?

On 31 Oct 2016, at 21:44, Ashton Reimer <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Sorry, what code is being duplicated? Backscatter includes a python version of the dmap file reading and writing software that is currently only in RST. davitpy 'borrowed' the C dmap stuff from RST and wrapped it in python. Backscatter also includes a python version of make_fit, which is otherwise only available as a C version in RST. I don't think there is any code being duplicated here.

Good idea with the survey!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ vtsuperdarn/davitpy/issues/216#issuecomment-257429817, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_ mxyliY6iYItv4XSwOm3chKo4uM0ks5q5mFKgaJpZM4HYDv-.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/ 216#issuecomment-257436793, or mute the thread https://github.com/notifications/unsubscribe-auth/ AIEZK1WhFoYTsAY3KlqgeColpgG_C1dWks5q5miTgaJpZM4HYDv- .

Best wishes, Xueling

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/216#issuecomment-257471041, or mute the thread https://github.com/notifications/unsubscribe-auth/AG1iW_pNddSMXukNy2BGm9qlSNLIsvpAks5q5psjgaJpZM4HYDv- .

[asreimer]

I suppose that the code in backscatter is a duplication of some RST code, in so much as it is a port to another language. The code is written entirely in python and documented such that an outsider actually has hope of understanding what the code is doing on a first read through (unlike RST). For example, does anyone here understand how the python wrapped dmapio library in davitpy works (@ksterne probably does)? A "fun" way to spend your weekend is trying to figure out how make_fit in RST works :P

Oooo also, backscatter already has a hdw.dat file parser for python (so it will be much easier for us to do something like that in davitpy in addressing issue #278.

I have also filled out the survey. The google groups for davitpy are: https://groups.google.com/forum/#!forum/davitpy and https://groups.google.com/forum/#!forum/davitpy-users

[aburrell]

Right, this is why I didn’t have the info. My google email wasn’t invited to join the group(s). Fun!

On 1 Nov 2016, at 05:21, Ashton Reimer notifications@github.com wrote:

I suppose that the code in backscatter is a duplication of some RST code, in so much as it is a port to another language. The code is written entirely in python and documented such that an outsider actually has hope of understanding what the code is doing on a first read through (unlike RST). For example, does anyone here understand how the python wrapped dmapio library in davitpy works (@ksterne https://github.com/ksterne probably does)? A "fun" way to spend your weekend is trying to figure out how make_fit in RST works :P

Oooo also, backscatter already has a hdw.dat file parser for python (so it will be much easier for us to do something like that in davitpy in addressing issue #278 https://github.com/vtsuperdarn/davitpy/issues/278.

I have also filled out the survey. The google groups for davitpy are: https://groups.google.com/forum/#!forum/davitpy https://groups.google.com/forum/#!forum/davitpy and https://groups.google.com/forum/#!forum/davitpy-users https://groups.google.com/forum/#!forum/davitpy-users — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/216#issuecomment-257491083, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_hQugnvOjgVNw_qUoLdfekRaftjXks5q5sxCgaJpZM4HYDv-.