Longyearbyen hdw.dat file

[egthomas]

@ecbland raised the issue during the working group meeting last week that the hdw.dat file for the Longyearbyen radar in this repository does not match the hdw.dat file in the vtsuperdarn hdw.dat github repository (https://github.com/vtsuperdarn/hdw.dat/blob/master/hdw.dat.lyr). After comparing the two hdw.dat directories, the hdw.dat.lyr file is actually completely missing from this RST repository (and the VTRST3.5 repository we branched from).

So my question is this - do we add the hdw.dat.lyr file directly to the master branch as a "hotfix"? Unfortunately the 4.0 tag is already set so users who go to the download link we sent over darn-users won't get the hdw.dat.lyr file with their copy of the RST.

[ecbland]

This issue is a little broader than just the missing hdw.dat.lyr file -- there are actually several other radars for which the hardware file in RST4.0 is not the same as the version in the hdw.dat repository. E.g., CLY, INV, PGR, RKN & SAS.

[egthomas]

Ah, that's worse than I realized. It also brings us back to issue #3 raised by @ksterne.

Personally, I think that the definitive set of hdw.dat files should reside within the RST. Our core processing / data analysis software for producing fit, grid, and map files does not work without those hdw.dat files (and introduces errors when those files are incorrect). One might say, hang on, there are users of Go or Davit or Davitpy or something else who don't care about the RST and just want to plot the data. Well, there won't be any SuperDARN data files for them to read or plot if we can't produce them using the RST in the first place.

Why not have users submit changes in hdw.dat files to us so we can apply them directly to the master branch (or they can submit a pull request themselves)? That way if someone was git-savvy enough to run scripts to keep their tables/hdw directory up to date they can still pull changes from the master branch as necessary. Also, it will allow for better record keeping of changes to hdw.dat files than commit messages like "syncing to git repo" or "Updating from automated script on 20XXXXX".

If there is still a desire for a separate repository, it should sync from the tables/superdarn/hdw directory of the RST and not the other way around.

[kkotyk]

Yes we will need to address this soon. Are there going to be any issues that could arise from circumventing the release model for just hdw.dat files?

[pasha-ponomarenko]

Well, I don't think that the stand-alone repository for the hdw files worked anyways: PIs are too busy to take care of such minor things. That's why Mark suggested introduction of the deadline/last warning system for approving changes in the hardware records. I see nothing wrong with what Evan proposed if it is technically viable.

[egthomas]

@kkotyk - if a hardware file changes before we're ready for a full release of the RST, why not just send an email over darn-users with a link to the correct hdw.dat file for users to download? I think such updates to hdw.dat files would fall under the "hotfix" character on @ksterne 's diagram and not break the release model.

[sshepherd]

Agreed.

On Thu, Jun 15, 2017 at 1:26 PM, Evan Thomas notifications@github.com wrote:

@kkotyk https://github.com/kkotyk - if a hardware file changes before we're ready for a full release of the RST, why not just send an email over darn-users with a link to the correct hdw.dat file for users to download? I think such updates to hdw.dat files would fall under the "hotfix" character on @ksterne https://github.com/ksterne 's diagram and not break the release model.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/SuperDARN/rst/issues/70#issuecomment-308812494, or mute the thread https://github.com/notifications/unsubscribe-auth/AC2DP71nDG72e7Mu9rdi_e9MdB1DIQLWks5sEWlWgaJpZM4N6T_K .

[egthomas]

So we now have the Longyearbyen and updated Canadian hdw.dat files on the master branch. What's the preferred method for getting these same changes onto the develop branch? Do we merge master into develop? Do we merge the hdw_update branch into develop?

[ksterne]

First one sounds good. Develop should get updates from master.

---

....thumbed from my phone; auto-correct may have improved this message...

----- Reply message ----- From: "Evan Thomas" notifications@github.com To: "SuperDARN/rst" rst@noreply.github.com Cc: "Kevin Sterne" kevintyler@vt.edu, "Mention" mention@noreply.github.com Subject: [SuperDARN/rst] Longyearbyen hdw.dat file (#70) Date: Sat, Jun 17, 2017 8:28 AM

So we now have the Longyearbyen and updated Canadian hdw.dat files on the master branch. What's the preferred method for getting these same changes onto the develop branch? Do we merge master into develop? Do we merge the hdw_update branch into develop?

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[ksterne]

Hey all, sorry I'm a little late to the party and mostly out of the conversation. I'll try to get back more into it next week when I'm off of vacation.

Was @kevinkrieger and the DDWG in on the conversation here? At one point in time the DDWG had taken on ownership of a definitive set of hdw.dat files and distributing them out. It's referenced on the working group page (http://vt.superdarn.org/tiki-index.php?page=Data+Dist+Working+Group) under the 'Supplement Material' That's how we got the git repo that was noted in the opening comment, this was being updated/referenced by about half of the institutions at last check (it's been a year or so since it's been checked).

I would say something to @egthomas' comment on 'hang on here, what about other software packages that just want the hdw.dat files?' It's conceivable (and I think it's happening right now) that not all institutions aren't using RST for their processing (I know gasp). Maybe it's something I missed, but there's a high-level question here of hdw.dat files being independent of the RST code. And as well, a clear assignment of which working group within SuperDARN will maintain them. I think so far, adding the DAWG brings it up to 3 groups that are trying to maintain the definitive set of files. I could argue adding the DAWG as a candidate is adding confusion as there used to be 2 (which the OSWG is all but a closet of moths at this point). The DDWG had something going (which is still in use at VT) but the problem is not enough changes to make it appear as active.

I don't have strong feelings here, but there's a lot of unanswered questions from the outside (of the workshop) looking in that do not appear to be clear. Maybe others here know more than I do, to which I'll go back to other things.

[egthomas]

I would say something to @egthomas' comment on 'hang on here, what about other software packages that just want the hdw.dat files?' It's conceivable (and I think it's happening right now) that not all institutions aren't using RST for their processing (I know gasp).

I'm a little confused here - what are the current alternatives to producing higher level data products from our rawacf files? Pasha and Keith's FITACF3 requires the RST and has already been integrated on a pull request branch. Ashton has a python-only version of his lmfit2 technique, but if the C version takes ~10 minutes to process 1 hour of raw data (as stated in the readme) then it doesn't seem to me the python code would be a viable alternative for mass-processing data (particularly if the native python reading is slower than the C dmap library). I know Angeline has been doing a lot of good work producing high-quality grid-level data for potential mapping but those still come from fitacf files.

[pasha-ponomarenko]

Ok, guys! I can see here at least two issues to address: (1) demarkation of WGs' responsibilities on manitaining/modifying hardware information; (2) functioning of other's software utilising hardware information

On mantaining the hdw.dat files, I have no particular preference or opinion except the people responsible for this should perform it accurately and in a timely manner. The location where the files are maintained is not that important as long as everything works. On the other hand, I am not convinced by the reasoning behind creating an extra step in modifying these files somewhere outside of RST and then synchronising the respective RST copies. If this is only for the sake of taking care of the someone else's packages, I tend to side with Evan and should add that their sofware lies outside of the DA-WG's (as well as any other WG's) area of respoisibilities and control. While in no terms I want to look hostile or unfriendly to the people involved, our charitablity should not have any ill effects on the functionality and convinience of maintaing of our own software.

By the way, is anyone going to update the software page at the VT website? It still points to VTRST3.5 as the best package available...

[egthomas]

Just found 3 more hdw.dat files that did not agree with the VT hdw.dat repository (#78). While these won't affect the generation of fitacf files from rawacf using make_fit, they will have an impact on geolocation when mapping FOVs or gridding the data.

[ksterne]

Hey @pasha-ponomarenko, that seems like a good breakdown of things here. I can see your point on the extra step. That's what led to @egthomas finding #78, #79 as I hadn't kept a good update on both sets. Though my apologies here, I'd been working with a focus on the stand-alone hdw.dat repo since before a few months ago this codebase hadn't seen much attention. As is the trouble with hdw.dat files, updates are too infrequent and it's hard to keep them sync'ed up. Though I understand a stand-alone repo seems to be outside of the DAWG if we're equating RST to the only software this WG is working on, which is currently true.

Is there much more to coordinating or figuring out responsibilities pertaining to @pasha-ponomarenko's point (1)? There's lots of ways to argue it between the OSWG, DAWG, and DDWG. There seems to be a lack of cross-WG coordination here to get some kind of discussion/determination on this.

Also, btw, I'm guilty of sitting on a several month old change to the hdw.dat.bks file. I need to find the time to ensure the accuracy of the change before releasing it.

[egthomas]

It still does not sound like we've reached a consensus (or executive decision) on this issue yet. I stand by my comment on this issue from 2 weeks ago (https://github.com/SuperDARN/rst/issues/70#issuecomment-308718997) that the definitive set of hdw.dat files needs to reside within the RST. If we want to keep an external directory of hdw files for other software packages to link to, that directory should synchronize from the RST and not the other way around. I don't have any preference as far as which working group (or task force) maintains our hdw.dat files as long as they do so in as timely a fashion as possible.

[pasha-ponomarenko]

I tend to side with Evan on that the hdw.dat files should be maintained within the RST repository and the external copies should be synchronised with it. However, the decision on who should be responsible for this has to be made at a higher level, keeping in mind the multitude of references to different WGs. I would like to hear what Mike and Mark think about it, although I suspect that it will be dumped on us anyways... ;-) @egthomas, from my conversation with Mark, I understood that any task force has a limited lifespan defined by their particular task.

[egthomas]

Well my original question about whether to apply hdw.dat fixes directly to the master branch has been answered - I'm going to close this issue and suggest the hdw.dat discussion continue in the original issue raised by @ksterne (#3).

[pasha-ponomarenko]

I looks like this has become one of the issues to be addressed by the tdiff task force. I just received a message from Mark in which one of the Terms of reference is

(v) to assess the current hardware.dat files and produce recommendations on how best to proceed in the future on maintaining these files, should they be needed;