search

SuperDARN / rst

Radar Software Toolkit (RST)
https://superdarn.github.io/rst/
GNU General Public License v3.0
22 stars 18 forks source link

Release RST 5.0 #533

Closed ecbland closed 1 year ago

ecbland commented 1 year ago

Release RST 5.0

As announced in #529, the next version of the RST will be a major release. Key changes are:

Deadline: 30 November 2022

Please comment in the Pull Request conversation if you need additional time to test the release.

Testing

Do any binaries/libraries require special attention during testing?

We have made major changes to the following binaries (please add the ones I have missed!)

The IDL code for reading fitacf files has also been changed, and there are many other changes related to the tdiff updates.

Checklist

Extra Notes

A description of the release will appear soon.

Approving the release

After testing, please state which operating system you used and what you tested.

If you found no bugs during testing, please approve the PR.

pasha-ponomarenko commented 1 year ago

Hi Emma, The optional spectral width threshold for HMB determination (#509) can be added to the new features' list. 😊

Cheers, Pasha

From: Emma Bland @.> Sent: Friday, October 28, 2022, 7:21 a.m. To: SuperDARN/rst @.> Cc: Subscribed @.***> Subject: [SuperDARN/rst] Release RST 5.0 (PR #533)

CAUTION: External to USask. Verify sender and use caution with links and attachments. Forward suspicious emails to @.***

Release RST 5.0

As announced in #529https://github.com/SuperDARN/rst/issues/529, the next version of the RST will be a major release. Key changes are:

Deadline: 30 November 2022

Please comment in the Pull Request conversation if you need additional time to test the release.

Testing

Do any binaries/libraries require special attention during testing?

We have made major changes to the following binaries (please add the ones I have missed!)

The IDL code for reading fitacf files has also been changed, and there are many other changes related to the tdiff updates.

Checklist

Extra Notes

A description of the release will appear soon.

Approving the release

After testing, please state which operating system you used and what you tested.

If you found no bugs during testing, please approve the PR.

You can view, comment on, or merge this pull request online at:

https://github.com/SuperDARN/rst/pull/533

Commit Summary

File Changes

(168 fileshttps://github.com/SuperDARN/rst/pull/533/files)

Patch Links:

— Reply to this email directly, view it on GitHubhttps://github.com/SuperDARN/rst/pull/533, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABIRTSDDFNLLQMXDPWKMOIDWFPHOBANCNFSM6AAAAAARRCE2LA. You are receiving this because you are subscribed to this thread.Message ID: @.***>

ecbland commented 1 year ago

I've just updated the make_fit tutorial on readthedocs to include some basic information about elevation angle calibration with TDIFF, and updated the url to the SuperDARN noise paper (which was accepted earlier this year)

You can preview the full readthedocs documentation for RST 5.0 here.

agrocott commented 1 year ago

I have successfully* installed on macOS Catalina 10.15.7. I had to make a minor change to the makefile in /rst-release-5.0/codebase/superdarn/src.bin/tk/tcpip/fitacfclientgui.1.0/makefile – there was a “-ltinfo” flag that I think might refer to a library that does not exist for macos. I deleted the flag and the install proceeded, but I have not checked what consequences this might have for “fitacfclientgui”.

pasha-ponomarenko commented 1 year ago

@ecbland, here are some suggestions for readthedocs on make_fit just to avoid potential confusion:

egthomas commented 1 year ago

I have successfully* installed on macOS Catalina 10.15.7. I had to make a minor change to the makefile in /rst-release-5.0/codebase/superdarn/src.bin/tk/tcpip/fitacfclientgui.1.0/makefile – there was a “-ltinfo” flag that I think might refer to a library that does not exist for macos. I deleted the flag and the install proceeded, but I have not checked what consequences this might have for “fitacfclientgui”.

  • I have tested make_fit (fitacf3) on a rawacf file. The file size is significantly reduced, but from a quick glance I can’t see anything different in the data. make_grid subsequently returns identical files whichever version was used.

@agrocott thanks for testing this, and apologies for the new fitacfclientgui breaking your and @mtwalach's code (and probably anyone else on macOS). I'm not sure how to handle this - it won't compile properly on Linux distributions without the -ltinfo flag, and I don't have access to a Mac for testing.

ecbland commented 1 year ago

Below is the proposed author list. We've identified @egthomas, @asreimer and @ecbland as the major contributors to this release, and all other contributors are listed alphabetically. Anyone else who helps with testing will be added as well. Feedback on the proposed author list is welcome, either in the comments or by email to the DAWG chairs.

SuperDARN Data Analysis Working Group, Thomas, E.G., Reimer, A.S., Bland, E.C., Burrell, A.G., Grocott, A., Ponomarenko, P.V., Schmidt, M.T., Shepherd, S.G., Sterne, K.T., and Walach, M.-T.

egthomas commented 1 year ago

@ecbland two more thoughts:

agrocott commented 1 year ago

@agrocott thanks for testing this, and apologies for the new fitacfclientgui breaking your and @mtwalach's code (and probably anyone else on macOS). I'm not sure how to handle this - it won't compile properly on Linux distributions without the -ltinfo flag, and I don't have access to a Mac for testing.

I can test it if you can point me to a suitable fitacf data stream? If it works on a Mac without the "-ltinfo" flag then it's a simple workaround at least to have macOS users modify the makefile.

ecbland commented 1 year ago

@agrocott thanks for testing this, and apologies for the new fitacfclientgui breaking your and @mtwalach's code (and probably anyone else on macOS). I'm not sure how to handle this - it won't compile properly on Linux distributions without the -ltinfo flag, and I don't have access to a Mac for testing.

I can test it if you can point me to a suitable fitacf data stream? If it works on a Mac without the "-ltinfo" flag then it's a simple workaround at least to have macOS users modify the makefile.

I think we need to find a solution that allows macOSX users to compile RST without having to modify any of the makefiles. libtinfo seems to have some connection with ncurses, so I wonder if there's a dependency that needs to be updated/changed. Please speak up if you know anything about this stuff!

ecbland commented 1 year ago

@agrocott You can make your own fitacf data stream with RST:

fitacfserver -r 20150308.1001.00.sps.a.fitacf3

Fri Nov  4 11:33:33 2022:Child Server Process Starting
Fri Nov  4 11:33:33 2022:Listening on port 45687.
...

Take note of the port number. Then open a new terminal and do the following, replacing 45687 with the port number you got. 

fitacfclientgui -color localhost 45687
egthomas commented 1 year ago

@agrocott You can make your own fitacf data stream with RST:

fitacfserver -r 20150308.1001.00.sps.a.fitacf3

Fri Nov  4 11:33:33 2022:Child Server Process Starting
Fri Nov  4 11:33:33 2022:Listening on port 45687.
...

Take note of the port number. Then open a new terminal and do the following, replacing 45687 with the port number you got.

fitacfclientgui -color localhost 45687

@ecbland thanks for providing an example - I believe one can also force fitacfserver to use a specific port via the -lp option (rather than looking for the one assigned by your machine), eg

fitacfserver -r -lp 1024 20150308.1001.00.sps.a.fitacf3
agrocott commented 1 year ago

@agrocott You can make your own fitacf data stream with RST:

fitacfserver -r 20150308.1001.00.sps.a.fitacf3

Fri Nov  4 11:33:33 2022:Child Server Process Starting
Fri Nov  4 11:33:33 2022:Listening on port 45687.
...

Take note of the port number. Then open a new terminal and do the following, replacing 45687 with the port number you got.

fitacfclientgui -color localhost 45687

OK, so that worked (neat!). So, clearly on macOS the "-ltinfo" flag is not required.

Excuse my ignorance of how makefiles are generated, but does the script that produces the makefile not have access to the OSTYPE system variable, such that the makefile could be modified based on that?

agrocott commented 1 year ago

Whilst testing @ecbland's fix for fitacfclientgui on macOS #536 I encountered another issue with the install. I think it's just a macOS paths issue that may be dependent on the OS version: "directory not found for option '-L/opt/local/lib'" and "directory not found for option '-L/opt/local/include'"

I had a directory "/opt/X11" in place of "/opt/local". This issue didn't occur yesterday because I was using a different Mac (on which I had already symlinked "/opt/X11" to "/opt/local", I presume to fix the same issue on a previous RST install). On the Mac I'm using today I hadn't done an install for some time, so I guess something has changed.

Anyway, it wasn't a big deal to fix my paths, but perhaps RST ought to have been looking in a different place? I may have missed it, but I can't see a path to "/opt/local" in any of the .profiles where the other paths are set by the user. Is this one hardwired?

I also forgot to mention that I get quite a lot of non-fatal warnings - should we be reporting these too?

ecbland commented 1 year ago

Whilst testing @ecbland's fix for fitacfclientgui on macOS #536 I encountered another issue with the install. I think it's just a macOS paths issue that may be dependent on the OS version: "directory not found for option '-L/opt/local/lib'" and "directory not found for option '-L/opt/local/include'"

I had a directory "/opt/X11" in place of "/opt/local". This issue didn't occur yesterday because I was using a different Mac (on which I had already symlinked "/opt/X11" to "/opt/local", I presume to fix the same issue on a previous RST install). On the Mac I'm using today I hadn't done an install for some time, so I guess something has changed.

Anyway, it wasn't a big deal to fix my paths, but perhaps RST ought to have been looking in a different place? I may have missed it, but I can't see a path to "/opt/local" in any of the .profiles where the other paths are set by the user. Is this one hardwired?

There is a hardcoded path here that looks suspicious:

https://github.com/SuperDARN/rst/blob/75b871ed258c4340751b6936e2d6a880355d1cac/build/make/makebin.darwin#L10

@mtwalach Have you also created a symlink as @agrocott described?

I also forgot to mention that I get quite a lot of non-fatal warnings - should we be reporting these too?

There are indeed quite a few warnings when compiling RST (I get 194). I don't think there's any need to report them unless you're worried about a particular warning causing problems. We did a cleanup a few years ago (#337), and we found that some of them are non-trivial to fix. We were also asked not to fix the warnings in the AACGM_v2 library. If anyone would like to try removing some more warnings then go for it!

pasha-ponomarenko commented 1 year ago

@ecbland , I performed a very basic test:

Please let me know is you can think of some other important tests I might be capable to perform. :-)

ecbland commented 1 year ago

Thanks for testing, @pasha-ponomarenko! :100: That's a helpful set of tests on the fitacf data. If you (or others) have time, the following would be helpful:

I've checked compilation on Linux Mint 20, so with @pasha-ponomarenko and @agrocott 's testing efforts, we have covered all three OS types listed in the description. Bugs have previously been reported when compiling on Fedora (#459 #460), but that should have been fixed with #462 in RST4.7

pasha-ponomarenko commented 1 year ago

@ecbland, I run a following test for the IDL routines for reading new FITACF fields for CLY 16UT on 3 Feb 2021 (first record):

  fitfp=fitOpen(fname,/read)

  record=fitRead(fitfp,prm,fit)

  print, 'tdiff: ',fit.tdiff
  print, 'algorithm: ',fit.algorithm

  l_ef=where(fit.elv_fitted[0:70] gt 0, count_ef)
  l_ee=where(fit.elv_error[0:70] gt 0, count_ee)
  print, 'Non-zero fit.elv_fitted records: '+string(count_ef)
  print, 'Non-zero fit.elv_error records: '+string(count_ee) 
  close, fitfp

The outputs are following:

tdiff:       0.00000
algorithm: fitacf2
Non-zero fit.elv_fitted records:            0
Non-zero fit.elv_error records:            0

tdiff:       0.00000
algorithm: fitacf3
Non-zero fit.elv_fitted records:           36
Non-zero fit.elv_error records:           36
tdiff:     0.0100000
algorithm: fitacf3
Non-zero fit.elv_fitted records:           36
Non-zero fit.elv_error records:           36

tdiff: 0.00000 algorithm: Non-zero fit.elv_fitted records: 0 Non-zero fit.elv_error records: 0

agrocott commented 1 year ago

There is a hardcoded path here that looks suspicious:

https://github.com/SuperDARN/rst/blob/75b871ed258c4340751b6936e2d6a880355d1cac/build/make/makebin.darwin#L10

@mtwalach Have you also created a symlink as @agrocott described?

Could this path be changed to use a user defined one in the RST .profile?

ecbland commented 1 year ago

Could this path be changed to use a user defined one in the RST .profile?

@agrocott Probably. You and @mtwalach are the only mac users in this group, so I'll need to leave it to you to find a solution that works on your system.

agrocott commented 1 year ago

Could this path be changed to use a user defined one in the RST .profile?

@agrocott Probably. You and @mtwalach are the only mac users in this group, so I'll need to leave it to you to find a solution that works on your system.

Maria and I are going to look at it on Thursday 17th.

agrocott commented 1 year ago

@mtwalach and I have created a branch with a fix for the macOS installation issue. It seems that /opt/local has been replaced in the latest macOS with /user/local. It also seems that the only use of the existing /opt/local directory is to access the X libraries, so we have just replaced the hardcoded "/opt/local" with $(XPATH). This seems to work fine on my system. I notice that no equivalent path is even specified in the linux makefile, so it might be superfluous.

ecbland commented 1 year ago

Hi All

Thanks for all your efforts with testing the major release, and for adding some nice last-minute new features as well! :100: We're almost ready to go (just waiting for someone to merge #545).

Below are my proposed release notes. Any help to fill in gaps or clarify the language would be highly appreciated!

Key updates in version 5.0 of the Radar Software Toolkit (RST) include:

egthomas commented 1 year ago

@ecbland That looks good to me, although you don't need to include my name within the list of individual features

ecbland commented 1 year ago

Hi All

I think we're almost there with RST5.0 💯 If I have time on Monday I will test #554 (otherwise it can wait until the next release as @egthomas suggested). If there are no further issues then we can publish the release on Tuesday!

Cheers, Emma