Continuous integration tests

[bennahugo]

@twillis449 can you give me a few examples of ALBUS scripts for say MeerKAT and VLA at a given date/time range and coordinate (say 3C286). I will spend some time in the morning to set up continuous integration testing before I attempt any changes to the code base

[bennahugo]

also if there are any tests (there seems to be...) can you point them out to me so that we can run as many of them as possible

[twillis449]

I will provide some examples input scripts) and resulting output, later today.

[twillis449]

A few issues... I assume you compiled the programs crx2rnx and rnx2crx? you need them for un-compressing compressed rinex file (the compressed files end in 'd').

Also, you need to get the binary executable gfzrnx (see gnss.gfz-potsdam.de) Unfortunately they only provide binary executables and you have to sign up for an account. The current binary is gfzrnx_2.0-8219_lx64 and I just make a soft link gfzrnx to the binary. We need this so that if a site only provides data in RINEX3 format we can convert it into RINEX 2 format for analysis by ALBUS (which does not currently directly understand RINEX3). I already had to develop a conversion routine as Geosciences Australia converted to RINEX3 about a year or more ago.

You actually need this for Meerkat as the SUTM South Africa GPS receiver situated at the SALT site seems to have converted to use RINEX 3 format. Trignet claims that there are three other GPS receivers at the SALT site but none of them seem to provide data to the actual network for download. The python script Albus_rnx3_to_rnx2.py in ~/albus/share/python does the conversion. If you google for 'gfzrnx user manual' you will be drected to a download page which will nominally help you to understand the incantation in Albus_rnx3_to_rnx2.py

SUTM is an important GPS site as it is one of the ~ 500 sites that comprise the world GNSS system.

Cheers

[twillis449]

Here is a tar file with a script to generate ionosphere files over the VLA, MeerKat, and the SUTM GPS station on ionosphere_examples.tar.gz one of those days in August that you had requested. There are also corresponding output files and plots of STEC as a function of time. Its pretty obvious how to adjust the script to select one of the locations.

[bennahugo]

@twillis449 thanks a lot I will look at integrating them into a proper test, and search the rest of the codebase for stuff that can be run as tests possibly.

Inline

A few issues... I assume you compiled the programs crx2rnx and rnx2crx? you need them for un-compressing compressed rinex file (the compressed files end in 'd').

Yes I do, that is the first step in the build process in the Dockerfile. They are installed into /optsoft/bin in the container (and put in the PATH)

Also, you need to get the binary executable gfzrnx (see gnss.gfz-potsdam.de) Unfortunately they only provide binary executables and you have to sign up for an account. The current binary is gfzrnx_2.0-8219_lx64 and I just make a soft link gfzrnx to the binary. We need this so that if a site only provides data in RINEX3 format we can convert it into RINEX 2 format for analysis by ALBUS (which does not currently directly understand RINEX3). I already had to develop a conversion routine as Geosciences Australia converted to RINEX3 about a year or more ago.

This is more of a spanner in the works -- I need to review their licensing then. I presume we can't just dump the binary into the repo (not that I'm a fan of binary in a git repo -- this repo is already absolutely massive in its history). We certainly need to consider substantial cleanup and pruining

[twillis449]

Yeah - I don't think one can just a binary to the repo. I looked at their license - they are fairly flexible in most areas but it looks like one needs to have a lawyer investigate the wording for allowing distribution. Basically, they want individual users to register.

There is also a FORTRAN program cnvrnx3-rnx2 (see http://acc.igs.org/misc/cnvrnx3-rnx2-v3.f) which does not appear to have any license at all! But I (vaguely) remember trying it out about 2 years ago and had some problems with it, and thought gfzrnx more bullet-proof and friendly. But I will have another look.

Unfortunately, cnvrnx3-rnx2 seems to be basically brain-dead, plus it is written in basic F77, which does not help. I got it to psrse the input OK but get the error messages: Processing input file sutm2260.22o_rnx3

More than one s/n "S" column for L1 or L2 data and/or more than one phase for L1, according to the RINEX 3 file header. To convert to RINEX 2,11, only one such column can be present for L1's phase, only one for the s/n of the pair L1+P1, and one for the s/n of L2+P2. Chose one type of L1 phase, or of s/n signal, by changing the first letter in the identifier of the other phase or s/n data types to a "T" in the "SYS / # / OBS TYPES" header record.

so it does not seem to like 'current' Rinex 3 files, which gfzrnx handles perfectly.

[bennahugo]

Yup I've been dealing with various build issues all day in other projects. I have a way to deploy this on the build server but it does make the build not a one liner unfortunately (you have to tell docker from where to mount in the binary...), but that is first world problems so to speak -- maybe we can look at alternatives later.

I should have a working solution sometime later or perhaps tomorrow morning through our Jenkins server. Thanks for the tarball with the scripts.

[twillis449]

I have updated Albus_rnx3_to_rnx2.py in the 'python_scripts' subdirectory to have better error-handing in case no gfzrnx binary has been installed. Previously it would crash, but now in the case of the Sutherland test, the script will go to completion and say that no GPS stations were found.

[bennahugo]

Thanks Tony

Busy working on the Jenkins script now. Will request a review once I think it is working properly. You should see build statuses start appearing shortly once I commit the working changes.

On Tue, Sep 27, 2022 at 8:52 PM Tony Willis @.***> wrote:

I have updated Albus_rnx3_to_rnx2.py in the 'python_scripts' subdirectory to have better error-handing in case no gfzrnx binary has been installed.

— Reply to this email directly, view it on GitHub https://github.com/twillis449/ALBUS_ionosphere/issues/6#issuecomment-1259918505, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6TLYRXI2VJANULLRRLWAM65NANCNFSM6AAAAAAQWEPWJU . You are receiving this because you authored the thread.Message ID: @.***>

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[twillis449]

I processed your pull request. Things seem to be working. However, I haven't had any time actually to play with Docker. Instead, I've had to modify the system to handle the fact that GPS sites are starting to move away from simple anonymous FTP because of the vulnerability of anonymous ftp to hackers. Geosciences Australia has implemented sftp and changed the RINEX files from RINEX2 to RINEX3. This required quite a bit of work on my part as the actual analysis has to be done on RINEX2 files. Not that I expect ASKAP to actually do accurate ionosphere corrections in the near future as even their new supercomputer has trouble keeping up with the load.

[bennahugo]

Thanks Tony

Ok - yes sftp is probably the way we need to go (FTP is a largely deprecated protocol) - and I will need to likely do the same mounting trick to load in a key into the container? You can start a PR and I will review it and make the necessary changes to the build script to accommodate

Sorry for not yet getting to changing the ephemeris - been under the weather the past few.

On Tue, 11 Oct 2022, 07:19 Tony Willis, @.***> wrote:

I processed your pull request. Things seem to be working. However, I haven't had any time actually to play with Docker. Instead, I've had to modify the system to handle the fact that GPS sites are starting to move away from simple anonymous FTP because of the vulnerability of anonymous ftp to hackers. Geosciences Australia has implemented sftp and changed the RINEX files from RINEX2 to RINEX3. This required quite a bit of work on my part as the actual analysis has to be done on RINEX2 files. Not that I expect ASKAP to actually do accurate ionosphere corrections in the near future as even their new supercomputer has trouble keeping up with the load.

— Reply to this email directly, view it on GitHub https://github.com/twillis449/ALBUS_ionosphere/issues/6#issuecomment-1274101095, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6RCX4UZPM3LIUCFBCLWCT2H5ANCNFSM6AAAAAAQWEPWJU . You are receiving this because you authored the thread.Message ID: @.***>

[bennahugo]

Complete