Cleanup dockerize

[bennahugo]

Easier to work through this via GH. Here is the current compilation log with the problem (I had to add a few things just even for this to pick up e.g. all the include paths (esp. compiling that test case).

compilation.err.log

[bennahugo]

@twillis449 please see above along with my changes needed to get this far.

[bennahugo]

Got a bit further now running into this...

 Processing the orthogonal polynomial coefficients file ...
At line 202 of file usutounf.f (unit = 1, file = 'fshsmhde.nop')
Fortran runtime error: Bad value during floating point read

Error termination. Backtrace:
#0  0x7f96ff8ff32a
#1  0x7f96ff8ffed5
#2  0x7f96ff90069d
#3  0x7f96ffa74d6d
#4  0x7f96ffa780a8
#5  0x7f96ffa7972c
#6  0x55ba3438f4d6
#7  0x55ba3438f881
#8  0x7f96ff176c86
#9  0x55ba34387dc9
#10  0xffffffffffffffff
make[3]: *** [install] Error 2

deep inside Fortran land with GNU 7 going to try GNU 6 -- I believe that is shipped with Ubuntu 18.04

[bennahugo]

hmm it appears this runs on GNU 6.... however I'm not sure whether this is actually a deterministic build error

[twillis449]

Amongst other things its really annoying that James Anderson tossed everything including a dog's breakfast into ALBUS. The International Reference Ionosphere (IRI) stuff which you have had issues with is not even used anywhere for data reduction in ALBUS. Also He has all this VEX and SOFA software which can probabably be gotten rid of eventually. Unfortunately if one comments out the IRI section in the top level Makefile then other sections fail because of 'missing' include files. Ugh!

The GNU compilers on my Ubuntu 18.04 are V 7.5

[bennahugo]

Yea... hang on I think I MAY have it working now.. just struggling with the last of the install imports not finding libraries

[twillis449]

As I recall libraries should get copied into ~/albus/lib as soon as they are generated.

[bennahugo]

Of course it will help considerably if I could spell LD_LIBRARY_PATH properly..... inbound just some last cleanup edit -- few missing python imports eg urllib3

[bennahugo]

Oh hell yea!!

chmod -R ugo+rX /optsoft/ALBUS/share
Removing intermediate container c52a4c002f4f
 ---> 23369f3ee2ea
Step 59/59 : RUN python -c "import AlbusIonosphere" && echo "Crack the bubbly - this hog is airborne!!!"
 ---> Running in 4780d8d610a1
Crack the bubbly - this hog is airborne!!!
Removing intermediate container 4780d8d610a1
 ---> 42b7da9f5a20
Successfully built 42b7da9f5a20
Successfully tagged albus:latest

[bennahugo]

Need to move that one last apt install up to the master list at the top, but that can really wait till tomorrow - I need to leave the office before security throws me out...

Here is a tutorial on how to get docker running on Ubuntu 18.04 @twillis449 https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04

Take careful note to add the docker group to your user (and log out and back in as per usual) otherwise all docker commands to the daemon requires sudo. Be careful with docker though ... it is only intended as development helper -- don't use this on a user that you have publically exposed in any way shape or form (e.g. a Jenkins user with a public-facing login --- @o-smirnov can tell you all about root-escalations).

After that navigate to the root directory of the checked out branch (containing the Dockerfile) Then run

docker build -t albus .

We can work on cleaning up and mounting scripts tomorrow

[bennahugo]

@twillis449

Ok cleaned up dockerfile and added a blurp with help string, build as

$ docker build -t albus .

should yield

Step 58/61 : RUN python -c "import AlbusIonosphere" && echo "Crack the bubbly - this hog is airborne!!!"
 ---> Running in f7d4ee8c14ac
Crack the bubbly - this hog is airborne!!!
Removing intermediate container f7d4ee8c14ac
 ---> 690482e6b95b
Step 59/61 : ENTRYPOINT [ "/usr/bin/python3.6" ]
 ---> Running in 07da21da7f07
Removing intermediate container 07da21da7f07
 ---> a226fa733f51
Step 60/61 : ENV HELPSTRING "docker run -v <absolute path to gfzrnx>:/optsoft/bin/gfzrnx ""-v <absolute path to your waterhole with scripts>:/albus_waterhole ""--workdir /albus_waterhole ""--rm ""--user $(id -u <your user>):$(id -g <your user>) ""albus:latest <path to script mounted inside the waterhole>"
 ---> Running in 43252b0c65e9
Removing intermediate container 43252b0c65e9
 ---> a78143cd9dad
Step 61/61 : CMD [ "-c", "import AlbusIonosphere; import os; import pkg_resources; print('ALBUS\\n====='); print(AlbusIonosphere.__doc__); version = pkg_resources.require('AlbusIonosphere')[0].version; print('Version {}'.format(version)); print('Usage: ' + os.environ['HELPSTRING'])" ]
 ---> Running in 070a2599ef86
Removing intermediate container 070a2599ef86
 ---> a736fa293499
Successfully built a736fa293499
Successfully tagged albus:latest

this excludes including gfzrnx which you will have to mount in manually when running. You can run and get the blurp as

$ docker run -v /home/hugo/workspace/gfzrnx_2.0-8219_lx64:/optsoft/bin/gfzrnx -v acceptance_tests:/albus_waterhole --workdir /albus_waterhole --rm albus:latest

ALBUS
=====
Interface to c++ ionosphere stuff.
Version 1.0
Usage: docker run -v <absolute path to gfzrnx>:/optsoft/bin/gfzrnx -v <absolute path to your waterhole with scripts>:/albus_waterhole --workdir /albus_waterhole --rm --user $(id -u <your user>):$(id -g <your user>) albus:latest <path to script mounted inside the waterhole>

I've defined a special folder inside the container called /albus_waterhole to mount in scripts and get output (under the uid and gid you specify to docker run) to make workflows uniform

We can run your basic acceptance test like so (or any other script for that matter without the -m nose)

$docker run -v /home/hugo/workspace/gfzrnx_2.0-8219_lx64:/optsoft/bin/gfzrnx -v /home/hugo/workspace/ALBUS_ionosphere/acceptance_tests:/albus_waterhole --workdir /albus_waterhole --rm --user $(id -u hugo):$(id -g hugo) albus:latest -m nose /albus_waterhole

....
number of valid receivers 1
Starting total bias correction step 18
number of valid receivers 1
Starting total bias correction step 19
number of valid receivers 1
Starting total bias correction step 20
number of valid receivers 1
Starting total bias correction step 21
number of valid receivers 1
Got 1 receivers close to rec 0
Total bias correction finished
Wed Sep 28 17:45:38 UTC 2022
.
----------------------------------------------------------------------
Ran 1 test in 86.666s

OK

However I cannot get Sutherland and VLA to work Getting

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/albus_waterhole/test_rinex_autodownload_convert_v2_sutherland_mk_vla.py", line 81, in test_sutherland
    __run_rinex_predict(LONG, LAT, HEIGHT, OBJECT, DATA_DIR, MAX_DIST)
  File "/albus_waterhole/test_rinex_autodownload_convert_v2_sutherland_mk_vla.py", line 54, in __run_rinex_predict
    gps_data_directory=DATA_DIR)
  File "/optsoft/ALBUS/share/python/MS_Iono_functions.py", line 1458, in process_ionosphere
    MS,start_time,end_time, lat_set,Lat1,Long1,Ht1 = setup_AlbusIonosphere_for_ref_date(log,MSname,MSdir,Lat,Long,Height,start_time,end_time, object,time_step,telescope_pos,station_pos_x,station_pos_y,station_pos_z,max_dist, processing_option,tolerance,overwrite,do_serial,raise_bias_error,num_processors,use_pim,use_global_data, gps_data_directory)
  File "/optsoft/ALBUS/share/python/MS_Iono_functions.py", line 930, in setup_AlbusIonosphere_for_ref_date
    num_receivers, potential_num_receivers = ionosphere_GPS_setup_GPS(MS,MSlabel, processing_option,overwrite=overwrite,start_time=start_time, end_time=end_time,do_serial=do_serial,raise_bias_error=raise_bias_error,num_processors=num_processors, telescope_pos=station_pos, output_directory=gps_data_directory)
  File "/optsoft/ALBUS/share/python/MS_Iono_functions.py", line 663, in ionosphere_GPS_setup_GPS
    ionosphere_GPS_set_criteria.Bias_Fit_Type)):
  File "/optsoft/ALBUS/share/python/Albus_iono_object.py", line 42, in cal_observations_set_parameters
    raise RuntimeError("could not set criteria for self.iono")

[bennahugo]

I wonder if the last errors are not LST issues... I will play with the rise and sets

[bennahugo]

Hmm no albus good stations are empty.... I'm being stupid will fix

[twillis449]

The UT times you gave me should produce output for all 3 of those sites, assuming that you get get the Sutherland Rinex 3 file translated. The VLA data is there but would be pretty well useless as the elevation is very low,

[bennahugo]

hmm odd ok getting these for VLA

********** test_downloaded_RINEX failed with file  /albus_waterhole/.output/VLA_3C286_test_Aug_14_2022/mdo12260.22d.Z because file only has size  0  which is less than minimum allowed  4096

for

def test_vla():
    # VLA position
    LAT = "34:04:43.75"     
    LONG = "252:22:54.09"  
    # == 107:37:05.91 W 
    HEIGHT = 2115   
    OBJECT="VLA_3C286_Aug_14_2022"
    __checkmake_outputdir()
    DATA_DIR = '/albus_waterhole/.output/VLA_3C286_test_Aug_14_2022'
    START_TIME="2022/08/14 11:43:33.4"
    END_TIME="2022/08/14 16:55:20.9"

    MAX_DIST = 850E3
    __run_rinex_predict(LONG, LAT, HEIGHT, OBJECT, DATA_DIR, MAX_DIST, START_TIME, END_TIME)

[bennahugo]

Lets see if Jenkins calls... -- done with a cron job so lets give it 5

[ratt-priv-ci]

Can one of the admins verify this patch?

[bennahugo]

Alright @twillis449 I just need you to add a user called ratt-priv-ci to the contributors with write access and we should be good to go (just allows it to set status indicators and reports)

Its under repository settings

[twillis449]

I think I got the contributor thing to work. ratt-priv-ci should get an invitation.

[bennahugo]

ok to test

[bennahugo]

@twillis449 yup received and accepted. Sorry there is likely going to be some spam commits while I configure the test -- it is always trial by error

[bennahugo]

Alright @twillis449 can you review to see if you are happy with this. It at least set a basis for me to work on changes without worrying about breakage - if there are other tests you can think we should add then they can all be placed in the acceptance tests with the same format -- I presume we are only testing a small portion of the entire codebase with this 1 test case

[twillis449]

I'm not sure what I'm supposed to do here for review etc. If you are getting data etc you have probably tested ~85 % of the useful code base. As I've mentioned before there's a lot of stuff that can be thrown out. The question is how much work is required to do the throwing :)

[bennahugo]

You can just have a look and merge if you are happy with my changes?

[twillis449]

Ok - I hit the merge button :)

[bennahugo]

Great thanks. I'm busy reading the code on and off to see where I need to make changes to cover the various tickets. I will try to build a module for AIPS magnetosphere and add lunar ephemeris as first stabs I think

On Wed, 05 Oct 2022, 20:29 Tony Willis, @.***> wrote:

Ok - I hit the merge button :)

— Reply to this email directly, view it on GitHub https://github.com/twillis449/ALBUS_ionosphere/pull/2#issuecomment-1268794858, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6UZQB4LS2DFQ7PC373WBXCINANCNFSM6AAAAAAQSL5PLU . You are receiving this because you authored the thread.Message ID: @.***>