search

twillis449 / ALBUS_ionosphere

software to determine ionosphere TEC and RM from GPS receiver data
Other
18 stars 9 forks source link

[DO NOT MERGE] Investigation for #14 [Numerical problem with python3.x environment] #15

Closed bennahugo closed 1 year ago

bennahugo commented 1 year ago

This branch is only here for line-by-line investigative purposes. DO NOT try and merge

bennahugo commented 1 year ago

@twillis449 I started a branch here with the dropbox changes you sent. All I can immediately spot is we had a change in the IGRF. Do you think this is the cause behind this?

bennahugo commented 1 year ago

Hmm I've tried building both versions and running the same script, however I'm running into header issues which causes this to fail on the older version.

get_url had success
********** test_downloaded_RINEX failed with file  /albus_waterhole/smirnov_data_2021_zenith_dell_precision/codg2240.21i.Z because file only has size  0  which is less than minimum allowed  4096
in get_IONEX_file_from_web FTP_site = 1
Trying ftp://ftp.aiub.unibe.ch/CODE/2021/CODG2240.21I.Z
 in get_url to get  ftp://ftp.aiub.unibe.ch/CODE/2021/CODG2240.21I.Z /albus_waterhole/smirnov_data_2021_zenith_dell_precision/CODG2240.21I.Z
get_url had success
           Done
*** uncompressing file 1 to file 2  /albus_waterhole/smirnov_data_2021_zenith_dell_precision/CODG2240.21I.Z /albus_waterhole/smirnov_data_2021_zenith_dell_precision/codg2240.21i
gunzip executing  gunzip -dc /albus_waterhole/smirnov_data_2021_zenith_dell_precision/CODG2240.21I.Z > /albus_waterhole/smirnov_data_2021_zenith_dell_precision/codg2240.21i
gunzip returned  0
IONEX_filename + directory  /albus_waterhole/smirnov_data_2021_zenith_dell_precision/codg2240.21i
rinex_2 calling find_DCB_info_from_IONEX
in get_CODE_P1P2_file_from_web
in get_CODE_P1P2_file_from_web
year doy  2021 220
*********** accessing South African Trignet FTP site!
Trying ftp://ftp.trignet.co.za/RefData.21/220/L1L2_30sec/DEAR220Z.zip
 in get_url to get  ftp://ftp.trignet.co.za/RefData.21/220/L1L2_30sec/DEAR220Z.zip /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR220Z.zip
get_url had success
           Done
*** uncompressing file 1 to file 2  /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR220Z.zip /albus_waterhole/smirnov_data_2021_zenith_dell_precision/dear2200.21d
gunzip executing  unzip -np /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR220Z.zip '*d' > /albus_waterhole/smirnov_data_2021_zenith_dell_precision/dear2200.21d
gunzip returned  0
raise_bias_error =  0
******************** allowing data with no bias correction!!! ********
year doy  2021 221
*********** accessing South African Trignet FTP site!
Trying ftp://ftp.trignet.co.za/RefData.21/221/L1L2_30sec/DEAR221Z.zip
 in get_url to get  ftp://ftp.trignet.co.za/RefData.21/221/L1L2_30sec/DEAR221Z.zip /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR221Z.zip
get_url had success
           Done
*** uncompressing file 1 to file 2  /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR221Z.zip /albus_waterhole/smirnov_data_2021_zenith_dell_precision/dear2210.21d
gunzip executing  unzip -np /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR221Z.zip '*d' > /albus_waterhole/smirnov_data_2021_zenith_dell_precision/dear2210.21d
gunzip returned  0
raise_bias_error =  0
******************** allowing data with no bias correction!!! ********
year doy  2021 222
*********** accessing South African Trignet FTP site!
Trying ftp://ftp.trignet.co.za/RefData.21/222/L1L2_30sec/DEAR222Z.zip
 in get_url to get  ftp://ftp.trignet.co.za/RefData.21/222/L1L2_30sec/DEAR222Z.zip /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR222Z.zip
get_url had success
           Done
*** uncompressing file 1 to file 2  /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR222Z.zip /albus_waterhole/smirnov_data_2021_zenith_dell_precision/dear2220.21d
gunzip executing  unzip -np /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR222Z.zip '*d' > /albus_waterhole/smirnov_data_2021_zenith_dell_precision/dear2220.21d
gunzip returned  0
raise_bias_error =  0
******************** allowing data with no bias correction!!! ********
year doy  2021 223
*********** accessing South African Trignet FTP site!
Trying ftp://ftp.trignet.co.za/RefData.21/223/L1L2_30sec/DEAR223Z.zip
 in get_url to get  ftp://ftp.trignet.co.za/RefData.21/223/L1L2_30sec/DEAR223Z.zip /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR223Z.zip
get_url had success
           Done
*** uncompressing file 1 to file 2  /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR223Z.zip /albus_waterhole/smirnov_data_2021_zenith_dell_precision/dear2230.21d
gunzip executing  unzip -np /albus_waterhole/smirnov_data_2021_zenith_dell_precision/DEAR223Z.zip '*d' > /albus_waterhole/smirnov_data_2021_zenith_dell_precision/dear2230.21d
gunzip returned  0
raise_bias_error =  0
******************** allowing data with no bias correction!!! ********
MJD, gps_week, gps_dow, gps_seconds 59434 2170 0 0.0
 in get_GPS_ephemeris_file_from_web with FTP_site = 0
Trying ftp://garner.ucsd.edu/pub/products/2170/cod21700.eph.Z
 in get_url to get  ftp://garner.ucsd.edu/pub/products/2170/cod21700.eph.Z /albus_waterhole/smirnov_data_2021_zenith_dell_precision/cod21700.eph.Z
get_url had success
           Done
*** uncompressing file 1 to file 2  /albus_waterhole/smirnov_data_2021_zenith_dell_precision/cod21700.eph.Z /albus_waterhole/smirnov_data_2021_zenith_dell_precision/cod21700.eph
gunzip executing  gunzip -dc /albus_waterhole/smirnov_data_2021_zenith_dell_precision/cod21700.eph.Z > /albus_waterhole/smirnov_data_2021_zenith_dell_precision/cod21700.eph
gunzip returned  0
read_RINEX_sp3_file: trying to open  /albus_waterhole/smirnov_data_2021_zenith_dell_precision/cod21700.eph
Unsupported data type in line '/* PCV:IGS14      OL/AL:FES2014b NONE     YN ORB:CoN CLK:CoN                    
'
Failed to get data for GPS receiver 'dear' because of header/data problems, skipping
*********** finished initialization!!! ***************
 initial counter value  0
number of tasks  0
/usr/lib/python3/dist-packages/astropy/config/configuration.py:536: ConfigurationMissingWarning: Configuration defaults will be used due to PermissionError:13 on None
  warn(ConfigurationMissingWarning(msg))
/optsoft/ALBUS/share/python/Albus_RINEX.py:1119: UserWarning: No Differential Code Bias information found in '/albus_waterhole/smirnov_data_2021_zenith_dell_precision/codg2190.21i'
  warnings.warn("No Differential Code Bias information found in '%s'"%IONEX_filename)
/optsoft/ALBUS/share/python/Albus_RINEX.py:1119: UserWarning: No Differential Code Bias information found in '/albus_waterhole/smirnov_data_2021_zenith_dell_precision/codg2200.21i'
  warnings.warn("No Differential Code Bias information found in '%s'"%IONEX_filename)
/optsoft/ALBUS/share/python/Albus_RINEX.py:1119: UserWarning: No Differential Code Bias information found in '/albus_waterhole/smirnov_data_2021_zenith_dell_precision/codg2210.21i'
  warnings.warn("No Differential Code Bias information found in '%s'"%IONEX_filename)
/optsoft/ALBUS/share/python/Albus_RINEX.py:1119: UserWarning: No Differential Code Bias information found in '/albus_waterhole/smirnov_data_2021_zenith_dell_precision/codg2220.21i'
  warnings.warn("No Differential Code Bias information found in '%s'"%IONEX_filename)
/optsoft/ALBUS/share/python/Albus_RINEX.py:1119: UserWarning: No Differential Code Bias information found in '/albus_waterhole/smirnov_data_2021_zenith_dell_precision/codg2230.21i'
  warnings.warn("No Differential Code Bias information found in '%s'"%IONEX_filename)
/optsoft/ALBUS/share/python/Albus_RINEX.py:1119: UserWarning: No Differential Code Bias information found in '/albus_waterhole/smirnov_data_2021_zenith_dell_precision/codg2240.21i'
  warnings.warn("No Differential Code Bias information found in '%s'"%IONEX_filename)
/optsoft/ALBUS/share/python/Albus_RINEX_2.py:583: UserWarning: ValueError: probable satellite LOSS OF LOCK
  warnings.warn("ValueError: probable satellite LOSS OF LOCK")
/optsoft/ALBUS/share/python/Albus_RINEX_2.py:1894: UserWarning: Station 'dear' has no bias correction
  warnings.warn("Station '%s' has no bias correction"%station_code)
/optsoft/ALBUS/share/python/Albus_RINEX_2.py:1901: UserWarning: ********* warning: Station 'dear' has no bias correction
  warnings.warn("********* warning: Station '%s' has no bias correction"%station_code)
/optsoft/ALBUS/share/python/Albus_RINEX_2.py:944: UserWarning: Unknown SP3 version 'd' in file '/albus_waterhole/smirnov_data_2021_zenith_dell_precision/cod21700.eph', trying anyway ....
  warnings.warn("Unknown SP3 version '%s' in file '%s', trying anyway ...."%(version, filename))
*********** finished collecting station data !!! ***************
Traceback (most recent call last):
  File "/albus_waterhole/smirnov_experiment.py", line 50, in <module>
    iono.process_ionosphere(time_step=TIME_STEP,object=OBJECT,El=90.0,Lat=LAT,Long=LONG,Height=HEIGHT,start_time=START_TIME,end_time=END_TIME,max_dist=MAX_DIST,processing_option=RED_TYPE,do_serial=DO_SER,num_processors=NUM_PROCESSORS, gps_data_directory=DATA_DIR);
  File "/optsoft/ALBUS/share/python/MS_Iono_functions.py", line 1458, in process_ionosphere
    MS,start_time,end_time, lat_set,Lat1,Long1,Ht1 = setup_AlbusIonosphere_for_ref_date(log,MSname,MSdir,Lat,Long,Height,start_time,end_time, object,time_step,telescope_pos,station_pos_x,station_pos_y,station_pos_z,max_dist, processing_option,tolerance,overwrite,do_serial,raise_bias_error,num_processors,use_pim,use_global_data, gps_data_directory)
  File "/optsoft/ALBUS/share/python/MS_Iono_functions.py", line 930, in setup_AlbusIonosphere_for_ref_date
    num_receivers, potential_num_receivers = ionosphere_GPS_setup_GPS(MS,MSlabel, processing_option,overwrite=overwrite,start_time=start_time, end_time=end_time,do_serial=do_serial,raise_bias_error=raise_bias_error,num_processors=num_processors, telescope_pos=station_pos, output_directory=gps_data_directory)
  File "/optsoft/ALBUS/share/python/MS_Iono_functions.py", line 775, in ionosphere_GPS_setup_GPS
    raise RuntimeError("could not find any good GPS receivers for Iono_agw")
RuntimeError: could not find any good GPS receivers for Iono_agw

So I'm not sure how to reproduce your result with the tarballed version @twillis449

bennahugo commented 1 year ago

I'm building with exactly the same dockerfile both versions

bennahugo commented 1 year ago

Hmm @twillis449 I've now done a two way comparison with the same build under the v13 of the global magnetospheric model, but the results I get are almost exactly the same between the two runs on the smirnov_experiment script you sent Old: image

master: image

for the following script:

#
# A basic python script that tracks a specified position on the 
# sky over the time range from START_TIME to END_TIME from
# a specific location on the Earth's surface.

# The output is a text file giving Slant Tec (STEC) and
# ionosphere rotation measure (RM) as a function of time

import os
import time

import MS_Iono_functions as iono 
import math

if __name__ == "__main__":
  os.system('date')
  process_start = time.time()
  startime = time.strftime("%a, %d %b %Y %H:%M:%S", time.localtime())
  print("program start at %s" % startime)

  use_elev = True # to monitor variations of the ionosphere directly overhead
                   # set to False to track a given RA/DEC
# DEAR station position
  LONG="23:59:33.5"
  LAT= "-30:39:54.7"
  HEIGHT=1321.7

# source direction
  RA="20:09:36.80"
  DEC="-20:26:46.00"
  START_TIME="2022/8/8 16:45:31.00"
  END_TIME="2022/8/9 04:45:31.00"
  START_TIME="2021/8/8 14:00:00.00"
  END_TIME="2021/8/11 14:00:00.00"
  DATA_DIR = '/albus_waterhole/smirnov_data_2021_zenith_6540'
  DATA_DIR = '/albus_waterhole/smirnov_data_2021_zenith_dell_precision'
  RED_TYPE = 'RI_G01'
  TIME_STEP = 300
  MAX_DIST = 300E3
  NUM_PROCESSORS = 6
  DO_SER = 0
  DO_SER = 1

# call the following function if you want to just track changes in the ionosphere at an elevation of 90 deg
# i.e. direcly overhead
  if use_elev:
    OBJECT="PSR_J1141-6545_300km_300s_2017_90El"
    OBJECT="SMIRNOV_test_2021_zenith"
    iono.process_ionosphere(time_step=TIME_STEP,object=OBJECT,El=90.0,Lat=LAT,Long=LONG,Height=HEIGHT,start_time=START_TIME,end_time=END_TIME,max_dist=MAX_DIST,processing_option=RED_TYPE,do_serial=DO_SER,num_processors=NUM_PROCESSORS, gps_data_directory=DATA_DIR);

#  use the following function if you want to track pulsar position
  else:
    OBJECT="SMIRNOV_test_2021"
    iono.process_ionosphere(time_step=TIME_STEP,object=OBJECT,Ra=RA,Dec=DEC,Lat=LAT,Long=LONG,Height=HEIGHT,start_time=START_TIME,end_time=END_TIME,max_dist=MAX_DIST,processing_option=RED_TYPE,do_serial=DO_SER,num_processors=NUM_PROCESSORS, gps_data_directory=DATA_DIR);

  os.system('date')
  endtime = time.strftime("%a, %d %b %Y %H:%M:%S", time.localtime())
  print("program end at %s" % endtime)
  print (' ')
  process_end = time.time()
  duration = (process_end - process_start)/60.0
  print("program total run time: %7.2f minutes" % duration)

If you are getting a different result then the issue is likely different versions of dependencies. I'm using RNXCMP 4.1.0 at the moment along with gfzrnx 2.00-8219 as quoted by the binary.

Pip freeze says the following is my environment

AlbusIonosphere==1.0
asn1crypto==0.24.0
astropy==3.0
attrs==17.4.0
configobj==5.0.6
cryptography==2.1.4
cycler==0.10.0
decorator==4.1.2
ephem==3.7.6.0
future==0.15.2
idna==2.6
ipdb==0.10.3
ipython==5.5.0
ipython-genutils==0.2.0
keyring==10.6.0
keyrings.alt==3.0
matplotlib==2.1.1
nose==1.3.7
numpy==1.13.3
olefile==0.45.1
pexpect==4.2.1
pickleshare==0.7.4
Pillow==5.1.0
pluggy==0.6.0
ply==3.11
prompt-toolkit==1.0.15
psutil==5.4.2
py==1.5.2
pycrypto==2.6.1
pycurl==7.43.0.1
Pygments==2.2.0
pygobject==3.26.1
pyparsing==2.2.0
pytest==3.3.2
pytest-arraydiff==0.2
pytest-astropy==0.2.1
pytest-doctestplus==0.1.2
pytest-openfiles==0.2.0
pytest-remotedata==0.2.0
python-apt==1.6.5+ubuntu0.2
python-casacore==3.3.1
python-dateutil==2.6.1
pytz==2018.3
pyxdg==0.25
SecretStorage==2.3.1
simplegeneric==0.8.1
six==1.11.0
traitlets==4.3.2
unattended-upgrades==0.1
urllib3==1.22
wcwidth==0.1.7

Which versions are you running

bennahugo commented 1 year ago

Maybe I should try redoing the build for python2.7 explicitly. I will try that next

bennahugo commented 1 year ago

Hmm. Volla... changing the build to 2.7 and keeping the gfzrnx and rnxcmp the same I get the right thing

Here is a dockerfile to build this @o-smirnov and @twillis449

# ubuntu 18.04... for now
FROM kernsuite/base:6

# need to test whether we can migrate to GCC 10
ENV GNUCOMPILER 6
RUN docker-apt-install build-essential\ 
                       g++-$GNUCOMPILER \
                       gcc-$GNUCOMPILER \
                       gfortran-$GNUCOMPILER \
                       python-dev \
                       python-all \
                       ipython \
                       python-ipdb \
                       wcslib-dev \
                       python-astropy \
                       python-casacore \
                       casacore-data \
                       curl \
                       wget \
                       rsync \
                       python-pycurl \
                       python-matplotlib \
                       python-numpy \
                       python-ephem \
                       f2c \                                                               
                       libf2c2-dev \                                                       
                       bison \                                                             
                       flex \
                       python-urllib3 \
                       unzip \
                       python-nose

RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-$GNUCOMPILER 100 && \
    update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-$GNUCOMPILER 100 && \
    update-alternatives --install /usr/bin/x86_64-linux-gnu-gcc x86_64-linux-gnu-gcc /usr/bin/gcc-$GNUCOMPILER 100 && \
    update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-$GNUCOMPILER 100 && \
    update-alternatives --install /usr/bin/x86_64-linux-gnu-g++ x86_64-linux-gnu-g++ /usr/bin/g++-$GNUCOMPILER 100 && \
    update-alternatives --install /usr/bin/cpp cpp /usr/bin/g++-$GNUCOMPILER 100 && \
    update-alternatives --install /usr/bin/x86_64-linux-gnu-cpp x86_64-linux-gnu-cpp /usr/bin/g++-$GNUCOMPILER 100 && \
    update-alternatives --install /usr/bin/gfortran gfortran /usr/bin/gfortran-$GNUCOMPILER 100 && \
    update-alternatives --install /usr/bin/x86_64-linux-gnu-gfortran x86_64-linux-gnu-gfortran /usr/bin/gfortran-$GNUCOMPILER 100

RUN mkdir -p /optsoft/bin && \
    mkdir -p /optsoft/lib && \
    mkdir -p /optsoft/src && \
    mkdir -p /optsoft/include

ENV PATH $PATH:/optsoft/bin
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/optsoft/lib
ENV C_INCLUDE_PATH $C_INCLUDE_PATH:/optsoft/include
ENV CPLUS_INCLUDE_PATH $C_INCLUDE_PATH:/optsoft/include

# ---------- Build RNXCMP from source into /optsoft ---------- 
RUN mkdir /src
ADD RNXCMP_4.1.0_src.tar.gz /src/
WORKDIR /src/RNXCMP_4.1.0_src/source
RUN gcc-$GNUCOMPILER -ansi -O2 -static rnx2crx.c -o /optsoft/bin/RNX2CRX && \
    gcc-$GNUCOMPILER -ansi -O2 -static crx2rnx.c -o /optsoft/bin/CRX2RNX && \
    ln -s /optsoft/bin/CRX2RNX /optsoft/bin/crx2rnx && \
    ln -s /optsoft/bin/RNX2CRX /optsoft/bin/rnx2crx

# ---------- COMPILE ALBUS ----------
# Step 1 setup directory structure and move source code into container
RUN mkdir /src/ALBUS
ENV ALBUSPATH /src/ALBUS

ADD IMF24_request.png $ALBUSPATH
ADD build_rnx_crx $ALBUSPATH
ADD dates $ALBUSPATH
ADD definitions $ALBUSPATH
ADD fast_orbital_data $ALBUSPATH
ADD INSTALL $ALBUSPATH
ADD Makefile $ALBUSPATH
ADD iri.update $ALBUSPATH
ADD kill_python $ALBUSPATH
ADD LICENSE $ALBUSPATH
ADD README.md $ALBUSPATH
ADD remove_remainder $ALBUSPATH
ADD test_iri.f $ALBUSPATH
ADD UPDATING_SPACE_WEATHER $ALBUSPATH
ADD bin $ALBUSPATH/bin
ADD C++ $ALBUSPATH/C++
ADD examples $ALBUSPATH/examples
ADD FORTRAN $ALBUSPATH/FORTRAN
ADD include $ALBUSPATH/include
ADD IRI_2012 $ALBUSPATH/IRI_2012
ADD libdata $ALBUSPATH/libdata
ADD Python $ALBUSPATH/Python
ADD python_scripts $ALBUSPATH/python_scripts
ADD share $ALBUSPATH/share

# Step 2 configure install path and apply necessary patches to build system
RUN mkdir /optsoft/ALBUS
ENV ALBUSINSTALL /optsoft/ALBUS

## Configure Make custom paths .. should really convert this to cmake or something....
RUN sed -i '15s/.*/export INSTALLDIR = '$(echo ${ALBUSINSTALL} | sed 's/\//\\\//g')'/' $ALBUSPATH/Makefile
RUN sed -i '19s/.*/export CURRENT_PYTHON = python2.7/' $ALBUSPATH/Makefile
RUN sed -i '22s/.*/export PYTHONINCLUDEDIR = \/usr\/include\/python2.7/' $ALBUSPATH/Makefile
#### Build with specified GNU toolchain
RUN sed -i '39s/.*/export CC = gcc-'"${GNUCOMPILER}"'/' $ALBUSPATH/Makefile
RUN sed -i '40s/.*/export F77 = gfortran-'"${GNUCOMPILER}"' --std=legacy/' $ALBUSPATH/Makefile
RUN sed -i '43s/.*/export F77_RECL_UNIT = bytes/' $ALBUSPATH/Makefile
RUN sed -i '45s/.*/export C++ = g++-'"${GNUCOMPILER}"'/' $ALBUSPATH/Makefile
RUN sed -i '47s/.*/export CPP = cpp-'"${GNUCOMPILER}"' -P/' $ALBUSPATH/Makefile
## Patch up CFLAG passing... various missing includes and variable passing....
RUN sed -i '10s/.*/CFLAGS += -I$(PYTHONINCLUDEDIR) -I$(INSTALLDIR)\/include -DINSTALLDIR=\\"$(INSTALLDIR)\\"/' $ALBUSPATH/C++/mim/test/PIMrunner/Makefile
RUN sed -i '20s/.*/CFLAGS += -I$(PYTHONINCLUDEDIR) -I$(INSTALLDIR)\/include/' $ALBUSPATH/C++/AlbusIonosphere/python_attempt/Makefile
## Dead symlinks????
##> $ ls -lah share/python                                                                                                   [±cleanup_dockerize ●●]
##total 8.0K
##drwxrwxr-x 2 hugo hugo 4.0K Sep 26 11:44 .
##drwxrwxr-x 3 hugo hugo 4.0K Sep 26 11:44 ..
##lrwxrwxrwx 1 hugo hugo   42 Sep 26 11:44 Albus_Coordinates.py -> ../../maaijke_scripts/Albus_Coordinates.py
##lrwxrwxrwx 1 hugo hugo   38 Sep 26 11:44 Albus_RINEX_2.py -> ../../maaijke_scripts/Albus_RINEX_2.py
##lrwxrwxrwx 1 hugo hugo   36 Sep 26 11:44 Albus_RINEX.py -> ../../maaijke_scripts/Albus_RINEX.py
##lrwxrwxrwx 1 hugo hugo   37 Sep 26 11:44 GPS_stations.py -> ../../maaijke_scripts/GPS_stations.py
##lrwxrwxrwx 1 hugo hugo   34 Sep 26 11:44 jma_tools.py -> ../../maaijke_scripts/jma_tools.py
##lrwxrwxrwx 1 hugo hugo   36 Sep 26 11:44 MS_Iono_agw.py -> ../../maaijke_scripts/MS_Iono_agw.py

RUN rm $ALBUSPATH/share/python/*

# Step 3 Configure environment
ENV PATH "$ALBUSINSTALL/bin:$PATH"
ENV LD_LIBRARY_PATH "$ALBUSINSTALL/lib:$LD_LIBRARY_PATH"
ENV PYTHONPATH "$ALBUSINSTALL/share/python:$ALBUSINSTALL/lib:$PYTHONPATH"

# Step 4 Fingers crossed -- build
WORKDIR $ALBUSPATH
RUN make install
RUN python2.7 -c "import AlbusIonosphere" && echo "Crack the bubbly - this hog is airborne!!!"

# can run any script mounted inside the container py calling (assuming you run Bash or equiv.)
# docker run -v <absolute path to gfzrnx>:/optsoft/bin/gfzrnx \
#            -v <absolute path to your waterhole with scripts>:/albus_waterhole \
#            --workdir /albus_waterhole \
#            --rm \
#            --user $(id -u <your user>):$(id -g <your user>) \
#            albus:latest <path to script mounted inside the waterhole>
# if you used 'albus as a tag when building the image'
# you must download and accept the license for gfzrnx separately 
# see https://dataservices.gfz-potsdam.de/panmetaworks/showshort.php?id=escidoc:1577894
# Note: adding -it in the flags above should give you an interactive python session
ENTRYPOINT [ "/usr/bin/python2.7" ]
# print some default stuffs
ENV HELPSTRING "docker run -v <absolute path to gfzrnx>:/optsoft/bin/gfzrnx "\
"-v <absolute path to your waterhole with scripts>:/albus_waterhole "\
"--workdir /albus_waterhole "\
"--rm "\
"--user $(id -u <your user>):$(id -g <your user>) "\
"albus:latest <path to script mounted inside the waterhole>"
CMD [ "-c", "import AlbusIonosphere; import os; import pkg_resources; print('ALBUS\\n====='); print(AlbusIonosphere.__doc__); version = pkg_resources.require('AlbusIonosphere')[0].version; print('Version {}'.format(version)); print('Usage: ' + os.environ['HELPSTRING'])" ]
# crack the bubbly: this hog is airborne!

Here is the result image

This narrows it down at least. It is most likely a division error or something

bennahugo commented 1 year ago

At least this confirms it is not the Earth Magnetic field model change which underpins this.

bennahugo commented 1 year ago

I've put in some code in both versions downloaders not to redownload the curls. We need to put in an acceptance test as a first order test and then try to track down the issue.

bennahugo commented 1 year ago

I can at least reproduce these results with master when built with python 2.7. So I don't think we need multiple versions floating about. I will continue trying to track down the numerical issue on master

bennahugo commented 1 year ago

I've now systematically worked through the python_scripts folder with divisions (__future__ division) and try and track this one down by trying to introduce the breaking behavior under python 2.7. No cigar as of yet. It may not be a division-related issue. I will try and see if it is breakage in an external python package tomorrow

bennahugo commented 1 year ago

Downgrading the dependencies down to python2.7 dependency versions also does not resolve the issue.

bennahugo commented 1 year ago

Ok so as far as I can tell the good station list is markedly missing all stations except DEAR under version 3.x. This may begin to explain the numerical issues, although I still think negative sTEC may be indicative of an incorrect prior on the solver? I am perusing logs to see where the differences are

bennahugo commented 1 year ago

I've kludged in filters to reject all stations other than dear to make the inputs match between the 3.x and 2.7 builds. Even with just this station as input the 2.7 stec readings are physical. So I'm happy there are two distinct issues in the 3.x build @twillis449. For now I'm going to flip out the 3.x dockerfile for a 2.x version to make sure this doesn't trip up users. There is most definitely a numerical and parser issue to resolve. I will continue digging on the sidelines