search

twillis449 / ALBUS_ionosphere

software to determine ionosphere TEC and RM from GPS receiver data
Other
17 stars 8 forks source link

Spaceweather autoupdater #24

Closed bennahugo closed 1 year ago

bennahugo commented 1 year ago

@twillis449 Autoupdater for spaceweather. Have not looked at auto updating IRI values or the missing sunspots. The PIM paper says the daily F layer variations correlate poorly with sunspots. How important are they to fill?

Edit: not IRI but IMF24... hopefully this is not the International Monetary Fund.... Anyways jokes aside it can be done automated presumably: https://omniweb.gsfc.nasa.gov/html/command_line_sample.txt

bennahugo commented 1 year ago

@twillis449 can I ask that you please very carefully look at the diff before you merge this - you know these files better than I do.

bennahugo commented 1 year ago

The run with the new file is:

Observing  MeerKat_Moon_S0_2022_3C286
ALBUS data processing option: RI_G03
station position [[ 5109286.83908611  2006759.06551835 -3239067.78122689]]
Potential number of receivers for GPS fit  11
Final number of receivers for GPS fit  7
set ground station position to 5109286.83909 2006759.06552 -3239067.78123
reference time for rel_time=0: year,month,day,hr,min,sec  2022 9 4 11 57 56.0
Measurement set actual integration centroid start and end times  5169009476.400001 5169023438.01
observation direction  13:31:08.28811 30:30:32.9600

ALBUS report column explanation (zero relative):
0 - sequence number (zero relative)
1 - separator colon
2 - value of 0 - valid data/calculation
    value of 1 - invalid data/calculation (usually, but not always, because elevation < 0 )
3 - relative time in seconds relative to reference time
4 - time step between sequence calculations (default is 300 sec)
5 - elevation in degrees
6 - azimuth in degrees
7 - TEC (in tec units) in the current azimuth/elevation direction
8 - Rotation Measure (radians/m^2) in the current azimuth/elevation direction
9 - correction factor to convert STEC to value at the zenith
10 - formal error in STEC

seq  rel_time time_width El         Az         STEC           RM (rad/m2)   VTEC factor   STEC_error
0 : 0 -300.0 300.0 26.067083636011564 18.966418029364842 62.79653430523871 -2.822220785941102 0.513533525585749 4.793596306544505
1 : 0 0.0 300.0 26.40714241412465 17.8222401190778 62.286591655622836 -2.8171530224173496 0.517698481749122 6.047283911798521
2 : 0 300.0 300.0 26.726618557665358 16.66741766139365 61.93489936864836 -2.818141890382677 0.5216149905947867 7.442339204543259
3 : 0 600.0 300.0 27.02519573698834 15.50246668414697 62.43523752150068 -2.8574340503700792 0.5252779802016834 7.586308385032968
4 : 0 900.0 300.0 27.302572504857707 14.327940830557159 62.810410973738236 -2.8904477391456442 0.5286828029289912 6.071530382887026
5 : 0 1200.0 300.0 27.558463671529967 13.144430353298246 62.96673674727239 -2.9127963593987807 0.5318252188016825 3.3562382714928507
6 : 0 1500.0 300.0 27.79260163977962 11.952560831318353 62.81740132174839 -2.9202623077787586 0.5347013806670826 3.249922749986776
7 : 0 1800.0 300.0 28.004737704343633 10.752991530396585 62.59737741034027 -2.9235833931002384 0.5373078211822933 3.254946122014815
8 : 0 2100.0 300.0 28.194643285088393 9.546413488843841 62.458481794242545 -2.929838738211307 0.5396414412806908 3.467602226117205
9 : 0 2400.0 300.0 28.362111096074354 8.333547255636143 62.81609414670308 -2.958577341084005 0.541699500185499 4.260404283911528
10 : 0 2700.0 300.0 28.506956223098676 7.115140373720686 63.013781449115406 -2.979069699415482 0.5434796066868798 5.270764235673915
11 : 0 3000.0 300.0 28.62901710954322 5.891964547800309 62.1305191527567 -2.947520855485651 0.5449797117415541 5.580573567468946
12 : 0 3300.0 300.0 28.728156429072854 4.664812582644619 61.61645998097358 -2.932426576266436 0.5461981021961826 5.745916277500707
13 : 0 3600.0 300.0 28.80426183907879 3.434495094388786 60.79514175554899 -2.9016923054474404 0.5471333956241635 5.380439071700933
14 : 0 3900.0 300.0 28.857246606234312 2.2018370009125596 60.32963517057651 -2.8866012017891225 0.547784536230672 5.219577116602876
15 : 0 4200.0 300.0 28.887050091790023 0.967673891795571 58.74776766478641 -2.816747758213982 0.548150791727748 4.945480994065768
16 : 0 4500.0 300.0 28.893638095049997 -0.2671517628518134 58.003287573229464 -2.786008732583601 0.5482317512043864 4.530293294264458
17 : 0 4800.0 300.0 28.877003046970753 -1.5017944402155434 57.26074303255857 -2.755168735193792 0.5480273239247242 4.034826160869079
18 : 0 5100.0 300.0 28.83716405331317 -2.7354094685836094 56.25875851700602 -2.7103535139636357 0.5475377390659254 3.7671271182514205
19 : 0 5400.0 300.0 28.77416678633463 -3.96715688019751 55.846691542241686 -2.6928533092790863 0.5467635463874699 3.4122663008951233
20 : 0 5700.0 300.0 28.68808322555064 -5.196205249367292 55.50731327186293 -2.678081365437372 0.5457056178278856 3.0540756814514936
21 : 0 6000.0 300.0 28.579011252220308 -6.421735408348518 55.245372459665106 -2.6661014663692986 0.5443651500615181 3.0156805812586573
22 : 0 6300.0 300.0 28.44707410324199 -7.642944026158666 56.26078335833374 -2.715013041668464 0.5427436680476777 4.284574932175485
23 : 0 6600.0 300.0 28.29241968804921 -8.859047037901258 56.937291359525325 -2.746811148179755 0.5408430295676561 4.7054927090183245
24 : 0 6900.0 300.0 28.11521978375086 -10.06928282615459 56.23105784619358 -2.710999073172954 0.5386654308796693 5.095057302325786
25 : 0 7200.0 300.0 27.915669109887858 -11.272915203476432 56.491692928831604 -2.721048211870961 0.5362134134456218 4.9772930851107935
26 : 0 7500.0 300.0 27.693984304455036 -12.469236088997782 55.732303489211375 -2.6812847347983157 0.5334898719304988 4.457043509357754
27 : 0 7800.0 300.0 27.450402801557782 -13.6575679396574 55.28189444624888 -2.6557099983592285 0.5304980634151761 3.9173657961971697
28 : 0 8100.0 300.0 27.185181637001353 -14.83726584008549 54.923920153327856 -2.634081959482488 0.5272416180875028 3.5726751757269226
29 : 0 8400.0 300.0 26.898596180952993 -16.00771931878105 54.72843510199977 -2.6197866950729862 0.5237245513524623 3.5480304044014925
30 : 0 8700.0 300.0 26.590938823197177 -17.16835382308539 54.45822127783691 -2.60088957662732 0.519951277640089 3.5804447261065073
31 : 0 9000.0 300.0 26.262517620499636 -18.318631878361103 54.10736739830144 -2.577522006923189 0.5159266260109905 3.7140678882789833
32 : 0 9300.0 300.0 25.91365491175317 -19.458053959807007 53.7671236675739 -2.5541782160270357 0.5116558576342383 3.8455040267971894
33 : 0 9600.0 300.0 25.544685930534254 -20.58615902533604 53.24122780169558 -2.521040168172503 0.5071446855314464 3.9587512229539534
34 : 0 9900.0 300.0 25.155957405699844 -21.702524796697073 52.67313800624638 -2.4853652847580343 0.5023992965316328 4.082212994401453
35 : 0 10200.0 300.0 24.7478261847011 -22.806767728217498 52.99757199165004 -2.491151176404649 0.497426375952179 4.314166144311095
36 : 0 10500.0 300.0 24.32065786527883 -23.89854275717747 53.92116760975414 -2.524246380930307 0.4922331349552427 4.627449632106717
37 : 0 10800.0 300.0 23.87482547096017 -24.977542781458155 53.920661763102785 -2.5132528480379146 0.4868273411712104 4.5818889510663325
38 : 0 11100.0 300.0 23.410708152326162 -26.043497958216573 53.508524346203174 -2.4813294717136873 0.48121735256382914 4.35880598645088
39 : 0 11400.0 300.0 22.928689942945894 -27.096174788897716 52.59892203690448 -2.425437546051169 0.4754121551167569 4.540212338844392
40 : 0 11700.0 300.0 22.4291585681063 -28.135375037556194 52.05460189587109 -2.387743613713035 0.4694214045762609 4.751148368175066
41 : 0 12000.0 300.0 21.912504300314485 -29.16093452729693 51.161017442221436 -2.3326255198712724 0.46325547246251364 5.139954861453135
42 : 0 12300.0 300.0 21.379118891435702 -30.172721783259117 49.66868138187231 -2.2502449845880097 0.45692549700889346 5.702672158754782
43 : 0 12600.0 300.0 20.829394551889937 -31.17063661185692 49.867060335097605 -2.245342155605326 0.4504434389802302 6.654339587374337
44 : 0 12900.0 300.0 20.263723013554213 -32.15460856885505 49.89407743496629 -2.2324061094095646 0.44382214309927787 6.9408209588411225
45 : 0 13200.0 300.0 19.682494641793227 -33.124595404196576 49.19626483959527 -2.187074919061472 0.43707540492695984 6.657466504906699
46 : 0 13500.0 300.0 19.086097633837515 -34.080581435333755 49.409018181101466 -2.1822024999783474 0.4302180438212194 5.778235219822726
47 : 0 13800.0 300.0 18.474917266572824 -35.02257593020088 48.61423790521059 -2.1328763488953597 0.4232659816167038 6.092401791327637
48 : 0 14100.0 300.0 17.849335222668806 -35.95061146402504 48.04111112816871 -2.093660865471808 0.41623632727006993 6.4311158013593435
49 : 0 14400.0 300.0 17.209728982183936 -36.864742284866644 47.39547440908906 -2.051620863667761 0.40914746698505666 6.697756448018017
process_ionosphere: total run time:   12.62 minutes

the old file is

Observing  MeerKat_Moon_S0_2022_3C286
ALBUS data processing option: RI_G03
station position [[ 5109286.83908611  2006759.06551835 -3239067.78122689]]
Potential number of receivers for GPS fit  21
Final number of receivers for GPS fit  4
set ground station position to 5109286.83909 2006759.06552 -3239067.78123
reference time for rel_time=0: year,month,day,hr,min,sec  2022 9 4 11 57 56.0
Measurement set actual integration centroid start and end times  5169009476.400001 5169023438.01
observation direction  13:31:08.28811 30:30:32.9600

ALBUS report column explanation (zero relative):
0 - sequence number (zero relative)
1 - separator colon
2 - value of 0 - valid data/calculation
    value of 1 - invalid data/calculation (usually, but not always, because elevation < 0 )
3 - relative time in seconds relative to reference time
4 - time step between sequence calculations (default is 300 sec)
5 - elevation in degrees
6 - azimuth in degrees
7 - TEC (in tec units) in the current azimuth/elevation direction
8 - Rotation Measure (radians/m^2) in the current azimuth/elevation direction
9 - correction factor to convert STEC to value at the zenith
10 - formal error in STEC

seq  rel_time time_width El         Az         STEC           RM (rad/m2)   VTEC factor   STEC_error
0 : 0 -300.0 300.0 26.067083636011564 18.966418029364842 73.18465247487828 -3.0194597604681146 0.513533525585749 11.456240375737742
1 : 0 0.0 300.0 26.40714241412465 17.8222401190778 73.30910880843892 -3.042931175016207 0.517698481749122 12.243195857487231
2 : 0 300.0 300.0 26.726618557665358 16.66741766139365 72.23895386270891 -3.016252928285899 0.5216149905947867 12.922086904226818
3 : 0 600.0 300.0 27.02519573698834 15.50246668414697 74.49405894057051 -3.1281208257914535 0.5252779802016834 12.506080897295908
4 : 0 900.0 300.0 27.302572504857707 14.327940830557159 74.84084390094813 -3.159785181966726 0.5286828029289912 10.286953130255728
5 : 0 1200.0 300.0 27.558463671529967 13.144430353298246 75.78318797995918 -3.2162221728577443 0.5318252188016825 6.0347729630108224
6 : 0 1500.0 300.0 27.79260163977962 11.952560831318353 75.3557347224573 -3.213776987726646 0.5347013806670826 6.79919265053067
7 : 0 1800.0 300.0 28.004737704343633 10.752991530396585 75.09116210671738 -3.2174885616345867 0.5373078211822933 6.349867232154754
8 : 0 2100.0 300.0 28.194643285088393 9.546413488843841 74.21091876309393 -3.1938339898193298 0.5396414412806908 6.19749913965423
9 : 0 2400.0 300.0 28.362111096074354 8.333547255636143 74.14040006602247 -3.204002092661839 0.541699500185499 6.316002974841953
10 : 0 2700.0 300.0 28.506956223098676 7.115140373720686 73.67987046219642 -3.1964520127475216 0.5434796066868798 6.491044535350615
11 : 0 3000.0 300.0 28.62901710954322 5.891964547800309 73.00767906385812 -3.178747885130047 0.5449797117415541 7.042330690796795
12 : 0 3300.0 300.0 28.728156429072854 4.664812582644619 74.05740154326556 -3.2352863303370847 0.5461981021961826 5.217674680489558
13 : 0 3600.0 300.0 28.80426183907879 3.434495094388786 73.22010938004998 -3.2085611303473374 0.5471333956241635 5.249287347555491
14 : 0 3900.0 300.0 28.857246606234312 2.2018370009125596 73.0794233905558 -3.2096269844344407 0.547784536230672 2.607473382096013
15 : 0 4200.0 300.0 28.887050091790023 0.967673891795571 70.90525955324452 -3.1206729386139 0.548150791727748 1.7410250542205248
16 : 0 4500.0 300.0 28.893638095049997 -0.2671517628518134 70.25313600842733 -3.0982009633341887 0.5482317512043864 1.789452302562636
17 : 0 4800.0 300.0 28.877003046970753 -1.5017944402155434 69.45519728564696 -3.0686787735156797 0.5480273239247242 1.8937965137222563
18 : 0 5100.0 300.0 28.83716405331317 -2.7354094685836094 68.15090023294348 -3.0150903286076987 0.5475377390659254 3.1358649751346013
19 : 0 5400.0 300.0 28.77416678633463 -3.96715688019751 67.73781323134438 -2.9997397165011797 0.5467635463874699 3.943130430633735
20 : 0 5700.0 300.0 28.68808322555064 -5.196205249367292 67.39732332638744 -2.9865496300711887 0.5457056178278856 3.298662804651856
21 : 0 6000.0 300.0 28.579011252220308 -6.421735408348518 66.74281769256586 -2.9577748104978996 0.5443651500615181 3.894826166203927
22 : 0 6300.0 300.0 28.44707410324199 -7.642944026158666 66.32790974774642 -2.9380774821369293 0.5427436680476777 3.332638693330647
23 : 0 6600.0 300.0 28.29241968804921 -8.859047037901258 66.03733400502304 -2.9231081090663857 0.5408430295676561 2.737007588691665
24 : 0 6900.0 300.0 28.11521978375086 -10.06928282615459 65.47716268387606 -2.895026675894929 0.5386654308796693 2.6244003101179696
25 : 0 7200.0 300.0 27.915669109887858 -11.272915203476432 64.91863652817356 -2.868500274882319 0.5362134134456218 2.7932663177958403
26 : 0 7500.0 300.0 27.693984304455036 -12.469236088997782 64.62995375097323 -2.853362044199943 0.5334898719304988 2.851794692419934
27 : 0 7800.0 300.0 27.450402801557782 -13.6575679396574 64.11347904439894 -2.826858306915105 0.5304980634151761 2.7307149867364426
28 : 0 8100.0 300.0 27.185181637001353 -14.83726584008549 63.823915410006755 -2.8093315314150753 0.5272416180875028 2.4751066921244393
29 : 0 8400.0 300.0 26.898596180952993 -16.00771931878105 63.54910024630958 -2.7919074842431013 0.5237245513524623 2.290612403527631
30 : 0 8700.0 300.0 26.590938823197177 -17.16835382308539 63.14949766827498 -2.768205926190405 0.519951277640089 2.087543370570916
31 : 0 9000.0 300.0 26.262517620499636 -18.318631878361103 62.820730385772386 -2.7474313133904498 0.5159266260109905 1.9733424437246287
32 : 0 9300.0 300.0 25.91365491175317 -19.458053959807007 62.32872022459411 -2.718680555835648 0.5116558576342383 1.8457322496378747
33 : 0 9600.0 300.0 25.544685930534254 -20.58615902533604 61.27654366559681 -2.664862149753833 0.5071446855314464 2.917619868982485
34 : 0 9900.0 300.0 25.155957405699844 -21.702524796697073 60.61479678547387 -2.6273220870473337 0.5023992965316328 2.485271671623027
35 : 0 10200.0 300.0 24.7478261847011 -22.806767728217498 58.99717299773414 -2.548091112203247 0.497426375952179 2.703932096029943
36 : 0 10500.0 300.0 24.32065786527883 -23.89854275717747 59.19520219024237 -2.5469839037217583 0.4922331349552427 2.5722171012251906
37 : 0 10800.0 300.0 23.87482547096017 -24.977542781458155 58.83998691771816 -2.521490481196294 0.4868273411712104 2.438680339942598
38 : 0 11100.0 300.0 23.410708152326162 -26.043497958216573 58.27913114804294 -2.4845190186287955 0.48121735256382914 2.655828961473429
39 : 0 11400.0 300.0 22.928689942945894 -27.096174788897716 57.39063527137855 -2.4314969735770875 0.4754121551167569 2.9303436007142434
40 : 0 11700.0 300.0 22.4291585681063 -28.135375037556194 56.5772353986603 -2.382309389367273 0.4694214045762609 3.403450500135402
41 : 0 12000.0 300.0 21.912504300314485 -29.16093452729693 55.561353151403345 -2.327182311389531 0.46325547246251364 3.8973330712301166
42 : 0 12300.0 300.0 21.379118891435702 -30.172721783259117 54.61184365285579 -2.277472538210712 0.45692549700889346 4.516509399219488
43 : 0 12600.0 300.0 20.829394551889937 -31.17063661185692 55.93831305841488 -2.3213346621350914 0.4504434389802302 5.26411229243974
44 : 0 12900.0 300.0 20.263723013554213 -32.15460856885505 56.891146788730474 -2.346083289998608 0.44382214309927787 4.798287902585112
45 : 0 13200.0 300.0 19.682494641793227 -33.124595404196576 55.403541373900936 -2.2703595869855433 0.43707540492695984 5.693124436644665
46 : 0 13500.0 300.0 19.086097633837515 -34.080581435333755 55.82616700567625 -2.272998677871582 0.4302180438212194 3.390838061432316
47 : 0 13800.0 300.0 18.474917266572824 -35.02257593020088 54.92695850732902 -2.221975550241884 0.4232659816167038 3.7088300243763337
48 : 0 14100.0 300.0 17.849335222668806 -35.95061146402504 54.27984026086597 -2.1817181976725872 0.41623632727006993 4.063356880717958
49 : 0 14400.0 300.0 17.209728982183936 -36.864742284866644 53.69657394887901 -2.1445581908187417 0.40914746698505666 4.401691197938949
process_ionosphere: total run time:    7.69 minutes
bennahugo commented 1 year ago

Also addresses a problem in the recursive tries to get a working site running out of keys.

twillis449 commented 1 year ago

OMG! You have certainly been busy today!! A couple of comments - if you look at the files produced by Ottawa you will note that each day contains 3 values. That's because there are 3 'official measurements made each day - 9AM Noon, 3 PM. (Although looking at the entire file, back in early 2000's the number of measurements per day seems to vary. We are only interested in the Noon value. How are you stripping out that value and inserting it into the correct location in the kpf107 file? I absolutely refuse to create constipated FORTRAN login to make that happen - its already hard enough to figure out what's happening there as James Anderson creates a running average to smooth data over the last month. We want the fluxadjflux value. However late yesterday I found that Potsdam produces a separate file named Kp_ap_Ap_SN_F107_since_1932.txt which contains all solar info going back to 1932 which is updated daily and could easily replace the current file and can be handled by James Anderson's FORTRAN login just by modifying one FORTRAN format statement.I will send that info to you separately.

BTW - we do not need Sunspot number - that correlates very well with F10.7cm solar flux and I would say the solar flux is certainly a more accurate measuerment.

bennahugo commented 1 year ago

Tony you will see that I made the utc measurement for the flux number the 17h one as it says in the header so we continue getting the closest nearest neighbour to that ( I do a match between that and the values from the Canadian database). We can trivially change it to noon though.

On Fri, 21 Apr 2023, 19:13 Tony Willis, @.***> wrote:

OMG! You have certainly been busy today!! A couple of comments - if you look at the files produced by Ottawa you will note that each day contains 3 values. That's because there are 3 'official measurements made each day - 9AM Noon, 3 PM. (Although looking at the entire file, back in early 2000's the number of measurements per day seems to vary. We are only interested in the Noon value. How are you stripping out that value and inserting it into the correct location in the kpf107 file? I absolutely refuse to create constipated FORTRAN login to make that happen - its already hard enough to figure out what's happening there as James Anderson creates a running average to smooth data over the last month. We want the fluxadjflux value. However late yesterday I found that Potsdam produces a separate file named Kp_ap_Ap_SN_F107_since_1932.txt which contains all solar info going back to 1932 which is updated daily and could easily replace the current file and can be handled by James Anderson's FORTRAN login just by modifying one FORTRAN format statement.I will send that info to you separately.

BTW - we do not need Sunspot number - that correlates very well with F10.7cm solar flux and I would say the solar flux is certainly a more accurate measuerment.

— Reply to this email directly, view it on GitHub https://github.com/twillis449/ALBUS_ionosphere/pull/24#issuecomment-1518116518, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6W435GB3AFTX53TJMLXCK5ZZANCNFSM6AAAAAAXG4MRLE . You are receiving this because you authored the thread.Message ID: @.***>

twillis449 commented 1 year ago

Re: the old observation vs new observation files listed above - I re-ran this observation here and things look OK. But interesting that space weather adjustment seems to be lowering the TEC in this case. I also re-ran this observation with a single processor instead of using parallel processing and get the same result. I did this to make sure that whatever the space weather is doing it didn't just modify the first 'subdivision' when doing parallel processing but that things get propagated to the other subdivisions. That seems to be the case although it appears that the error bars change slightly. Below from left to right - your old data. your new data, my re-run here with one processor. no_space_weather_vs_space_weather_new

bennahugo commented 1 year ago

Thanks Tony

I'm going to try this out in anger now to get my final ALBUS results for the memo/paper. Will let you know once my spreadsheet is updated.

On Sun, Apr 23, 2023 at 10:33 PM Tony Willis @.***> wrote:

Re: the old observation vs new observation files listed above - I re-ran this observation here and things look OK. But interesting that space weather adjustment seems to be lowering the TEC in this case. I also re-ran this observation with a single processor instead of using parallel processing and get the same result. I did this to make sure that whatever the space weather is doing it didn't just modify the first 'subdivision' when doing parallel processing but that things get propagated to the other subdivisions. That seems to be the case although it appears that the error bars change slightly. Below from left to right - your old data. your new data, my re-run here with one processor. [image: no_space_weather_vs_space_weather_new] https://user-images.githubusercontent.com/6174767/233864593-36b307db-5791-408e-afd0-00d337a04861.png

— Reply to this email directly, view it on GitHub https://github.com/twillis449/ALBUS_ionosphere/pull/24#issuecomment-1519150148, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6QHBUH6KA32ZEBSVB3XCWG25ANCNFSM6AAAAAAXG4MRLE . You are receiving this because you authored the thread.Message ID: @.***>

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town