-> hdus.info():
Filename: /data/curtin_mwaeor/data/1059505936/raw/1059505936.metafits
No. Name Ver Type Cards Dimensions Format
0 PRIMARY 1 PrimaryHDU 63 ()
1 TILEDATA 1 BinTableHDU 52 256R x 19C [I, I, I, 8A, A, I, I, I, 14A, E, E, E, 24I, E, 16I, I, 10A, E, 24E]
HEADER
-> primary_hdu.header
SIMPLE = T / conforms to FITS standard
BITPIX = 8 / array data type
NAXIS = 0 / number of array dimensions
EXTEND = T
GPSTIME = 1059505936 / [s] GPS time of observation start
EXPOSURE= 112 / [s] duration of observation
FILENAME= 'high_season1_2456507' / Name of observation
MJD = 56506.8 / [days] MJD of observation
DATE-OBS= '2013-08-02T19:12:00' / [UT] Date and time of observation
LST = 356.212681700136 / [deg] LST
DUT1 = 0.05084286 / UT1-UTC from astropy.time.Time.delta_ut1_utc
HA = ' 00:00:00.00' / [hours] hour angle of pointing center
AZIMUTH = 0.0 / [deg] Azimuth of pointing center
ALTITUDE= 90.0 / [deg] Altitude of pointing center
RA = 356.033853326675 / [deg] RA of pointing center
DEC = -26.78033855169699 / [deg] Dec of pointing center
RAPHASE = 0.0 / [deg] RA of desired phase center
DECPHASE= -27.0 / [deg] DEC of desired phase center
DELAYMOD= / MWAX correlator delay mode
ATTEN_DB= 1.0 / [dB] global analogue attenuation, in dB
SUN-DIST= 139.231890839331 / [deg] Distance from pointing center to Sun
MOONDIST= 98.80546109496019 / [deg] Distance from pointing center to Moon
JUP-DIST= 111.036656502106 / [deg] Distance from pointing center to Jupiter
GRIDNAME= 'EOR1 ' / Pointing grid name
GRIDNUM = 0 / Pointing grid number
CREATOR = 'DJacobs ' / Observation creator
PROJECT = 'G0009 ' / Project ID
MODE = 'HW_LFILES' / Observation mode
RECVRS = '1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16' / Active receivers
DELAYS = '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0' / Beamformer delays
CALIBRAT= F / Not intended for calibration
CENTCHAN= 143 / Center coarse channel
CHANNELS= '131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147&'
CONTINUE ',148,149,150,151,152,153,154&'
CONTINUE '' / Coarse channels
CHANSEL = '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23' / Indi
SUN-ALT = -49.2318908393306 / [deg] Altitude of Sun
FINECHAN= 40.0 / [kHz] Fine channel width - correlator freq_res
INTTIME = 0.5 / [s] Individual integration time
NAV_FREQ= 4 / # of 10kHz chans avgd to make each fine chan
NSCANS = 224 / Number of time instants in correlation products
NINPUTS = 256 / Number of inputs into the correlation products
NCHANS = 768 / Number of (averaged) fine channels in spectrum
BANDWDTH= 30.72 / [MHz] Total bandwidth
FREQCENT= 182.415 / [MHz] Center frequency of observation
TIMEOFF = 0 / [s] Deprecated, use QUACKTIM or GOODTIME
DATESTRT= '2013-08-02T19:12:00' / [UT] Date and time of correlations start
RAWSCALE= 0.003 / Raw visibilities multiplied by this value
VERSION = 2.2 / METAFITS version number
TELESCOP= 'MWA '
INSTRUME= '128T '
QUACKTIM= 0.5 / Seconds of bad data after observation starts
GOODTIME= 1375470720.5 / OBSID+QUACKTIME as Unix timestamp
DATE = '2022-07-25T06:01:12' / UT Date of file creation
COMMENT After an observation starts, the receiver hardware changes take a
COMMENT few seconds to stabilise.
COMMENT QUACKTIM and GOODTIME represent the start of the first uncontaminated
COMMENT data, rounded up to the next time-averaged data packet. Note that this
COMMENT time may be before the first actual data in some or all gpubox files.
COMMENT See metafits file documentation here:
COMMENT https://wiki.mwatelescope.org/display/MP/Metafits+files
COMMENT https://wiki.mwatelescope.org/display/MP/MWAX+Metafits+Changes
HISTORY Created by user apache
HISTORY Created on host mwa-django01p.curtin.edu.au
HISTORY Command: "mod_wsgi"
first few ImageHDUs in the gpubox file:
-> hdus.info():
Filename: 1059505936_20130802191158_gpubox01_00.fits
No. Name Ver Type Cards Dimensions Format
0 PRIMARY 1 PrimaryHDU 11 ()
1 1 ImageHDU 12 (66048, 32) float32
2 1 ImageHDU 10 (66048, 32) float32
3 1 ImageHDU 12 (66048, 32) float32
4 1 ImageHDU 10 (66048, 32) float32
5 1 ImageHDU 12 (66048, 32) float32
6 1 ImageHDU 10 (66048, 32) float32
7 1 ImageHDU 12 (66048, 32) float32
8 1 ImageHDU 10 (66048, 32) float32
9 1 ImageHDU 12 (66048, 32) float32
10 1 ImageHDU 10 (66048, 32) float32
11 1 ImageHDU 12 (66048, 32) float32
12 1 ImageHDU 10 (66048, 32) float32
13 1 ImageHDU 12 (66048, 32) float32
14 1 ImageHDU 10 (66048, 32) float32
15 1 ImageHDU 12 (66048, 32) float32
16 1 ImageHDU 10 (66048, 32) float32
17 1 ImageHDU 12 (66048, 32) float32
18 1 ImageHDU 10 (66048, 32) float32
19 1 ImageHDU 12 (66048, 32) float32
20 1 ImageHDU 10 (66048, 32) float32
21 1 ImageHDU 12 (66048, 32) float32
22 1 ImageHDU 10 (66048, 32) float32
23 1 ImageHDU 12 (66048, 32) float32
24 1 ImageHDU 10 (66048, 32) float32
25 1 ImageHDU 12 (66048, 32) float32
26 1 ImageHDU 10 (66048, 32) float32
27 1 ImageHDU 12 (66048, 32) float32
28 1 ImageHDU 10 (66048, 32) float32
29 1 ImageHDU 12 (66048, 32) float32
30 1 ImageHDU 10 (66048, 32) float32
31 1 ImageHDU 12 (66048, 32) float32
32 1 ImageHDU 10 (66048, 32) float32
33 1 ImageHDU 12 (66048, 32) float32
34 1 ImageHDU 10 (66048, 32) float32
35 1 ImageHDU 12 (66048, 32) float32
36 1 ImageHDU 10 (66048, 32) float32
37 1 ImageHDU 12 (66048, 32) float32
38 1 ImageHDU 10 (66048, 32) float32
39 1 ImageHDU 12 (66048, 32) float32
40 1 ImageHDU 10 (66048, 32) float32
41 1 ImageHDU 12 (66048, 32) float32
42 1 ImageHDU 10 (66048, 32) float32
43 1 ImageHDU 12 (66048, 32) float32
44 1 ImageHDU 10 (66048, 32) float32
45 1 ImageHDU 12 (66048, 32) float32
46 1 ImageHDU 10 (66048, 32) float32
47 1 ImageHDU 12 (66048, 32) float32
48 1 ImageHDU 10 (66048, 32) float32
49 1 ImageHDU 12 (66048, 32) float32
50 1 ImageHDU 10 (66048, 32) float32
51 1 ImageHDU 12 (66048, 32) float32
52 1 ImageHDU 10 (66048, 32) float32
53 1 ImageHDU 12 (66048, 32) float32
54 1 ImageHDU 10 (66048, 32) float32
55 1 ImageHDU 12 (66048, 32) float32
56 1 ImageHDU 10 (66048, 32) float32
57 1 ImageHDU 12 (66048, 32) float32
58 1 ImageHDU 10 (66048, 32) float32
59 1 ImageHDU 12 (66048, 32) float32
60 1 ImageHDU 10 (66048, 32) float32
61 1 ImageHDU 12 (66048, 32) float32
62 1 ImageHDU 10 (66048, 32) float32
63 1 ImageHDU 12 (66048, 32) float32
64 1 ImageHDU 10 (66048, 32) float32
65 1 ImageHDU 12 (66048, 32) float32
66 1 ImageHDU 10 (66048, 32) float32
67 1 ImageHDU 12 (66048, 32) float32
68 1 ImageHDU 10 (66048, 32) float32
69 1 ImageHDU 12 (66048, 32) float32
70 1 ImageHDU 10 (66048, 32) float32
71 1 ImageHDU 12 (66048, 32) float32
72 1 ImageHDU 10 (66048, 32) float32
73 1 ImageHDU 12 (66048, 32) float32
74 1 ImageHDU 10 (66048, 32) float32
75 1 ImageHDU 12 (66048, 32) float32
76 1 ImageHDU 10 (66048, 32) float32
77 1 ImageHDU 12 (66048, 32) float32
78 1 ImageHDU 10 (66048, 32) float32
79 1 ImageHDU 12 (66048, 32) float32
80 1 ImageHDU 10 (66048, 32) float32
81 1 ImageHDU 12 (66048, 32) float32
82 1 ImageHDU 10 (66048, 32) float32
83 1 ImageHDU 12 (66048, 32) float32
84 1 ImageHDU 10 (66048, 32) float32
85 1 ImageHDU 12 (66048, 32) float32
86 1 ImageHDU 10 (66048, 32) float32
87 1 ImageHDU 12 (66048, 32) float32
88 1 ImageHDU 10 (66048, 32) float32
89 1 ImageHDU 12 (66048, 32) float32
90 1 ImageHDU 10 (66048, 32) float32
91 1 ImageHDU 12 (66048, 32) float32
92 1 ImageHDU 10 (66048, 32) float32
93 1 ImageHDU 12 (66048, 32) float32
94 1 ImageHDU 10 (66048, 32) float32
95 1 ImageHDU 12 (66048, 32) float32
96 1 ImageHDU 10 (66048, 32) float32
97 1 ImageHDU 12 (66048, 32) float32
98 1 ImageHDU 10 (66048, 32) float32
99 1 ImageHDU 12 (66048, 32) float32
100 1 ImageHDU 10 (66048, 32) float32
101 1 ImageHDU 12 (66048, 32) float32
102 1 ImageHDU 10 (66048, 32) float32
103 1 ImageHDU 12 (66048, 32) float32
104 1 ImageHDU 10 (66048, 32) float32
105 1 ImageHDU 12 (66048, 32) float32
106 1 ImageHDU 10 (66048, 32) float32
107 1 ImageHDU 12 (66048, 32) float32
108 1 ImageHDU 10 (66048, 32) float32
109 1 ImageHDU 12 (66048, 32) float32
110 1 ImageHDU 10 (66048, 32) float32
111 1 ImageHDU 12 (66048, 32) float32
112 1 ImageHDU 10 (66048, 32) float32
113 1 ImageHDU 12 (66048, 32) float32
114 1 ImageHDU 10 (66048, 32) float32
115 1 ImageHDU 12 (66048, 32) float32
116 1 ImageHDU 10 (66048, 32) float32
117 1 ImageHDU 12 (66048, 32) float32
118 1 ImageHDU 10 (66048, 32) float32
119 1 ImageHDU 12 (66048, 32) float32
120 1 ImageHDU 10 (66048, 32) float32
HEADER
SIMPLE = T / file does conform to FITS standard
BITPIX = -32 / number of bits per data pixel
NAXIS = 0 / number of data axes
EXTEND = T / FITS dataset may contain extensions
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
TIME = 1375470717 / Unix time (seconds)
MILLITIM= 0 / Milliseconds since TIME
INTTIME = 0.5 / Integration time (s)
PROJID = 'G0009' / MWA Project Id
OBSID = '1059505936' / MWA Observation Id
VISIBILITIES (120):
-> SCAN 0 IMAGE
XTENSION= 'IMAGE ' /Image Extension created by MWA DataCapture
BITPIX = -32 / number of bits per data pixel
NAXIS = 2 / number of data axes
NAXIS1 = 66048 / length of data axis 1
NAXIS2 = 32 / length of data axis 2
PCOUNT = 0 /
GCOUNT = 1 /
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
TIME = 1375470717 / Unix time (seconds)
MILLITIM= 0 / Milliseconds since TIME
INTTIME = 0.5 / Integration time (s)
-> SCAN 1 IMAGE
XTENSION= 'IMAGE ' / IMAGE extension
BITPIX = -32 / number of bits per data pixel
NAXIS = 2 / number of data axes
NAXIS1 = 66048 / length of data axis 1
NAXIS2 = 32 / length of data axis 2
PCOUNT = 0 / required keyword; must = 0
GCOUNT = 1 / required keyword; must = 1
TIME = 1375470717 / Unix time (seconds)
MILLITIM= 0 / Milliseconds since TIME
INTTIME = 0.5 / Integration time (s)
-> SCAN 2 IMAGE
XTENSION= 'IMAGE ' /Image Extension created by MWA DataCapture
BITPIX = -32 / number of bits per data pixel
NAXIS = 2 / number of data axes
NAXIS1 = 66048 / length of data axis 1
NAXIS2 = 32 / length of data axis 2
PCOUNT = 0 /
GCOUNT = 1 /
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
TIME = 1375470718 / Unix time (seconds)
MILLITIM= 0 / Milliseconds since TIME
INTTIME = 0.5 / Integration time (s)
-> SCAN 3 IMAGE
XTENSION= 'IMAGE ' / IMAGE extension
BITPIX = -32 / number of bits per data pixel
NAXIS = 2 / number of data axes
NAXIS1 = 66048 / length of data axis 1
NAXIS2 = 32 / length of data axis 2
PCOUNT = 0 / required keyword; must = 0
GCOUNT = 1 / required keyword; must = 1
TIME = 1375470718 / Unix time (seconds)
MILLITIM= 0 / Milliseconds since TIME
INTTIME = 0.5 / Integration time (s)
-> SCAN 4 IMAGE
XTENSION= 'IMAGE ' /Image Extension created by MWA DataCapture
BITPIX = -32 / number of bits per data pixel
NAXIS = 2 / number of data axes
NAXIS1 = 66048 / length of data axis 1
NAXIS2 = 32 / length of data axis 2
PCOUNT = 0 /
GCOUNT = 1 /
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
TIME = 1375470719 / Unix time (seconds)
MILLITIM= 0 / Milliseconds since TIME
INTTIME = 0.5 / Integration time (s)
-> SCAN 5 IMAGE
XTENSION= 'IMAGE ' / IMAGE extension
BITPIX = -32 / number of bits per data pixel
NAXIS = 2 / number of data axes
NAXIS1 = 66048 / length of data axis 1
NAXIS2 = 32 / length of data axis 2
PCOUNT = 0 / required keyword; must = 0
GCOUNT = 1 / required keyword; must = 1
TIME = 1375470719 / Unix time (seconds)
MILLITIM= 0 / Milliseconds since TIME
INTTIME = 0.5 / Integration time (s)
-> SCAN 6 IMAGE
XTENSION= 'IMAGE ' /Image Extension created by MWA DataCapture
BITPIX = -32 / number of bits per data pixel
NAXIS = 2 / number of data axes
NAXIS1 = 66048 / length of data axis 1
NAXIS2 = 32 / length of data axis 2
PCOUNT = 0 /
GCOUNT = 1 /
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
TIME = 1375470720 / Unix time (seconds)
MILLITIM= 0 / Milliseconds since TIME
INTTIME = 0.5 / Integration time (s)
I would expect that every second HDU should have the MILLITIM set to 500, but instead we have duplicate timestamps.
via birli, we can get the expected timesteps from mwalib according to the metafits. where there is a p next to the timestep, that timestep appears in provided timesteps.
[2022-07-25T06:49:30Z INFO birli::cli] birli version 0.7.0
Compiled on git commit hash: <no git info>
Thu, 14 Jul 2022 12:46:49 +0000
with compiler rustc 1.61.0 (fe5b13d68 2022-05-18)
observation name: high_season1_2456507
Array position: { longitude: 116.6708°, latitude: -26.7033°, height: 377.827m }
Phase centre: (0.0000°, -27.0000°) => (0h00m00.0000s, -27d00m00.0000s)
Pointing centre: (356.0339°, -26.7803°) => (23h44m08.1247s, -26d46m49.2187s)
Scheduled start: 2013-08-02 19:12:00.000 UTC, unix=1375470720.000, gps=1059505936.000, mjd=4882187520.000, lmst=356.2094°, lmst2k=356.0303°, lat2k=-26.7803°
Scheduled end: 2013-08-02 19:13:52.000 UTC, unix=1375470832.000, gps=1059506048.000, mjd=4882187632.000, lmst=356.6773°, lmst2k=356.4986°, lat2k=-26.7804°
Scheduled duration: 112.000s = 224 * 0.500s
Quack duration: 0.500s = 1 * 0.500s
Output duration: 112.000s = 56 * 2.000s (4x)
Scheduled Bandwidth: 30.720MHz = 24 * 32 * 40.000kHz
Output Bandwidth: 30.720MHz = 768 * 40.000kHz
Timestep details (all=230, provided=112, common=1, good=1, select=221, flag=230):
2013-08-02 UTC + unix [s] gps [s] p c g s f
ts0: 19:11:57.000 1375470717.000 1059505933.000 p f
ts1: 19:11:57.500 1375470717.500 1059505933.500 f
ts2: 19:11:58.000 1375470718.000 1059505934.000 p c s f
ts3: 19:11:58.500 1375470718.500 1059505934.500 s f
ts4: 19:11:59.000 1375470719.000 1059505935.000 p s f
ts5: 19:11:59.500 1375470719.500 1059505935.500 s f
ts6: 19:12:00.000 1375470720.000 1059505936.000 p s f
ts7: 19:12:00.500 1375470720.500 1059505936.500 s f
ts8: 19:12:01.000 1375470721.000 1059505937.000 p g s f
ts9: 19:12:01.500 1375470721.500 1059505937.500 s f
ts10: 19:12:02.000 1375470722.000 1059505938.000 p s f
ts11: 19:12:02.500 1375470722.500 1059505938.500 s f
...
mwalib does not give any warning about these duplicate timesteps.
Notes from Andrew Williams' analysis of GPU box timing, written in 2020:
2012 to 2014(ish): The Dark Ages - chaos and uncertainty, before the civilised era. Code changed day-to-day, there were no dipole tests or flags, and the QUACKTIME and GOODTIME cards were a distant dream. Leap second offsets were hard-coded in dozens of places. Don't trust any timestamps to better than a few seconds. The data in a visibility file starts anywhere from -1 to 4 seconds after the obsid, and the first 0-4 seconds should be discarded, depending on correlator averaging time. This era is why Cotter discards the first four seconds of data, by default. Here be dragons in the data.
2014(ish) to mid 2017 : The Renaissance - the code is stabilising, dipole tests and flags exist. From here, visibility files should always start 2 seconds after the start of an observation. There is no QUACKTIM or GOODTIME card in original the metafits file, but newly created metafits files will have those cards.
mid 2017 to now : The Modern Era - stable correlator code, no more leap seconds. QUACKTIM and GOODTIME headers, and data files that always start exactly 2 seconds after the obsid. Because of that 2 seconds without data, only attenuation settings should ever contaminate recorded data, only if the initial and final attenuations are different, and only for one correlator dump time.
2021 onwards: The Future - new (already tested) receiver code drops attenuation change times to < 0.5 seconds. The new MWAX correlator solves all of the problems of the old correlator, and has no bugs of its own. The promise of new receivers heralds the dawn of a new age
there are several obsids where the MILLITIM header is not set correctly.
from the metafits:
first few ImageHDUs in the gpubox file:
I would expect that every second HDU should have the
MILLITIMset to 500, but instead we have duplicate timestamps.via birli, we can get the expected timesteps from mwalib according to the metafits. where there is a p next to the timestep, that timestep appears in provided timesteps.
mwalib does not give any warning about these duplicate timesteps.
Notes from Andrew Williams' analysis of GPU box timing, written in 2020:
2012 to 2014(ish): The Dark Ages - chaos and uncertainty, before the civilised era. Code changed day-to-day, there were no dipole tests or flags, and the QUACKTIME and GOODTIME cards were a distant dream. Leap second offsets were hard-coded in dozens of places. Don't trust any timestamps to better than a few seconds. The data in a visibility file starts anywhere from -1 to 4 seconds after the obsid, and the first 0-4 seconds should be discarded, depending on correlator averaging time. This era is why Cotter discards the first four seconds of data, by default. Here be dragons in the data.
2014(ish) to mid 2017 : The Renaissance - the code is stabilising, dipole tests and flags exist. From here, visibility files should always start 2 seconds after the start of an observation. There is no QUACKTIM or GOODTIME card in original the metafits file, but newly created metafits files will have those cards.
mid 2017 to now : The Modern Era - stable correlator code, no more leap seconds. QUACKTIM and GOODTIME headers, and data files that always start exactly 2 seconds after the obsid. Because of that 2 seconds without data, only attenuation settings should ever contaminate recorded data, only if the initial and final attenuations are different, and only for one correlator dump time.
2021 onwards: The Future - new (already tested) receiver code drops attenuation change times to < 0.5 seconds. The new MWAX correlator solves all of the problems of the old correlator, and has no bugs of its own. The promise of new receivers heralds the dawn of a new age