Bug in Rdata when reading from disc

StuartLittlefair commented 6 years ago

Hi @trmrsh , bit of an urgent one this, since Tariq is trying to reduce the MAXI1820 data.

Found using drift mode data on 06-06-2018, run0011.fits.

When accessing data from disc using rtplot, reduce access to this file always stops at frame 2626. However, starting an instance of the fileserver and accessing the same data via this method works fine and can process all the data.

Similar issues are seen with 21-05-2018, run032.fits, which can't proceed past frame 6591.

We're using first=N in both cases.

trmrsh commented 6 years ago

That's very weird indeed. unfortunately our computer systems are going through a major upgrade today and I can't access the data. I suspect I won't be able to see from the code why this might be going wrong, but will take a look. Is there any error message?

trmrsh commented 6 years ago

thinking more, it has to be something about the data so until I can access it, I can't fix this and I am in Leicester tomorrow.

vikdhillon commented 6 years ago

No error message - it finishes cleanly (apparently).

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On Wed, 8 Aug 2018 at 14:06, Tom notifications@github.com wrote:

That's very weird indeed. unfortunately our computer systems are going through a major upgrade today and I can't access the data. I suspect I won't be able to see from the code why this might be going wrong, but will take a look. Is there any error message?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/HiPERCAM/hipercam/issues/54#issuecomment-411398258, or mute the thread https://github.com/notifications/unsubscribe-auth/ALnhnHnRCVG0bCC7GDw0dEV0gyTPcLqOks5uOuJZgaJpZM4VzySX .

vikdhillon commented 6 years ago

Hi Tom, If it helps, I could put it on our ftp site for you to copy over? Cheers, Vik.

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On Wed, 8 Aug 2018 at 14:08, Tom notifications@github.com wrote:

thinking more, it has to be something about the data so until I can access it, I can't fix this and I am in Leicester tomorrow.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

trmrsh commented 6 years ago

have you tried fiddling with twait / tmax? yes, make the smaller one available and I can download to my laptop at home.

vikdhillon commented 6 years ago

Hi Tom,

OK - you can download it using anonymous ftp at hydra.shef.ac.uk and then cd /local/ftp/pub/vsd, and then get run0032.fits (this is the file from 2018_05_21 that fails to plot beyond frame 6591).

Aren't tmax/twait more for the fileserver (it works fine when you plot using hs, but not hl)? Anyway, I tried messing around with the numbers - problem still there.

Cheers,

Vik.

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On Wed, 8 Aug 2018 at 14:11, Tom notifications@github.com wrote:

have you tried fiddling with twait / tmax? yes, make the smaller one available and I can download to my laptop at home.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/HiPERCAM/hipercam/issues/54#issuecomment-411399892, or mute the thread https://github.com/notifications/unsubscribe-auth/ALnhnJ6msXY1fGjhpof9qobfJyHzBuGAks5uOuOQgaJpZM4VzySX .

trmrsh commented 6 years ago

OK, downloading ...

trmrsh commented 6 years ago

and can replicate bug. So hang on for a fix (I hope)

trmrsh commented 6 years ago

OK, I can see why it is doing this. A bit worrying: there is an integer in the timing bytes giving the framecount and in this case its seems to have skipped one. I will look in some more detail.

StuartLittlefair commented 6 years ago

That's alarming - to be clear are you saying you are keeping your own running counter of the frame number and it disagrees with the one in the timing bytes?

trmrsh commented 6 years ago

Looking at the times, they seem consistent with it having dropped a frame -- does this seem possible Stu? Could a frame get dropped for some reason? Anyway, I will change the code to a warning rather than just stopping.

StuartLittlefair commented 6 years ago

I think the answer is yes, a frame could get dropped if the ring buffer used to stream data to disk gets full; this is really a question for @eh2006re though....

trmrsh commented 6 years ago

OK, I have pushed a fix. Look out for a warning appearing on screen when a frame gets dropped. Please try it out on the data I did not download.

vikdhillon commented 6 years ago

Oh damn - hate it when we find this kind of problem. This setup is pushing the system pretty hard, with 96x72 windows binned 8x8 and fast readout and clocks, but I thought we'd tested even faster without any problems. Last time we made a big difference when Gao removed the GPS status check on each timestamp write - I wonder if we can refine things even further in his timestamping code?

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On Wed, 8 Aug 2018 at 14:54, Tom notifications@github.com wrote:

OK, I have pushed a fix. Look out for a warning appearing on screen when a frame gets dropped. Please try it out on the data I did not download.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

trmrsh commented 6 years ago

Yes, I sort of assumed it was not going to happen so just stopping wasn't the right thing to do. Anyway, the warning statements should tell us when frames are dropped. But I can only have them in the non-server access mode because one does not necessarily expect sequential frame numbers when accessing via the server. Can I close this bug out now?

vikdhillon commented 6 years ago

Hi Tom,

Yes, the new version works fine. I ran rtplot on run0032 on 2018_05_21.

It reported 28 dropped frames in total, out of 164581, i.e. 0.02%.

I made rtplot output go to a file. At the end it wrote the following to the file:

So something weird happened with the last 4 frames.

On the screen, rtplot finished with an error, as follows:

vsd@pulsar/ucam10/vsd/hcam/2018_05_21> rtplot first=1 \ > log.dat

WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; 2686328123 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2686492704:2686492704:2686492704:2686492704:2686492706.6864929 warnings.warn('Bad timestamp: ' + time_string) WARNING: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)" [astropy._erfa.core] WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "dubious year (Note 6)" [astropy._erfa.core] WARNING: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)" [astropy._erfa.core] WARNING: frame count mis-match; -538976289 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2147516416:2147516416:2147516416:2147516416:2147516418.1475163 warnings.warn('Bad timestamp: ' + time_string) WARNING: frame count mis-match; 1411860813 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2686492704:2686492704:2686492704:2686492704:2686492706.6864929 warnings.warn('Bad timestamp: ' + time_string) WARNING: frame count mis-match; -1411860815 frames seem to have been dropped vsd@pulsar/ucam10/vsd/hcam/2018_05_21>

So it doesn't seem to have counted the dropped frames correctly.

Interestingly, if I start rtplot plotting at frame=164000, then it finishes with a similar error:

164579, utc = 2018-05-22 05:19:52.010, ccd 1: -11.7, 50.6, exp: 0.0020, ccd 2: -18.0, 414.7, exp: 0.0020, ccd 3: -14.0, 575.8, exp: 0.0020, ccd 4: -14.7, 520.6, exp: 0.0020, ccd 5: -14.7, 466.1, exp: 0.0020 164580, utc = 2018-05-22 05:19:52.013, ccd 1: -12.5, 53.0, exp: 0.0020, ccd 2: -18.7, 428.0, exp: 0.0020, ccd 3: -16.0, 572.0, exp: 0.0020, ccd 4: -15.7, 525.3, exp: 0.0020, ccd 5: -15.4, 459.2, exp: 0.0020 164581, utc = 2018-05-22 05:19:52.016, ccd 1: -12.0, 54.0, exp: 0.0020, ccd 2: -16.5, 425.0, exp: 0.0020, ccd 3: -16.0, 566.3, exp: 0.0020, ccd 4: -16.0, 556.1, exp: 0.0020, ccd 5: -14.0, 440.3, exp: 0.0020

WARNING: frame count mis-match; 2686328123 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2686492704:2686492704:2686492704:2686492704:2686492706.6864929 warnings.warn('Bad timestamp: ' + time_string) WARNING: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)" [astropy._erfa.core] WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "dubious year (Note 6)" [astropy._erfa.core] WARNING: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)" [astropy._erfa.core] 2686492705, utc = 2085-02-16 15:58:25.000, ccd 1: -8224.0, 21298.4, exp: 0.0020, ccd 2: -8224.0, 14640.0, exp: 0.0020, ccd 3: -8224.0, 20301.9, exp: 0.0020, ccd 4: -8224.0, 21298.4, exp: 0.0020, ccd 5: -8224.0, 13350.0, exp: 0.0020 WARNING: frame count mis-match; -538976289 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2147516416:2147516416:2147516416:2147516416:2147516418.1475163 warnings.warn('Bad timestamp: ' + time_string) 2147516417, utc = 2068-01-19 12:20:17.000, ccd 1: -38633.0, 21094.7, exp: 0.0020, ccd 2: -36700.1, 20943.6, exp: 0.0020, ccd 3: -37241.5, 21635.1, exp: 0.0020, ccd 4: -38215.4, 23125.2, exp: 0.0020, ccd 5: -32153.0, 22059.6, exp: 0.0020 WARNING: frame count mis-match; 1411860813 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2686492704:2686492704:2686492704:2686492704:2686492706.6864929 warnings.warn('Bad timestamp: ' + time_string) 3559377231, utc = 2112-10-16 11:53:51.000, ccd 1: -8224.0, 21415.5, exp: 0.0020, ccd 2: -8224.0, 20954.7, exp: 0.0020, ccd 3: -8224.0, 13350.0, exp: 0.0020, ccd 4: -8224.0, 21298.4, exp: 0.0020, ccd 5: -8224.0, 14640.0, exp: 0.0020 WARNING: frame count mis-match; -1411860815 frames seem to have been dropped 2147516417, utc = 2068-01-19 12:20:17.000, ccd 1: -33297.2, 24755.4, exp: 0.0020, ccd 2: -35101.2, 22501.2, exp: 0.0020, ccd 3: -31668.5, 23174.4, exp: 0.0020, ccd 4: -34379.3, 23669.5, exp: 0.0020, ccd 5: -32489.0, 26395.8, exp: 0.0020 henderror raised

Cheers,

Vik.

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On Wed, 8 Aug 2018 at 15:08, Tom notifications@github.com wrote:

Yes, I sort of assumed it was not going to happen so just stopping wasn't the right thing to do. Anyway, the warning statements should tell us when frames are dropped. But I can only have them in the non-server access mode because one does not necessarily expect sequential frame numbers when accessing via the server. Can I close this bug out now?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

vikdhillon commented 6 years ago

Sorry, omitted what had been written to the log file:

164579, utc = 2018-05-22 05:19:52.010, ccd 1: -11.7, 50.6, exp: 0.0020, ccd 2: -18.0, 414.7, exp: 0.0020, ccd 3: -14.0, 575.8, exp: 0.0020, ccd 4: -14.7, 520.6, exp: 0.0020, ccd 5: -14.7, 466.1, exp: 0.0020 164580, utc = 2018-05-22 05:19:52.013, ccd 1: -12.5, 53.0, exp: 0.0020, ccd 2: -18.7, 428.0, exp: 0.0020, ccd 3: -16.0, 572.0, exp: 0.0020, ccd 4: -15.7, 525.3, exp: 0.0020, ccd 5: -15.4, 459.2, exp: 0.0020 164581, utc = 2018-05-22 05:19:52.016, ccd 1: -12.0, 54.0, exp: 0.0020, ccd 2: -16.5, 425.0, exp: 0.0020, ccd 3: -16.0, 566.3, exp: 0.0020, ccd 4: -16.0, 556.1, exp: 0.0020, ccd 5: -14.0, 440.3, exp: 0.0020 2686492705, utc = 2085-02-16 15:58:25.000, ccd 1: -8224.0, 21298.4, exp: 0.0020, ccd 2: -8224.0, 14640.0, exp: 0.0020, ccd 3: -8224.0, 20301.9, exp: 0.0020, ccd 4: -8224.0, 21298.4, exp: 0.0020, ccd 5: -8224.0, 13350.0, exp: 0.0020 2147516417, utc = 2068-01-19 12:20:17.000, ccd 1: -38633.0, 21094.7, exp: 0.0020, ccd 2: -36700.1, 20943.6, exp: 0.0020, ccd 3: -37241.5, 21635.1, exp: 0.0020, ccd 4: -38215.4, 23125.2, exp: 0.0020, ccd 5: -32153.0, 22059.6, exp: 0.0020 3559377231, utc = 2112-10-16 11:53:51.000, ccd 1: -8224.0, 21415.5, exp: 0.0020, ccd 2: -8224.0, 20954.7, exp: 0.0020, ccd 3: -8224.0, 13350.0, exp: 0.0020, ccd 4: -8224.0, 21298.4, exp: 0.0020, ccd 5: -8224.0, 14640.0, exp: 0.0020 2147516417, utc = 2068-01-19 12:20:17.000, ccd 1: -33297.2, 24755.4, exp: 0.0020, ccd 2: -35101.2, 22501.2, exp: 0.0020, ccd 3: -31668.5, 23174.4, exp: 0.0020, ccd 4: -34379.3, 23669.5, exp: 0.0020, ccd 5: -32489.0, 26395.8, exp: 0.0020 henderror raised

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On Thu, 9 Aug 2018 at 11:37, Vik Dhillon vik.dhillon@sheffield.ac.uk wrote:

Hi Tom,

Yes, the new version works fine. I ran rtplot on run0032 on 2018_05_21.

It reported 28 dropped frames in total, out of 164581, i.e. 0.02%.

I made rtplot output go to a file. At the end it wrote the following to the file:

So something weird happened with the last 4 frames.

On the screen, rtplot finished with an error, as follows:

vsd@pulsar/ucam10/vsd/hcam/2018_05_21> rtplot first=1 \ > log.dat

WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; a frame seems to have been dropped WARNING: frame count mis-match; 2686328123 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2686492704:2686492704:2686492704:2686492704:2686492706.6864929 warnings.warn('Bad timestamp: ' + time_string) WARNING: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)" [astropy._erfa.core] WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "dubious year (Note 6)" [astropy._erfa.core] WARNING: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)" [astropy._erfa.core] WARNING: frame count mis-match; -538976289 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2147516416:2147516416:2147516416:2147516416:2147516418.1475163 warnings.warn('Bad timestamp: ' + time_string) WARNING: frame count mis-match; 1411860813 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2686492704:2686492704:2686492704:2686492704:2686492706.6864929 warnings.warn('Bad timestamp: ' + time_string) WARNING: frame count mis-match; -1411860815 frames seem to have been dropped vsd@pulsar/ucam10/vsd/hcam/2018_05_21>

So it doesn't seem to have counted the dropped frames correctly.

Interestingly, if I start rtplot plotting at frame=164000, then it finishes with a similar error:

164579, utc = 2018-05-22 05:19:52.010, ccd 1: -11.7, 50.6, exp: 0.0020, ccd 2: -18.0, 414.7, exp: 0.0020, ccd 3: -14.0, 575.8, exp: 0.0020, ccd 4: -14.7, 520.6, exp: 0.0020, ccd 5: -14.7, 466.1, exp: 0.0020 164580, utc = 2018-05-22 05:19:52.013, ccd 1: -12.5, 53.0, exp: 0.0020, ccd 2: -18.7, 428.0, exp: 0.0020, ccd 3: -16.0, 572.0, exp: 0.0020, ccd 4: -15.7, 525.3, exp: 0.0020, ccd 5: -15.4, 459.2, exp: 0.0020 164581, utc = 2018-05-22 05:19:52.016, ccd 1: -12.0, 54.0, exp: 0.0020, ccd 2: -16.5, 425.0, exp: 0.0020, ccd 3: -16.0, 566.3, exp: 0.0020, ccd 4: -16.0, 556.1, exp: 0.0020, ccd 5: -14.0, 440.3, exp: 0.0020

WARNING: frame count mis-match; 2686328123 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2686492704:2686492704:2686492704:2686492704:2686492706.6864929 warnings.warn('Bad timestamp: ' + time_string) WARNING: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)" [astropy._erfa.core] WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "dubious year (Note 6)" [astropy._erfa.core] WARNING: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)" [astropy._erfa.core] 2686492705, utc = 2085-02-16 15:58:25.000, ccd 1: -8224.0, 21298.4, exp: 0.0020, ccd 2: -8224.0, 14640.0, exp: 0.0020, ccd 3: -8224.0, 20301.9, exp: 0.0020, ccd 4: -8224.0, 21298.4, exp: 0.0020, ccd 5: -8224.0, 13350.0, exp: 0.0020 WARNING: frame count mis-match; -538976289 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2147516416:2147516416:2147516416:2147516416:2147516418.1475163 warnings.warn('Bad timestamp: ' + time_string) 2147516417, utc = 2068-01-19 12:20:17.000, ccd 1: -38633.0, 21094.7, exp: 0.0020, ccd 2: -36700.1, 20943.6, exp: 0.0020, ccd 3: -37241.5, 21635.1, exp: 0.0020, ccd 4: -38215.4, 23125.2, exp: 0.0020, ccd 5: -32153.0, 22059.6, exp: 0.0020 WARNING: frame count mis-match; 1411860813 frames seem to have been dropped /home/vsd/.local/lib/python3.5/site-packages/hipercam/hcam.py:993: UserWarning: Bad timestamp: 2686492704:2686492704:2686492704:2686492704:2686492706.6864929 warnings.warn('Bad timestamp: ' + time_string) 3559377231, utc = 2112-10-16 11:53:51.000, ccd 1: -8224.0, 21415.5, exp: 0.0020, ccd 2: -8224.0, 20954.7, exp: 0.0020, ccd 3: -8224.0, 13350.0, exp: 0.0020, ccd 4: -8224.0, 21298.4, exp: 0.0020, ccd 5: -8224.0, 14640.0, exp: 0.0020 WARNING: frame count mis-match; -1411860815 frames seem to have been dropped 2147516417, utc = 2068-01-19 12:20:17.000, ccd 1: -33297.2, 24755.4, exp: 0.0020, ccd 2: -35101.2, 22501.2, exp: 0.0020, ccd 3: -31668.5, 23174.4, exp: 0.0020, ccd 4: -34379.3, 23669.5, exp: 0.0020, ccd 5: -32489.0, 26395.8, exp: 0.0020 henderror raised

Cheers,

Vik.

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On Wed, 8 Aug 2018 at 15:08, Tom notifications@github.com wrote:

Yes, I sort of assumed it was not going to happen so just stopping wasn't the right thing to do. Anyway, the warning statements should tell us when frames are dropped. But I can only have them in the non-server access mode because one does not necessarily expect sequential frame numbers when accessing via the server. Can I close this bug out now?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

trmrsh commented 6 years ago

this is just bad data I assume; is there anything you would like me to do about it? I wonder if the run was terminated because of this?

trmrsh commented 6 years ago

In case it helps, what I do is retrieve the frame number from the timing bytes, store it, and then on the next frame I know what to expect and thus I am able to issue the warning. The enormous number of frames apparently dropped is presumably a sign of junk data in the timing bytes. The way it's done, it should make no difference where you start

vikdhillon commented 6 years ago

Hi Tom,

Yes, it looks like the last 4 frames have corrupt timestamps and frame numbers. I don't understand why this is, as it doesn't usually happen with drift mode runs, and it doesn't seem to be related numerically to the number of dropped frames (of which there were 28, not 4). Any ideas? (No need to change the pipeline to try to handle this.)

Cheers,

Vik

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On Thu, 9 Aug 2018 at 13:29, Tom notifications@github.com wrote:

this is just bad data I assume; is there anything you would like me to do about it? I wonder if the run was terminated because of this?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/HiPERCAM/hipercam/issues/54#issuecomment-411740370, or mute the thread https://github.com/notifications/unsubscribe-auth/ALnhnFJd31WacyBRDVPe26lsqxkMncntks5uPCsmgaJpZM4VzySX .

trmrsh commented 6 years ago

No, no idea. What concerns me is whether we noticed it at the time, or could it just be flushing some sort of dodgy buffer at the end of the run? Anyway, probably should not discuss under this thread any more.

HiPERCAM / hipercam