Can't grab or plot first frame in a run

[vikdhillon]

I think this is a known error, but it is really quite urgent to fix: I was doing dark frames today and had to wait until the second long-exposure frame before I could look at it.

[vikdhillon]

Sorry, I wasn't very accurate in my previous post. The issue is that you can't grab or rtplot the latest frame in a run. When the latest frame is >1, then no error is output to the screen, but the frame isn't plotted/grabbed. When the latest frame =0 or =1, then an error is output:

hipercam{observer}12: rtplot \ Traceback (most recent call last): File "/home/observer/.local/bin/rtplot", line 11, in load_entry_point('hipercam==0.7.dev7+g994a2b5', 'console_scripts', 'rtplot')() File "/home/observer/.local/lib/python3.5/site-packages/hipercam/scripts/rtplot.py", line 452, in rtplot for mccd in spool: File "/home/observer/.local/lib/python3.5/site-packages/hipercam/spooler.py", line 265, in next return self._iter.next() File "/home/observer/.local/lib/python3.5/site-packages/hipercam/hcam.py", line 802, in next return self.call() File "/home/observer/.local/lib/python3.5/site-packages/hipercam/hcam.py", line 937, in call seconds, nanoseconds, nsats, synced = decode_timing_bytes(tbytes) File "/home/observer/.local/lib/python3.5/site-packages/hipercam/hcam.py", line 1374, in decode_timing_bytes tbytes[-36:-2]))) TypeError: a bytes-like object is required, not 'str'

[trmrsh]

hi vik, sorry only just seen this. Will take a look.

tom

[trmrsh]

This seems to work with a local file, so I suspect an issue with the server. Is Stu around? I will see if I can add some diagnostic code.

[trmrsh]

Can I check something: when you say "When the latest frame is >1, then no error is output to the screen,", so you mean that you manually set "first" equal to what you know to be the last frame number? Is this while the run is going or stopped? Are you sure that you are not trying to read one more frame than has actually been taken up to that point? Finally, is this with a full frame format or a rather small window format? There is a difficulty knowing exactly what the last frame is I think, and I can't remember if Stu found a work-around.

[vikdhillon]

Hi Tom,

If I put first=0, then I get that error I reported, and it only disappears when the second frame has been written. If I put first=1, then I don't get the error and it carries on waiting (I set twait to a large number to test this) until the second frame appears, which it then plots.

This is all whilst the run is going. If I take a run with only 1 frame, then stop it, rtplot works fine.

This is all with a full frame format.

Hope this helps. Any other tests I can try to help debug this?

Cheers,

Vik.

---

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On 7 April 2018 at 15:47, Tom notifications@github.com wrote:

Can I check something: when you say "When the latest frame is >1, then no error is output to the screen,", so you mean that you manually set "first" equal to what you know to be the last frame number? Is this while the run is going or stopped? Are you sure that you are not trying to read one more frame than has actually been taken up to that point? Finally, is this with a full frame format or a rather small window format? There is a difficulty knowing exactly what the last frame is I think, and I can't remember if Stu found a work-around.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/HiPERCAM/hipercam/issues/30#issuecomment-379474647, or mute the thread https://github.com/notifications/unsubscribe-auth/ALnhnMm9E0Bd8dZz5G1QHnENw_GzZ8rAks5tmNGQgaJpZM4TLEQ7 .

[vikdhillon]

Hi Tom,

Sorry, probably didn't properly answer your exact question:

Can I check something: when you say "When the latest frame is >1, then no error is output to the screen,", so you mean that you manually set "first" equal to what you know to be the last frame number?

See my earlier email about this. I tried setting first to 0 and also to 1 (which was the last completed frame).

Is this while the run is going or stopped?

Again - see my earlier email: this was with the run going. When stopped, all seems ok, even if only 1 frame has been taken in the run.

Are you sure that you are not trying to read one more frame than has actually been taken up to that point?

No. It only ever seems to be able to plot the last but one frame.

Finally, is this with a full frame format or a rather small window format? There is a difficulty knowing exactly what the last frame is I think, and I can't remember if Stu found a work-around.

I think Stu had identified the problem (is it something to do with the timestamp? Might be completely misremembering here.) I also vaguely remember him saying it might be a fileserver issue rather than a pipeline one. Stu - I'm copying you in on this in case you don't get these emails...

Cheers,

Vik.

—

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/HiPERCAM/hipercam/issues/30#issuecomment-379474647, or mute the thread https://github.com/notifications/unsubscribe-auth/ALnhnMm9E0Bd8dZz5G1QHnENw_GzZ8rAks5tmNGQgaJpZM4TLEQ7 .

[trmrsh]

hi Vik, is Stu available? I really think this might be a fileserver issue. If you set first=0, it uses a special action "get_last". I am wondering if this is doing the right thing. I have found that with a static file on my laptop. when I set first=0. it comes back with the frame before last. i.e. on a 78 frame file, it returns frame 77 whereas my direct read from the disk returns correctly with frame 78. I can imagine that if it does the same at the start of the run, you will need 2 frames before it plots anything.

tom

[trmrsh]

what I don't understand is the behaviour with first=1, but I still think this is a fileserver issue as well, because obviously it should return with something.

tom

[vikdhillon]

Hi Tom, Not sure if Stu is around. No worries - I'm flying back tomorrow, then back our here again (with Stu) on Thursday, so we can look at it then. Yes, you're right about the 2 frames thing - rtplot won't plot anything until frame 3 has started, at which point it plots frame 1. Cheers, Vik.

---

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On 7 April 2018 at 16:55, Tom notifications@github.com wrote:

hi Vik, is Stu available? I really think this might be a fileserver issue. If you set first=0, it uses a special action "get_last". I am wondering if this is doing the right thing. I have found that with a static file on my laptop. when I set first=0. it comes back with the frame before last. i.e. on a 78 frame file, it returns frame 77 whereas my direct read from the disk returns correctly with frame 78. I can imagine that if it does the same at the start of the run, you will need 2 frames before it plots anything.

tom

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/HiPERCAM/hipercam/issues/30#issuecomment-379479452, or mute the thread https://github.com/notifications/unsubscribe-auth/ALnhnD8_5Rx5Qazy9wAKPFtmQN3cStlQks5tmOFqgaJpZM4TLEQ7 .

[vikdhillon]

Thanks for looking at this. I need to close things down now. When do you leave for Chile?

---

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On 7 April 2018 at 16:58, Vik Dhillon vik.dhillon@sheffield.ac.uk wrote:

Hi Tom, Not sure if Stu is around. No worries - I'm flying back tomorrow, then back our here again (with Stu) on Thursday, so we can look at it then. Yes, you're right about the 2 frames thing - rtplot won't plot anything until frame 3 has started, at which point it plots frame 1. Cheers, Vik.

---

Prof. Vik Dhillon, Dept of Physics & Astronomy, Univ of Sheffield, Sheffield S3 7RH, UK +44 114 222 4528; www.vikdhillon.staff.shef.ac.uk

On 7 April 2018 at 16:55, Tom notifications@github.com wrote:

hi Vik, is Stu available? I really think this might be a fileserver issue. If you set first=0, it uses a special action "get_last". I am wondering if this is doing the right thing. I have found that with a static file on my laptop. when I set first=0. it comes back with the frame before last. i.e. on a 78 frame file, it returns frame 77 whereas my direct read from the disk returns correctly with frame 78. I can imagine that if it does the same at the start of the run, you will need 2 frames before it plots anything.

tom

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/HiPERCAM/hipercam/issues/30#issuecomment-379479452, or mute the thread https://github.com/notifications/unsubscribe-auth/ALnhnD8_5Rx5Qazy9wAKPFtmQN3cStlQks5tmOFqgaJpZM4TLEQ7 .

[trmrsh]

tomorrow afternoon

[StuartLittlefair]

Hi All,

Sorry to miss all this. It is definitely related to the fileserver. See the issue that we opened in hcam drivers about this a couple of months ago. I had a look at it then and couldn’t work out how to fix it.

@trmrsh perhaps you could look at that issue and see if you have any bright ideas? The issue is that when a run is underway we have no way of knowing the current frame other than the file size and I don’t understand how the file size grows.

https://github.com/HiPERCAM/hcam-drivers/issues/101

Perhaps you want to close this issue, since it’s not really a pipeline problem.

[trmrsh]

Closing this issue, in favour of continuing the discussion under hcam-drivers as it is a fileserver issue.

[trmrsh]

now really closing it ...