PHD Crash during guiding

GoogleCodeExporter commented 9 years ago

Camera STT-8300M with FW8G-STT
Driving it through SGPpro and SGPApi. 
Connected to a Takhashi NJP mount via Tak ASCOM
Pulse guiding

Versions
PHD r1288 for windows (8)
STT driver 4.81 build 1
SGP v2.3.15.2568 

Have not been able to track it down exactly. I successfully guide for a period 
of time and either during a new Target sequence or at random times PHD crashes. 
Latest log entry:

22:57:27.818 09.008 2728 Alert: Camera timeout during capure
22:57:27.818 00.000 3148 Error thrown from worker_thread.cpp:149->Capture failed
22:57:27.818 00.000 3148 worker thread done servicing request
22:57:27.838 00.020 2728 Resizing image to 827,623
22:57:28.018 00.180 2728 Processing an image
22:57:28.018 00.000 2728 StopCapturing CaptureActive=1 continueCapturing=1 
exposurePending=0
22:57:28.018 00.000 2728 Status Line 0: Waiting for devices...
22:57:28.019 00.001 2728 Changing from state 6 to 7
22:57:28.019 00.000 2728 Guider::SetState: clearing mount guide algorithm 
history
22:57:28.020 00.001 2728 UpdateImageDisplay: Size=(648,488) min=992, max=14396, 
FiltMin=994, FiltMax=11889
22:57:28.039 00.019 2728 Resizing image to 827,623
22:57:28.118 00.079 2728 setting force full frames = true
22:57:28.118 00.000 2728 Status Line 0: Stopped.
22:57:28.120 00.002 2728 Changing from state 2 to 0
22:57:28.151 00.031 2728 Resizing image to 827,623
22:57:28.255 00.104 2728 Status Line 0: Stopped.
22:57:28.255 00.000 2728 OnExposeComplete(): Capture Error reported
22:57:28.255 00.000 2728 Status Line 2: 
22:57:28.256 00.001 2728 Error thrown from myframe_events.cpp:240->Error 
reported capturing image
22:57:28.294 00.038 2728 Resizing image to 827,623

Attaching a debug log

I crashed frequently for the last few days.  The crash was also with r1268 and 
r1246. 
I put all logs from 11/13 to 11/15 below
 https://drive.google.com/?authuser=0#folders/0B91c8lY19utvZlB3dFhVSWk0clE

Original issue reported on code.google.com by go...@msn.com on 16 Nov 2014 at 10:42

Attachments:

PHD2_DebugLog_2014-11-15_225535.txt

GoogleCodeExporter commented 9 years ago

gorgi,

Could you describe the symptom of the "crash"? Did you get an error message 
from Windows?

The camera timeout problem is a duplicate issue with what we discussing in this 
thread: https://groups.google.com/forum/#!topic/open-phd-guiding/zOEWtYEvbb8

The SGP developers are investigating why the SGP API Guider sometimes times-out 
returning a guide image to phd2.

2.4.0rc2 has a parameter on the Camera tab in the brain to allow you to 
increase the timeout value to avaoid phd2 disconnecting the SGP API Guider when 
it fails to respond. You should increase the timeout value to something like 60 
seconds.

I would still like to investigate the crash, so any more detail you can provide 
would be helpful (like a screenshot of the windows dialog showing the crash).

Andy

Original comment by andy.gal...@gmail.com on 16 Nov 2014 at 11:59

Changed state: WaitingForInput

GoogleCodeExporter commented 9 years ago

gorgi,

One more thing: could you please reproduce this problem one more time (keep the 
timeout at 5 seconds), but this time with the SGP API Guider ASCOM Driver Log 
file enabled? There's a check-box to enable the log in the driver properties 
window, accessible from the Connect Equipment window.

If you get the timeout (whether or not you see a phd2 crash), please attach the 
SGP API Guider Log file to this issue.

Thanks,
Andy

Original comment by andy.gal...@gmail.com on 17 Nov 2014 at 5:59

Added labels: Priority-High, OpSys-Windows
Removed labels: Priority-Medium, OpSys-All

GoogleCodeExporter commented 9 years ago

Will do

Sent from my iPad

Original comment by go...@msn.com on 17 Nov 2014 at 6:08

GoogleCodeExporter commented 9 years ago

Andy,
Happened again. Before I had a chance to change the brain setting. I have a 
folder in google drive with screenshot, debug, guide logs from PHD, SGPApi log 
and windows crash reporting logs

I placed all the logs 
here https://drive.google.com/folderview?id=0B91c8lY19utvSWdhdnFNZGxQZXM&usp=sh
aring

g

-- 
George Stathakopoulos
Sent with Airmail

On November 16, 2014 at 10:08:01 PM, george Stathakopoulos (gorgi@msn.com) 
wrote:

Will do  

Sent from my iPad

Original comment by go...@msn.com on 17 Nov 2014 at 6:46

GoogleCodeExporter commented 9 years ago

Aha! That info pinpointed the issue.

Here is the tail end of the API Guider Log:

22:30:36 - ImageData Written
22:30:36 - Image Complete, Camera Idle, Total Time (ms): 10375
22:30:36 - Image Requested
    Exposure:   4
    Binning:    1
    Light:  True
    StartX: 0
    StartY: 0
    NumX:   648
    NumY:   488
22:30:41 - ImageData Written
22:30:41 - Image Complete, Camera Idle, Total Time (ms): 10341
22:31:41 - Caught exception in RequteGuideImageApi - Error while waiting for 
image!

And the stack backtrace <PHDNetFrameCrash.rtf> shows this:

Description: The process was terminated due to an unhandled exception.

Exception Info: System.Exception

Stack:

at ASCOM.SGP_API_Guider.Camera.RequestGuideImageApi(System.Object)

at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)

---

So it appears to be an issue in the SGP API Guider code.  I'm not sure why 
George is seeing the crash but Bill is not.

At his point we need to forward the issue to SGP.

Andy

Original comment by andy.gal...@gmail.com on 17 Nov 2014 at 6:56

GoogleCodeExporter commented 9 years ago

George, 
Do you happen to have the SGP log from comment #4?  I would like to see if 
there is some correlation between the exception that the API Guider has caught 
and SGP. 

Thanks,
Jared

Original comment by jwellman@gmail.com on 17 Nov 2014 at 4:09

GoogleCodeExporter commented 9 years ago

Hi Jared.
I put the log in the directory. The one you want is   
     sg_logfile_20141115171012.txt
The PHD crash was  on 11/16/2014 10:31:41 PM

I have kept all SGP, SGPAPI, PHD and Windows log files. So happy to send you 
more. I had multiple crashes the past 3 days.
I placed all  SGP and SGPAPI log files 
https://drive.google.com/folderview?id=0B91c8lY19utvcUFfXzVmeU9Ga1E&usp=sharing

-- 
George Stathakopoulos
Sent with Airmail

On November 17, 2014 at 8:09:39 AM, open-phd-guiding@googlecode.com 
(open-phd-guiding@googlecode.com) wrote:

Original comment by go...@msn.com on 18 Nov 2014 at 1:42

GoogleCodeExporter commented 9 years ago

Thanks George, 
Can you reliably reproduce this?  If so what steps are you taking to reproduce 
it?  

I have a couple of working theories as what the problem may be but I don't want 
to "pollute the well" for the questions above. 

Thanks,
Jared

Original comment by jwellman@gmail.com on 18 Nov 2014 at 4:21

GoogleCodeExporter commented 9 years ago

Unfortunately it is intermittent. The last crash was after a meridian flip. SGP 
was trying to restart guiding. From what I recall, some of the crashes were 
during similar situations.They also happened during imaging. I am happy to look 
for all the crash event, time stamps, and stack traces if it will be helpful.

PS The weather has turned full Northwest for the foreseeable future so testing 
will be a challenge.

Original comment by go...@msn.com on 18 Nov 2014 at 4:49

GoogleCodeExporter commented 9 years ago

I added 3 more crash events with logs 
https://drive.google.com/folderview?id=0B91c8lY19utvd2FYSk0tTkRxN3c&usp=sharing

On November 17, 2014 at 8:49:31 PM, george Stathakopoulos (gorgi@msn.com) wrote:

Unfortunately it is intermittent. The last crash was after a meridian flip. SGP 
was trying to restart guiding. From what I recall, some of the crashes were 
during similar situations.They also happened during imaging. I am happy to look 
for all the crash event, time stamps, and stack traces if it will be helpful.  

PS The weather has turned full Northwest for the foreseeable future so testing 
will be a challenge.

Original comment by go...@msn.com on 18 Nov 2014 at 5:31

GoogleCodeExporter commented 9 years ago

George,

Jared has released a new version of the SGP API Guider ASCOM driver.

Could you please update and confirm that it no longer crashes?

For the experiment it is probably best to keep the camera timeout value at the 
default setting (5 seconds) as this will be the most like route to see if the 
problem reproduces.

Once we are sure the crash is no longer happening, you can increase the timeout 
value to 40-60 seconds.

The increased timeout should only be needed for the SGP API Guider when the 
internal guide chip is in use. For the SBIG external guide head, the default 
setting should be ok.

Andy

Original comment by andy.gal...@gmail.com on 19 Nov 2014 at 5:20

GoogleCodeExporter commented 9 years ago

The good news the crash is not happening anymore with the 1.30 SGP API
The bad news is that PHD fails to resume after a slew (when SGP moves to a new 
target)

logs are here 
https://drive.google.com/folderview?id=0B91c8lY19utvZlB3dFhVSWk0clE&usp=sharing

In the SGPApil log I get 
23:36:05 - Image Complete, Camera Idle, Total Time (ms): 15396
23:37:00 - Caught exception in RequestGuideImageApi - Error while waiting for 
image!
23:37:05 - Caught exception in RequestGuideImageApi - Error while waiting for 
image!

And the SGPlogs 
[[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] DoEventGroupChange: Slewed 
or Centered, attempting to start guiding.
[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] PHD Advanced: Auto Resume
[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] PHDA AutoResume: Attempting 
to resume auto guiding try 0 of 3
[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] PHDA AutoResume: Stopping 
Auto Guider.
[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] PHD Advanced: Stop
[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_STOP)...
[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_GETSTATUS)...
[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:37:07 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA Advanced: Start Capture
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_GETSTATUS)...
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_PRELOOP)...
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOP)...
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA AutoResume: Start 
capture successful
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA AutoResume: Using loop 
frame count
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:37:09 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:09 PM] [DEBUG] [Unknown] SgApi - Starting Guider Capture
[12/1/2014 11:37:09 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera: 
CaptureImage...
[12/1/2014 11:37:09 PM] [DEBUG] [API Guider Capture Thread] SBIG  Guider 
Camera: Start exposure...
[12/1/2014 11:37:09 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera: 
exposure time = 400
[12/1/2014 11:37:09 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera: 
capture command sent...
[12/1/2014 11:37:10 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera 
status check: Unknown (8)
[12/1/2014 11:37:10 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera 
status check: Unknown (8)
[12/1/2014 11:37:10 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:37:10 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:11 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:37:11 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:12 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:37:12 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:13 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:37:13 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:13 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera 
status check: Unknown (8)
[12/1/2014 11:37:14 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera 
status check: Unknown (12)
[12/1/2014 11:37:14 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera: 
end exposure called...
[12/1/2014 11:37:14 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera: 
read data...
[12/1/2014 11:37:14 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera: 
starting readout...
[12/1/2014 11:37:14 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera 
status check: IDLE
[12/1/2014 11:37:14 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera: 
reading lines...
[12/1/2014 11:37:14 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera: 
end readout...
[12/1/2014 11:37:14 PM] [DEBUG] [API Guider Capture Thread] SgApi - Guider 
Capture Complete
[12/1/2014 11:37:14 PM] [DEBUG] [API Guider Capture Thread] SgApi - Guider 
Image Added to Repo
[12/1/2014 11:37:14 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:37:14 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:14 PM] [DEBUG] [API Guider Capture Thread] SBIG Guider Camera 
status check: IDLE
[12/1/2014 11:37:15 PM] [DEBUG] [Main Thread] PopulateDataModel:  Transferring 
view to the data model...
[12/1/2014 11:37:15 PM] [DEBUG] [MF Update Thread] Performing serialize...
[12/1/2014 11:37:15 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:37:15 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:16 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:37:16 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:17 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:37:17 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:37:18 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[.......... repeated every 1 second]
[12/1/2014 11:38:09 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA AutoResume: Safety hit 
while using loop frame count!
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA AutoResume: Attempting 
to resume auto guiding try 1 of 3
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA AutoResume: Stopping 
Auto Guider.
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHD Advanced: Stop
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_STOP)...
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_GETSTATUS)...
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA: Received (101)...
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHD Advanced: Stop
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_STOP)...
[12/1/2014 11:38:10 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:38:11 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_GETSTATUS)...
[12/1/2014 11:38:11 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:38:11 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOPFRAMECOUNT)...
[12/1/2014 11:38:11 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:38:13 PM] [DEBUG] [Sequence Thread] PHDA Advanced: Start Capture
[12/1/2014 11:38:13 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_GETSTATUS)...
[12/1/2014 11:38:13 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:38:13 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_PRELOOP)...
[12/1/2014 11:38:13 PM] [DEBUG] [Sequence Thread] PHDA: Received (0)...
[12/1/2014 11:38:13 PM] [DEBUG] [Sequence Thread] PHDA: Sent command 
(PHD_LOOP)...

Original comment by go...@msn.com on 2 Dec 2014 at 8:00

GoogleCodeExporter commented 9 years ago

Some more thoughts. I have tried to increase the PHD time-out to 5,10,15 
minutes but that did not help

It seems that the SBIG camera gets implicated somehow. 

One of the programs SGP or SGPAPI or SBIG drv or PHD believes that the camera 
gets disconnected, (there is an SBIG timeout error)
When the issue appears, PHD seems stopped. SGP goes to recovery mode. When I 
manually restart the sequence SGP. It takes images in order to plate solve. So 
the camera is still connected. . To get PHD working again I have to disconnect 
the camera (SGPAPI) then reconnect. It seems to work until SGP tries to resume 
the guider again and then i am back where I started.

Original comment by go...@msn.com on 2 Dec 2014 at 8:13

GoogleCodeExporter commented 9 years ago

Added 2 screenshoots of SBIG Errors dialogs that eventually came up

1. SBIG Guider Error while checking for image complete : NO_ERROR
2. SBIG Error (Timeout): Camera status is : INTEGRATING

Original comment by go...@msn.com on 2 Dec 2014 at 3:22

Attachments:

[Screenshot 2014-12-02 07.16.04.png](https://storage.googleapis.com/google-code-attachments/open-phd-guiding/issue-366/comment-14/Screenshot 2014-12-02 07.16.04.png)

GoogleCodeExporter commented 9 years ago

crash was not a phd2 bug, it was a crash in the sgp api guider ascom driver

Original comment by andy.gal...@gmail.com on 11 Dec 2014 at 8:31

Changed state: Invalid

geofstone / open-phd-guiding

PHD Crash during guiding #366