nasa / CTF

This is a new repository for a new tool to be added to the cFS ecosystem called cFS Test Framework (CTF).
28 stars 6 forks source link

CTF hangs when regression testing against SP0 #26

Open dmknutsen opened 2 years ago

dmknutsen commented 2 years ago

CTF hangs intermittently when running the regression test against our SP0 at the following location:

[08:30:28.778] sp0_protocol (207) INFO: (CFS cfs_SP01) Reply from 192.168.1.102 bytes=64 ttl=64 seq=1 time=7ms [08:30:33.885] sp0_protocol (207) INFO: (CFS cfs_SP01) Reboot sp0 [08:30:33.987] sp0_protocol (207) *** INFO: (CFS cfs_SP01) Sending command reboot()

[08:30:34.087] sp0_protocol (207) *** INFO: (CFS cfs_SP01) No exit code found for command reboot()

[08:30:34.088] sp0_protocol (207) INFO: (CFS cfs_SP01) Resetting connection... [08:30:34.089] sp0_protocol (207) INFO: (CFS cfs_SP01) Disconnected from sp0 Console [08:30:39.097] sp0_protocol (207) INFO: (CFS cfs_SP01) Connection Attempts: 1 [08:30:39.100] sp0_protocol (207) INFO: (CFS cfs_SP01) Connect to sp0 for Console [08:30:39.102] sp0_protocol (207) INFO: (CFS cfs_SP01) Sorry, session limit reached. [08:30:39.102] sp0_protocol (207) INFO: (CFS cfs_SP01) Not Connected to sp0 Console [08:30:39.103] sp0_protocol (207) *** INFO: (CFS cfs_SP01) Resetting connection...

The error is not repeatable in the sense that if I re-execute the test that failed...it will run successfully. Attached are the log files from two test runs that demonstrate this behavior.

We have see a similar issue in the past - where if CTF is overloaded/receiving a large amount of messages...it will fail to send the reboot command. I suspect that may be what is happening here...but further investigation is required.