snoplus / orca

Git repository tracking the main Orca svn for SNO+ development
2 stars 13 forks source link

Issues saving ORCA at CROX #507

Open rcl11 opened 6 years ago

rcl11 commented 6 years ago

Obviously we've had occasional issues related to fluctuating connections from the overseas control rooms but I had some really weird ORCA behaviour today and Esther at least confirms it's something she hasn't seen before. I'd previously saved ORCA about 1 hour earlier, went to save again for handover and what happened was shown in the following printscreen:

screen shot 2018-05-17 at 07 57 07

Basically you can see the finder window in the background on the right showing the connection to the shared directory and ORCA attempting to save, but it never manages to update the snot_main.Orca file and the pop up warns you of this. BTW the ORCA log lies and tells you the save has been done (you can tell it hasn't from the timestamp on the snot_main.Orca file).

Any ideas what happened here?

rcl11 commented 6 years ago

Edit: from shift reports it seems this issue was also seen from CRAG, and it was seen at CRSU just now as well.

rcl11 commented 6 years ago

A bunch more info in this shift report: https://snopl.us/shift/view/d02f7ad9f6664c33928563ba6d7a23ba

Long story short - it appears the snot_main.Orca file is somehow not connecting correctly when mounting the shared drive. It is still possible to create and update other files in that same folder from remote, but not to modify/delete snot_main.Orca, or to see modifications made via ssh on buffer1 to snot_main.Orca.

jlidgard commented 6 years ago

seeing this a lot today (GY shift of 7th June)

when I connect the remote folder in finder, it connects ok. However, if I then click any other folder (e.g. downloads, desktop etc) and then click back to the remote folder, I get either the error screens hotted above in this git issue (i.e. could not be found) or another saying the location is 'not connected'.

The VPN is stable, the builder log is updating fine, the crug cam is streaming and the remote folder always connects ok. The remote folder just seems not to sustain its connection very well.

Seems the Mac is using samba version 2. (found this out by running 'smbutil statshares -a'). There is a version 3. Would upgrading help? (or something else other than smb be possible?)

stefannae commented 6 years ago

Today, 30 minutes into the shift and after a few successful saves during the switches from LASERBALL to TRANSITION and back, Orca started giving the "Orca file could not be saved." message. It is showing the same symptoms as above.

tannerbk commented 6 years ago

We've seen this running the detector in the CRUG several times. When we restart the run the message pops up and we have to hit "OK" to continue.

estherturner commented 6 years ago

Just to let people know, this issue is definitely still occurring: https://snopl.us/shift/view/613ea8f801e84b298075495a74dbead7