New Spectacular Failure at Scope (Not really urgent anymore...)

[jmdelahanty]

Deryn attempted to image today and something quite severe happened in the code. Unsure what actually happened yet specifically, other than the apparently no configurations were generated, the Arduino was skipping through trials erratically, and things in general failed quite badly. When Deryn is done for the day I'll check what happened... will update

[jmdelahanty]

Deryn updated me with the following (just for documentation purposes so people can see what happened) in the following order:

1. It looks like the iti generation or tone generation is failing
2. Basically nothing is being generated for configs, so the arduino is running through everything as fast as possible and then crashing, with no response to end python or prairieview, which continues forever. This is an issue because you can't even see where things go wrong, because nothing is saved or even reported in the command line. This is also why I save a version of arduino files so you can see where issues are
3. Uncertainty if v5.5.64.500 is the correct Prairie View Version
4. figured out the behavior issue (not sure what was done to fix this yet...)
5. looks like the arduino code runs fine, but the python code never realizes that the imaging is over, so the trigger never happens
6. terminal never changes, stays at 0% (meaning video isn't being recorded at all, meaning experiment never ends)
7. Prairieview records voltage, but no config file is saved either; Yeah it looks like the trial is never ending, so nothing is being saved or written (no configuration written because all the values are only available at the end of the experiment, namely the "dropped_frames" list

Still need to go in and see what's wrong.

This was all done on the TyeLab account, which has been generally neglected for some time. However it is quite strange that things are failing so severely like this since no updates to the main branch have been done since early August. Deryn said things were working before she went to the workshop and very suddenly things were broken. I can't remember anything I did to alter things on that account since I only develop remotely or on my own account and specifically in different branches/develop branch, so it might take some digging to get to the bottom of how things went so wrong...

[jmdelahanty]

I haven't had time to dive down into what happened here. On my account with Austin's setup things are working well during testing today. It might be some time before I can actually get to the bottom of what happened here... it could be hard to find out.

[jmdelahanty]

Updated name since things are working okay for Austin and I'm unsure when/if Deryn will use this scope in the near future. Once she plans to/if she wants to then it'll be a high priority, but for now this will stay open to both document what happened and also so it's on the list of things to work on.

[jmdelahanty]

Austin reported to me today that he had a strange failure as well with things on the scope where, after things had gotten to the Arduino communicating things back and forth, all that was left was a message stating "Waiting for Input Trigger" on both the terminal and in Prairie View. This makes me think that possibly the issue that he ran into (and maybe Deryn as well) was due to something in the serial communication going incorrectly more than likely due to a wiring problem. The wiring at the scope for the Arduino stuff is not right and frankly should be a priority to get into tip top shape... I'll update here if more odd things happen...

[jmdelahanty]

Looking at Deryn's configuration file on the server briefly today revealed at least one likely piece of information that yielded the severe failure.

The minITI and maxITI values put in the file are:

• "minITI": 18000
• "maxITI": 26000

This likely led to the stuffing of packets into the ITIArray buffer to fail or at least behave strangely.

There's not currently any sanitizing of inputs from the configuration files, which there should be, but I haven't had time to actually implement it. I guess I should add an issue from this. It's hard to say if this is the reason it failed so badly, but it's likely a piece of it.

For record keeping, here's a copy of the values:

{"beh_metadata": {
    "totalNumberOfTrials": 10,
    "startingReward": 3,
    "maxSequentialReward": null,
    "maxSequentialPunish": 3,
    "punishTone": 2000,
    "rewardTone": 9000,
    "USDeliveryTime_Sucrose": 300,
    "USDeliveryTime_Air": 100,
    "USConsumptionTime_Sucrose": 0,
    "vacuum" : true,
    "ITIJitter": true,
    "baseITI": 22000,
    "minITI": 18000,
    "maxITI": 26000,
    "toneJitter": false,
    "baseTone": 3,
    "minTone": null,
    "maxTone": null,
    "catchTrials": false,
    "numCatchReward": 2,
    "numCatchPunish": 0,
    "catchOffset" : 0.25,
    "percentPunish": 0.30,
    "stim": false,
    "shutterOnly": false,
    "stimFrequency": 20,
    "stimPulseTime": 5,
    "stimLambda": 590,
    "stimDeliveryTime_PreCS": 1000,
    "stimDeliveryTime_Total": 5000,
    "stimStartPosition": 20,
    "numStimReward": 5,
    "numStimPunish": 5,
    "numStimAlone": 3,
    "yoked": false,
    "trialArray": {},
    "ITIArray": {},
    "toneArray": {},
    "LEDArray": {},
    "dropped_frames": {}
  },
  "zstack_metadata": {
    "zstack": false,
    "stack_number": 3,
    "zdelta": 25.0,
    "zstep": 1.0
  },
  "weight_check": false
}

Basically nothing is being generated for configs, so the arduino is running through everything as fast as possible and then crashing, with no response to end python or prairieview, which continues forever.

This is I'm guessing due to the Valence branch Arduino code. The configurations are generated far before the uploading/serial communication. Either the serial comms failed/copying transfer code was off base or something else happened. The configs not being written yet, for reasons explained above, has nothing to do with what configs are generated. There's possible things wrong with array generation, but I'm not sure it was something to do with how the library operates.

looks like the arduino code runs fine, but the python code never realizes that the imaging is over, so the trigger never happens

I'm not sure how to determine what this means since I never saw it. I'm guessing it's related to the next comment.

terminal never changes, stays at 0% (meaning video isn't being recorded at all, meaning experiment never ends)

This is more than likely because the number of images expected to be taken is enormous. The progress bar doesn't move much if at all because it would take a huge amount of time to get through all the scheduled images as a result of the config file values for ITI durations. I just tested this on a known working version of main and saw the same behavior. It was more than likely recording both microscopy and video data, but the terminal didn't make this obvious since it would have been sitting on 0% which must have been confusing.

Again, something else may have happened, but I can't reproduce this behavior in other ways with what's in place on main which is nearly the exact same as what was in place today.

As for the comment before this one, I'm unsure why it would do that still...

At the moment, I'm satisfied with this explanation for how things got so messed up on accident especially after testing it today. I need to make docs more clear about what values each config expects although it is defined here...

I'm going to close this for now until I see similar weird things happen again...

[jmdelahanty]

Related: Triggers failed again today. I think it's because the connection from the new frame out to the camera's block became loose from having to move the camera back and forth between sides.