search

open-ephys / GUI

Archived source code for the Open Ephys GUI
https://github.com/open-ephys/plugin-GUI
126 stars 282 forks source link

Issue with cables #170

Closed nikolaskaralis closed 10 years ago

nikolaskaralis commented 10 years ago

I am facing a very complicated issue with the cables that took me a while to figure out.

I am in possesion of 3 SPI cables. 1 (3ft blue and thicker) and two UltraThin (3ft), purchased from Intan.

For quite some time I have only been using the blue cable and had noticed that bank C of OpenEphys was not working, thinking it is somehow broken.

However, testing with the other cables, I have observed the following:

Any ideas?

aacuevas commented 10 years ago

It's weird indeed. Which version of the hardware are you using? (it's written under the SPI connectors) The software, is it the latest code?

Could you please feed a sinusoid to the headstage, record a couple of seconds of it with different banks and combinations of banks, like the ones you described, and send us the recordings?

jvoigts commented 10 years ago

The fact that the daisy chaining seems to at least mitigate the issue on bank c makes me wonder if its somehow related to the automatic delay compensation. The API scans a bunch of different in>output clock delays in an attempt to compensate for propagation delays in long cables. Maybe this process is messed up due to signal degradation or reflections.

There's at least two other immediate things to check: see if you can find a difference between the cables - maybe the blue cable has a problem that causes higher resistance on one conductor? Also we should check what the delay compensation algo actually does in all the cases you described. I'll check if it prints debug output later today, if not we should just add a few prints that indicate the chosen delay for each connected headstage.

Oh, and maybe also check if the supply voltage at the headstage is still high enough, or at least if its different in cases where you see problems. If you have a board with one regulator for all headstages it may be possible that you're just a tad under the required voltage and maybe that's connector specific.

nikolaskaralis commented 10 years ago

Dear Aaron and Jakob, thanks for the responses.

Some more info about the problem.

Hardware: Acquisition Board v2.1 revision 3 (May 2013) Software: Latest Windows .exe build from website.

I measured the voltage on the headstage between VCC and ground. I noticed it is different when it is not selected in the software (by clicking Rescan in the Rhythm FPGA module), so I am reporting both values.

Voltage measurements

UltraThin SPI All banks 3.45V (not selected) 3.43V (selected)

Blue All banks 3.47V (not selected) 3.46V (selected)

Two UltraThin daisy-chained All banks 3.43V (not selected) 3.39V (selected)

Additionally, I measured the resistance of the cables. Resistance Blue All conductors are 3Ω except for two edge ones that are 1.5Ω

Ultra Thin All conductors are 8.5Ω except for two edge ones that are 2.5Ω

Finally I recorded using the various combinations, while passing a 10Hz sinusoid (Recording for 20 sec) The recorded data can be found here: https://www.dropbox.com/sh/uaijcbgqsnwsmm8/AADWQGnykbsoT0TlNa7hi7bQa?dl=0

I summarize the results here.

Blue cable Bank A: OK Bank B: OK Bank C: Artifacts/Noise Bank D: OK

Ultra Thin Bank A: OK Bank B: OK Bank C: OK Bank D: OK

Blue + UltraThin Bank A : OK Bank B : Lower amplitude Bank C: Artifacts/Noise Bank D: Artifacts/Noise

antsiro commented 10 years ago

Hi All, Thanks for all the test, Nikolas. IMHO it all points to delay compensation issue on specific banks. I think Reid should step in for this discussion. And debug mode for these low level issues would be very important indeed, as people start using different cables, daisy chaining, making custom cables and as cables/connectors start wearing off or oxidize. We need some calibration/diagnostic procedure that could test for all this.

jsiegle commented 10 years ago

I just let Reid know about it.

jvoigts commented 10 years ago

Hi Nikolas, thx for the thorough testing! I just enabled debug output, if you have the time, it would be interesting to see what delays the method picks, maybe just ultrathin on C vs. regular on C vs both on C.

Also, just as a almost free test - see if reid's software has the same issue, just to exclude that we subtly messed up the code when we ported it over. Pretty unlikely but very quick to verify.

reidharrison commented 10 years ago

Nikolas, (This is Reid Harrison with Intan.) I've never heard of anyone having this type of problem before. Please try running your system with our Intan GUI (Windows .exe is available on our downloads page), and see if you have the same problem. After you switch cables around, go to the Configure tab and click "Rescan Ports A-D". If you still see bad data from any of the ports, click the "Manual" button and and try setting manual delays for the troublesome ports that are +/-1 delay step from the automatic values.

Also: The ultrathin cables are 6 ft. in length, right? We don't sell 3 ft. versions of these cables.

What type of headstage are you using?

nikolaskaralis commented 10 years ago

Dear all, I managed to get my hands on another OpenEphys board as well as another blue cable. I replicated exactly the same behavior with the new board/cable, which rules out specific issues with my board/cable.

I figured out one "solution" which might hint towards the actual problem. If I reduce the sampling rate to 20 kS/s (in the OpenEphys software), things work fine both on port C with the blue cable and on all the ports with daisy-chaining.

Using the Intan GUI, things seem to work fine in all the cases. In this software I notice that the automatic delay compensation is dependent on the sampling rate selected (and of course the number of daisy-chained cables). In particular, with blue cable alone it is 3ms @ 30kS/s and 2ms @ 20kS/s. With two cables (blue and pink or two pink), it is 4ms @ 20kS/s and 5ms @ 30kS/s. If in the Intan GUI I change manually the delay, I mess up the signals recorded (as expected).

All in all, it seems there is some bug related to the delay compensation which seems to be differentially implemented for the different ports.

@Reid : I am using the 32ch with accelerometer. Indeed the cables are 6ft. @Jakob: Thanks for enabling the debug output, I will try to compile it and post here the output.

antsiro commented 10 years ago

Dear Reid, Josh and Jakob, could you point to some documentation, if you have it, on how you implement impedance balancing on cables, deal with reflections etc? I.e. for the end user these factors : hardware (on the cable), software (FPGA firmware), adaptive algorithm (based on actual data etc), sampling rate/ channel count effect etc are important to understand when they deal with your system. Some sort of white page documentation on this would be very desirable. E.g. Would I be able to make my own cable with twisted pairs (longer than 6ft) and pass it via the commutator and be able to use your system without problems? how many channels and what sampling rate (ie bandwidth) would such regular cable allow?

reidharrison commented 10 years ago

The issues of impedance balancing and reflections are dealt with in hardware: in the design of the cables, and in the use of 100-ohm terminations. The only issue left for software to deal with is the round-trip signal delay that depends on the length of the cable. The Intan Rhythm FPGA interface samples the digital signals from each cable at 16 different delays. When the Intan GUI starts, it tries reading known ROM registers from the chips on each SPI port using all 16 possible delays. For each SPI port, it selects a delay that returned correct, uncorrupted values for the ROM registers. Then if the user changes the sampling rate, it performs a simple calculation to estimate the new optimum delay.

It sounds like the Intan GUI works fine in this respect. In my all experience, our system automatically adapts to any cable length out to several meters. The only reason I have ever had to tweak the delays manually by +/-1 is when using the RHD2164 64-channel chips, which are particularly sensitive to SPI signal timing due to their use of double-data-rate SPI.

I wonder if perhaps the Open Ephys software is not measuring optimum delays for each SPI port? Does it assume that all the cables will be the same length? Or maybe there is a bug that has that effect. Or maybe it is not re-calculating the new optimum delay for different sample rates correctly.

If you want to see how the Intan GUI finds optimum cable delays, look in mainwindow.cpp at the function findConnectedAmplifiers().

nikolaskaralis commented 10 years ago

I built the GUI with the debug options. I ran 4 tests and saved the debug output. For all the tests, I performed the following: Add FPGA module ---> Add LFP Viewer Module --> Start Acquisition (default : 30kS/s) --> Stop Acquisition --> Set Sampling Rate (20kS/s) --> Start Acquisition --> Stop Acquisition

I did this for ports A and C and using : blue cable only and two daisy-chained ultrathin cables. In both cases, port A works fine, port C works only for 20kS/s.

The output can be found here: https://www.dropbox.com/sh/wmok0ffuzbzcb1a/AACcngn6sBtzyP6aWu2W3K0Ga?dl=0

jvoigts commented 10 years ago

So it looks like the delays are set to the same value for the blue cable, but slightly different for the daisy chained cables? Thats at least a bit inconsistent. Could you compare the delays that we get on the open ephys GUI with the ones knows to work from the intan GUI?

If we can't figure this out quickly and this is a issue holding you back we can just add a UI option for manually setting the delays. @aacuevas or @jsiegle : do you have downtime to look at the code and check for obvious bugs? I'll be stuck in paper writing mode for a bit longer.

jvoigts commented 10 years ago

@antsiro that's a good idea - i'll see if i can link to some resources on the wiki. I believe that this is a bug rather than a hard limit on cable length though. As Reid mentioned, multiple meters of daisy chained cables should work just fine - we have tested ~5-6m here with full sample rate, including one very thin wire tether (https://open-ephys.atlassian.net/wiki/display/OEW/Fine+wire+tether). As for commutators, a few labs have run promising tests with very cheap commutators so far, we have a bit of info on here: https://open-ephys.atlassian.net/wiki/display/OEW/Commutators . I believe that fee lab is working on this as well - might be worth pinging them to see what they've come up with so far.

aacuevas commented 10 years ago

I've found a possible cause for this and uploaded a patch. Please check if this fixes the issue.

nikolaskaralis commented 10 years ago

Hey Aaron. Thanks for the patch. This seems to fix the issue. I tried both with two and three daisy-chained cables on all ports and it seems to work fine.