Buffer underrun

[cdeprez]

Hello,

I am a new user of Cicero and I am in trouble with a buffer problem and I can't find the way to fix. I used a NI PXIE 6535 card for digital output and each time I run my sequence, it fails due to buffer underrun.

Please find below the exact error message. Do you have any idea ?

I am using a FPGA as external time reference with exactly the frequency 10MHz which is the upper limit for external clock of my digital card. COuld it be the origin of my problem ?

Thanks for your answer

ERROR MESSAGE :

09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Received settings. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Received sequence. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Stopping and cleaning up old tasks. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Generating buffers. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Creating Variable Timebase Task on fpga device 12360003WE. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: ...Done (10 segments total) 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Generating buffer for Devkiwi 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Buffer for Devkiwi generated. 8004 samples per channel. On board buffer size 4095 samples per channel. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Generating buffer for Devbanane 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Buffer for Devbanane generated. 8004 samples per channel. On board buffer size 16384 samples per channel. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Buffers generated succesfully. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Arming tasks 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Using FPGA task on device 12360003WE as software clock source. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: 2 tasks armed. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Generating triggers. 09/05/2016 14:48:26 AtticusServer.AtticusServerCommunicator: Triggers generated. Sequence running. 09/05/2016 14:48:34 AtticusServer.AtticusServerCommunicator: daqMx Task on device Devbanane finished. 09/05/2016 14:48:35 AtticusServer.AtticusServerCommunicator: Client thinks the run is finished. Buffer statuses: 09/05/2016 14:48:35 AtticusServer.AtticusServerCommunicator: Devbanane 7980/8004 09/05/2016 14:48:35 AtticusServer.AtticusServerCommunicator: * Expected generation of 8004 samples for this task. 09/05/2016 14:48:35 AtticusServer.AtticusServerCommunicator: Devkiwi 8002/8004 09/05/2016 14:48:35 AtticusServer.AtticusServerCommunicator: At least 1 daqMx task finished at least 1 second before the client notified the server that it thought the run was finished. This may indicate corruption of the timing signal driving this daqMx task. Reporting to client as error.** 09/05/2016 14:48:35 AtticusServer.AtticusServerCommunicator: ***At least 1 daqMx task generated a different number of samples from the expected number. Reporting to client as an error.

[atsommer]

There could be a physical problem with the timing signal from the FPGA, causing some triggers to be missed. If the signal is being carried by a 50 Ohm coaxial cable, you might benefit from adding a 50 Ohm terminator at the end of the cable in parallel with the digital card to provide impedance matching and reduce reflections.

[cdeprez]

Thanks for your answer. I will try to fix it. I am just worried about the fact that the card with which I have problems of sample missing is also the card directly connected to the FPGA (using a NI CB 2162) and which them provides the external clock to the other cards. I would have expected this card to be less affected than the others

[ryanolf]

Is it always the same number of samples that are under run? (24 it looks like.)

On Tue, May 10, 2016 at 6:21 AM cdeprez notifications@github.com wrote:

Thanks for your answer. I will try to fix it. I am just worried about the fact that the card with which I have problems of sample missing is also the card directly connected to the FPGA (using a NI CB 2162) and which them provides the external clock to the other cards. I would have expected this card to be less affected than the others

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/akeshet/Cicero-Word-Generator/issues/24#issuecomment-218155046

[cdeprez]

No, it's not always the same number. Typically 1 to 20 samples can be missing. And the number can change for different run for the same sequence

[ryanolf]

Does sound like termination issues. Triggers come from the FPGA back to your cards. It doesn't matter which card generates the clock. On Tue, May 10, 2016 at 12:32 PM cdeprez notifications@github.com wrote:

No, it's not always the same number. Typically 1 to 20 samples can be missing. And the number can change for different run for the same sequence

— You are receiving this because you commented.

Reply to this email directly or view it on GitHub https://github.com/akeshet/Cicero-Word-Generator/issues/24#issuecomment-218265391

[cdeprez]

After looking to the problem with more attention, it really seems to be a physical issue as the average number of sample missing is changing when I restart the card terminal or when I unplug/plug my card. I will try to fix it. Thank you very much for your answers

[camacazio]

One possibility could be to implement a Timing Constraint in the VHDL; it is possible that the FPGA timing is not working as desired, and I found that a 15 ns clock period timing constraint sufficiently generated a configuration file that would constraint the timing needs within specification. Potentially an avenue to consider if proper termination doesn't seem to help.

On Wed, May 11, 2016 at 4:29 AM, cdeprez notifications@github.com wrote:

After looking to the problem with more attention, it really seems to be a physical issue as the average number of sample missing is changing when I restart the card terminal or when I unplug/plug my card. I will try to fix it. Thank you very much for your answers

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/akeshet/Cicero-Word-Generator/issues/24#issuecomment-218432872

[cdeprez]

Hi everyone. Thank you for your suggestion. It was a hardware problem in fact because I plugged unplugged my digital card I managed to solve (its appears from times to times and reconnecting the card manage to solve the problem).

Now I have another problem of buffer underrun. For personnal purposes I need to use a modify version of Cicero v1.28 which allow me to control DAC to generate analog waveform (rather than using analog card directly). Unfortunately, it appears that my code work well theoretically but when I try to generetae the buffer I received a buffer underrun message. This problem appears only when the size of the buffer is more than 2047 samples which is obviously not a random number !!!

I first thought that it was due to a on board buffer size limitation of the card but this problem does not occured using Cicero lastest version. So I wanted to know if you have any idea of where this problem comes from. May be I have miss something to change in the code or I need to implement some additionnal features available in more recent version of Cicero.

[camacazio]

My assumption had been that cicero is able to go far beyond the max buffer size for the DAC cards by pushing new waveform data mid-experiment onto the buffer, not sure if that is true or not, is it? Is your implementation to directly fill the buffer at the start of the experiment?

On Tue, Jun 7, 2016 at 2:14 AM, cdeprez notifications@github.com wrote:

Hi everyone. Thank you for your suggestion. It was a hardware problem in fact because I plugged unplugged my digital card I managed to solve (its appears from times to times and reconnecting the card manage to solve the problem).

Now I have another problem of buffer underrun. For personnal purposes I need to use a modify version of Cicero v1.28 which allow me to control DAC to generate analog waveform (rather than using analog card directly). Unfortunately, it appears that my code work well theoretically but when I try to generetae the buffer I received a buffer underrun message. This problem appears only when the size of the buffer is more than 2047 samples which is obviously not a random number !!!

I first thought that it was due to a on board buffer size limitation of the card but this problem does not occured using Cicero lastest version. So I wanted to know if you have any idea of where this problem comes from. May be I have miss something to change in the code or I need to implement some additionnal features available in more recent version of Cicero.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/akeshet/Cicero-Word-Generator/issues/24#issuecomment-224224483, or mute the thread https://github.com/notifications/unsubscribe/AIGlgu7Cp1Yun9k8jfflSuKcj_E4I_GIks5qJTaAgaJpZM4IaHqn .

[cdeprez]

Maybe my explanations were not cleared, I have no porblem with the DAC buffer or the card to generate the buffer for all the digital channels for the DAC ( and yes the whole buffer is calculated before the sequence start then send to my digital card to generate the waveform using DAC).

Actually, I have problem with another analog card I use in the same sequence which seems to be stuck at 2048 samples which create a buffer underrun and then stop the all sequence. Typically, I receive the message "at least 1 daqMx task generated a different number of samples from the expected number AnalogCard 2047/8004"

[cdeprez]

As the problem does not appear using the same card (but without changes for DAC) in the last version of Cicero, I think I've missed something in the code but I can't find it ...

[akeshet]

Yes, Cicero can certainly generate sequences that are larger than the on-board buffer of the card. All of the details of streaming to the card's buffer to refresh it are taken care of by the NI Daqmx drivers anyway. All Cicero knows is that it pushed a large buffer to those drivers.

There are various options you can set in Atticus that tell the drivers what sort of data refill strategy to try to use on the card. Sounds to me like the one that it is using is too conservative, it is waiting for the card's on-board buffer to be completely depleted before streaming the next data over, but then it is unable to stream it in time once it reaches that point. I suggest playing with those options. I talk about some of them in the user manual. http://akeshet.github.io/Cicero-Word-Generator/Cicero%20Technical%20and%20User%20Manual.pdf look for the section "Tweaking buffer performance"

[cdeprez]

I'm already using this option but I still have this problem. Also, daqmx task generation for my digital card often finish more than a second before the real end of the sequence. I probably still have a hardware issue

[cdeprez]

Clearly my analog card is stuck at 2047 samples whereas my digital card is stuck at 4094 samples.

Using exactly the same options, with the lastest version of Cicero, I have no problem for generating very large buffer, sending them to the card and running the sequence. Did you had some special features since v1.28 regarding refilling of the buffer ?

Could it be an assembly or driver problem ? I sometimes receive warning compiling Atticus saying that the reference assemblies for NI drivers target a different processor than the application's one. (I'm not quite familiar with C# and I don't really understand what it mean...)

I'm generating special variabletimebasesegments for my specific using of Cicero (controlling DAC ) subdivising the initial analog timestep, it might also be the origin of the problem. What do you think ?

Precise error message is the following :

08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Received settings. 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Received sequence. 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Stopping and cleaning up old tasks. 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Generating buffers. 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Creating Variable Timebase Task on fpga device 12360003WE. 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: ...Done (5004 segments total) 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Generating buffer for Devkiwi 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Buffer for Devkiwi generated. 5004 samples per channel. On board buffer size 8191 samples per channel. 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Skipped buffer generation for Devbanane. No used channels. 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Buffers generated succesfully. 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Arming tasks 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: 1 tasks armed. 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Generating triggers. 08/06/2016 17:21:56 AtticusServer.AtticusServerRuntime: Triggers generated. Sequence running. 08/06/2016 17:21:57 AtticusServer.AtticusServerRuntime: Client thinks the run is finished. Buffer statuses: 08/06/2016 17:21:57 AtticusServer.AtticusServerRuntime: Devkiwi 2047/5004 08/06/2016 17:21:57 AtticusServer.AtticusServerRuntime: * Expected generation of 5004 samples for this task. 08/06/2016 17:21:57 AtticusServer.AtticusServerRuntime: *At least 1 daqMx task generated a different number of samples from the expected number. Reporting to client as an error.

Doing exactly the same thing with released Cicero v1.64 does not cause any problem

[cdeprez]

Ok I think I have found the origin of the bug. When I run a sequence the number of segments generated is exactly the number of samples ... which that I did a fundamental mistake in my understanding of the code and in my code changes. I will try to fix it. I suppose that in Cicero v1.28 original version this issue didn't exist, did it ? (I can't check by myself because old Atticus version seems not to find my FPGA device)