Fastino garbage output with long sequences

[marmeladapk]

Bug Report

One-Line Summary

Fastino generates garbage output with long transmission sequences.

Issue Details

Steps to Reproduce

1. Generate gateware from this json:

   {
   "target": "kasli",
   "variant": "test",
   "hw_rev": "v2.0",
   "base": "standalone",
   "rtio_frequency": 125e6,
   "peripherals": [
   {
       "type": "fastino",
       "hw_rev": "v1.2.1",
       "ports": [10]
   }
   ]
   }

2. Run this experiment:

   
   from artiq.experiment import *
   from numpy import sin, pi

class SystemExample(EnvExperiment): def build(self): self.setattr_device("core") self.setattr_device("core_dma") self.fastino = [self.get_device("fastino" + str(i)) for i in range(1)]

    # self.s = [1.2*sin(i/7) for i in range(32)]
    self.s = [self.fastino[0].voltage_to_mu(1.2*sin(2*pi*i/32)) for i in range(32)]
    print(self.s)

@kernel
def run(self):
    self.core.reset()
    self.init()
    delay(10 * ms)
    # This function never returns
    self.fastino_sine(amplitude=1.2*V)
    #self.record_waveform()
    #self.play_waveform()

@kernel
def init(self):
    for fastino in self.fastino:
        fastino.init()
    self.core.break_realtime()

@kernel
def fastino_sine(self, amplitude):
    i = 0
    while True:
        for j in range(8):
            self.fastino[0].set_dac_mu(j, self.s[i])
            delay(39200 * ns)
        i += 1
        i = i % 32
        delay(200*us)

@kernel
def record_waveform(self):
    with self.core_dma.record("fastino0"):
        for i in range(100):
            for j in range(32):
                self.fastino[0].set_dac(j, sin(i / 5)*1.5)
                delay(392*ns)
            delay(100*us)

@kernel
def play_waveform(self):
    dma_handle = self.core_dma.get_handle("fastino0")
    self.core.break_realtime()
    for _ in range(200):
        self.core_dma.playback_handle(dma_handle)

### Expected Behavior

Sine output on Fastino.

### Actual (undesired) Behavior

Malformed output.

![image](https://github.com/m-labs/artiq/assets/24512604/8e52df31-bb39-4a69-baf2-ac7844d22214)
![image](https://github.com/m-labs/artiq/assets/24512604/425a4c0c-3c1b-44de-ac49-34e0fb5459cf)
![image](https://github.com/m-labs/artiq/assets/24512604/a9355993-ed95-40a8-a23e-1af44b6352aa)
![image](https://github.com/m-labs/artiq/assets/24512604/d18ac6d9-6336-47ae-8c80-68d38fff3763)
![image](https://github.com/m-labs/artiq/assets/24512604/a5bd39b6-e5ae-45ed-ab90-435858f0492c)

The same behaviour happens with DMA and was tested with other Fastino and Kasli. This happens both in master and standalone variants. Notably, this happens after a long period of time when the signal is generated on <=4  channels, when generating signal on a single channel then the delay before garbage output is even longer. Sometimes the output corrects itself.

Error LED on Fastino does not light up.

### Your System (omit irrelevant parts)

* ARTIQ version: release-7, db79100; master baa5834, 3aaa7e0
* Version of the gateware and runtime loaded in the core device: same, respective for release-7 and master
* Hardware involved: Kasli v2.0.2, Fastino v1.2.1

<!--
For in-depth information on bug reporting, see:

http://www.chiark.greenend.org.uk/~sgtatham/bugs.html https://developer.mozilla.org/en-US/docs/Mozilla/QA/Bug_writing_guidelines
-->

[marmeladapk]

Ok, I tested this on 4th Fastino and this one is able to sustain such a sequence on all channels for an extended period of time (tens of minutes) before the output becomes garbage.

[nkrackow]

Hi, (un-)fortunately I was not able to reproduce this. I tried with two Fastino 1.2.1, different EEM ports on a Kasli-SoC and with/without log2_width = 5. The output was always a stable sine on all channels (only ch0 for log2_width = 5 ofc).

The last blue trace looks suspiciously like the runaway of a CIC if the poles and zeros didn't perfectly cancel out, which happens if you somehow get an extra bit into an accumulator.

Did the red LED (indicating communication problems with Kasli) on Fastino ever turn on?

[marmeladapk]

Red LED was off in all cases. It seems that it was the case of a quirky hw, maybe one that barely passed tests and only in longer sequences exhibited problems. I think we should wait for Fastino v1.3 and see if supply sequencing helps with intermittent issues like this.

[nkrackow]

While trying to reproduce your issue I also ran into some problems with the communication with Kasli. Sometimes I could turn on the red LED just by touching the negative output of a Fastino channel. Longer DMA sequences or your experiment never worked. It seems to have fixed itself after the solar storm.. :shrug: