Add support for automated in-circuit verification

maehw commented 2 years ago

Wokwi is a great tool. It does not only allow to simulate digital designs and then run some magic to get your own ASIC design out of it. It also allows to write and simulate software on an embedded target such as an Arduino Nano or Arduino Mega.

The designs generated by wokwi-lookup-table-generator are pure combinatorial logic designs. One entry from the truth table (combination of input signals, ones and zeros) directly generates a specific combination of logic levels on the outputs (ones and zeros). This makes testing the designs generated by the generator quite easy:

The test stimuli can be used from the truth table which is an input for the generator
The expected outputs of the digital design can also be taken from the truth table

The idea for automated verification is currently experimented with here: https://wokwi.com/projects/342092183606657619

Basic concept:

Add an Arduino to the wokwi sketch with plenty of I/Os, best choice seems to be the Arduino Mega at the moment
In the sketch.ino: Configure input and output pins in the setup() routine using the pinMode() function
Also configure a serial interface by calling Serial.begin(); make sure that the RX and TX pins are not used as input or outputs, but stay reserved to control the inputs of your digital design/ read the outputs from it
Add support for serialMonitor in your Wokwi design
Set the inputs of your digital design using a series of calls to digitalWrite()
Read back the outputs of your digital design using a series of calls to digitalRead()
Verify that the measured output signals match the logic levels as definged by the output signals in the truth table
Generate outputs and a test report using the serial console (several calls to Serial.print())

This can all be automated.

I suggest to add a command line flag -t/--in-circuit-test to switch between the normal design generation and a verification design generation with the additional Arduino and the additional connections required for in-circuit verification.

maehw commented 2 years ago

Related to verification: https://github.com/wiltonburke/wokwi_tools

maehw commented 2 years ago

Let's break this down a little bit.

Adding the Arduino MEGA: add the following to the parts section in the diagram JSON:

   {
      "type": "wokwi-arduino-mega",
      "id": "mega",
      "top": <top_position>,
      "left": <left_position>,
      "rotate": 90,
      "attrs": {}
    },

Add the serial monitor in the diagram JSON, just in the same level as parts:

  "serialMonitor": {
    "display": "always",
    "newline": "lf"
  }

GPIO configuration can be static as long as we stay within 10 input and 10 output pins: Arduino Pins 2, 3, 4, 5, 6, 7, 8, 9, 10, 11: output pins to control the design inputs! Arduino Pins 12, 13, 14, 15, 16, 17, 18, 19, 20, 21: output pins to control the design inputs! (Not all pins must be used. More than 10 input or 10 output pins would need dynamic reconfiguration. Unlikely use case!)

Note that pins 1 and 0 are used for Serial (UART) and should not be used.
Pin LED_BUILTIN should be in the output pin range or be skipped.

We can define aliases for the pin numbers, e.g.:

#define DESIGN_IN_A   ( 2)
#define DESIGN_IN_B   ( 3)
#define DESIGN_IN_C   ( 4)
#define DESIGN_IN_D   ( 5)
#define DESIGN_IN_E   ( 6)
#define DESIGN_IN_F   ( 7)
#define DESIGN_IN_G   ( 8)
#define DESIGN_IN_H   ( 9)
#define DESIGN_IN_I   (10)
#define DESIGN_IN_J   (11)

#define DESIGN_OUT_A  (12)
#define DESIGN_OUT_B  (13)
#define DESIGN_OUT_C  (14)
#define DESIGN_OUT_D  (15)
#define DESIGN_OUT_E  (16)
#define DESIGN_OUT_F  (17)
#define DESIGN_OUT_G  (18)
#define DESIGN_OUT_H  (19)
#define DESIGN_OUT_I  (20)
#define DESIGN_OUT_J  (21)

Initialize the input/output direction of those GPIO pins:

 pinMode(DESIGN_IN_A, OUTPUT);
  pinMode(DESIGN_IN_B, OUTPUT);
  pinMode(DESIGN_IN_C, OUTPUT);
  pinMode(DESIGN_IN_D, OUTPUT);
  pinMode(DESIGN_IN_E, OUTPUT);
  pinMode(DESIGN_IN_F, OUTPUT);
  pinMode(DESIGN_IN_G, OUTPUT);
  pinMode(DESIGN_IN_H, OUTPUT);
  pinMode(DESIGN_IN_I, OUTPUT);
  pinMode(DESIGN_IN_J, OUTPUT);

  pinMode(DESIGN_OUT_A, INPUT);
  pinMode(DESIGN_OUT_B, INPUT);
  pinMode(DESIGN_OUT_C, INPUT);
  pinMode(DESIGN_OUT_D, INPUT);
  pinMode(DESIGN_OUT_E, INPUT);
  pinMode(DESIGN_OUT_F, INPUT);
  pinMode(DESIGN_OUT_G, INPUT);
  pinMode(DESIGN_OUT_H, INPUT);
  pinMode(DESIGN_OUT_I, INPUT);
  pinMode(DESIGN_OUT_J, INPUT);

Add code to iterate over range of input values.
Add code to read output values and put them together to a word.
Add code for verification: match the read output pins with expected outputs from the truth table.

To be continued...

maehw commented 2 years ago

... of course, don't forget to add the required connections to the diagram JSON file. Here's an example:

    [ "mega:2", "input_a:IN", "green", [ "h57.09", "v-97.5" ] ],
    [ "mega:3", "input_b:IN", "green", [ "h107.15", "v0.05" ] ],
    [ "mega:4", "input_c:IN", "green", [ "v2.15", "h-178.48", "v-191.68" ] ],
    [ "mega:5", "input_d:IN", "green", [ "h58.27", "v-0.84" ] ],
    [ "mega:6", "input_e:IN", "green", [ "h36.63", "v4.31" ] ],
    [ "mega:12", "output_A:OUT", "green", [ "h57.09", "*", "v-97.5" ] ],
    [ "mega:13", "output_B:OUT", "green", [ "h57.09", "*", "v-97.5" ] ],
    [ "mega:14", "output_C:OUT", "green", [ "h57.09", "*", "v-97.5" ] ],
    [ "mega:15", "output_D:OUT", "green", [ "h57.09", "*", "v-97.5" ] ],
    [ "mega:16", "output_E:OUT", "green", [ "h57.09", "*", "v-97.5" ] ],
    [ "mega:17", "output_F:OUT", "green", [ "h57.09", "*", "v-97.5" ] ],
    [ "mega:18", "output_G:OUT", "green", [ "h57.09", "*", "v-97.5" ] ]

maehw commented 2 years ago

The wokwi demo project has been updated with quite generic Arduino code - generic enough to use other designs, and also customizable enough. Should not be too much effort to let the design-specific parts be generated by wokwi-lookup-table-generator!

/* Let's start with design specific configuration */

#define DESIGN_NUM_USED_INPUTS      (5u)
#define DESIGN_NUM_USED_OUTPUTS     (7u)

/* Define whether the verification shall halt on the detection of the 
 * first error (true) or keep on running to the end (false).
 */
#define VERIFICATION_STOP_ON_ERROR  (true)

/* This table shall be generated from the truth table!
 * Obviously, also design specific.
 */
uint16_t expected_out_val[2 << (DESIGN_NUM_USED_INPUTS-1)] = {
    0b1111101, // '@'
    0b1110111, // 'A'
    0b0011111, // 'B'
    0b0001101, // 'C'
    0b0111101, // 'D'
    0b1001111, // 'E'
    0b1000111, // 'F'
    0b1011110, // 'G'
    0b0010111, // 'H'
    0b0000110, // 'I'
    0b0111100, // 'J' // seems to be faulty!
    0b1010111, // 'K'
    0b0001110, // 'L'
    0b1010100, // 'M' // seems to be faulty!
    0b0010101, // 'N' // seems to be faulty!
    0b1111110, // 'O'
    0b1100111, // 'P'
    0b1110011, // 'Q'
    0b0000101, // 'R' // seems to be faulty!
    0b1011011, // 'S'
    0b0001111, // 'T'
    0b0111110, // 'U'
    0b0101010, // 'V'
    0b0101011, // 'W'
    0b0110111, // 'X'
    0b0111011, // 'Y'
    0b1101100, // 'Z' // seems to be faulty!
    0b1001110, // '['
    0b0010010, // '\'
    0b1111000, // ']' // seems to be faulty!
    0b1000000, // '^' // seems to be faulty!
    0b0001000, // '_' // seems to be faulty!
};

/* More advanced, design specific configuration;
 * comment out, when unused!
 */

/* Option to pretty print the input value,
 * dependent on your design.
 */
#define VERIFICATION_PRETTY_PRINT_INPUT_VAL(val) {\
  Serial.print(" (representing ASCII character '");\
  Serial.print((char)(val | 0x40));\
  Serial.print("')");\
  }

#define VERIFICATION_PRETTY_PRINT_EXPECTED_OUT_VAL(val) {\
  Serial.print(" representing 7-segment:\n    ");\
  Serial.println((val & 0x40) ? "_" : " ");\
  Serial.print("   ");\
  Serial.print((val & 0x02) ? "|" : " ");\
  Serial.print((val & 0x01) ? "_" : " ");\
  Serial.print((val & 0x20) ? "|" : " ");\
  Serial.println();\
  Serial.print("   ");\
  Serial.print((val & 0x04) ? "|" : " ");\
  Serial.print((val & 0x08) ? "_" : " ");\
  Serial.print((val & 0x10) ? "|" : " ");\
  Serial.println("");\
  }

// ------------------------------------------------------------------------------
/* Some parameters to tune verification, but actually
 * no real need to touch any code below this line
 */

/* As this is run in simulation, baud rate is not that important */
#define SERIAL_BAUDRATE             (230400u)

/* These params can be used to control simulation speed;
 * getting a correct reading the 7-segment display on my device takes quite some time.
 */
#define VERIFICATION_SETUP_TIME_MS  ( 50u)
#define VERIFICATION_HOLD_TIME_MS   (350u)

// ------------------------------------------------------------------------------
/* No real need to touch any code below this line */

/* The mapping of 10 input and 10 output pins each could be static;
 * however, this limits support to designs with <= 10 input and <= 10 output pins,
 * instead of limiting the support to designs with an overall pin count of 20!
 */
#define DESIGN_IN_A   ( 2)
#define DESIGN_IN_B   ( 3)
#define DESIGN_IN_C   ( 4)
#define DESIGN_IN_D   ( 5)
#define DESIGN_IN_E   ( 6)
#define DESIGN_IN_F   ( 7)
#define DESIGN_IN_G   ( 8)
#define DESIGN_IN_H   ( 9)
#define DESIGN_IN_I   (10)
#define DESIGN_IN_J   (11)

#define DESIGN_OUT_A  (12)
#define DESIGN_OUT_B  (13)
#define DESIGN_OUT_C  (14)
#define DESIGN_OUT_D  (15)
#define DESIGN_OUT_E  (16)
#define DESIGN_OUT_F  (17)
#define DESIGN_OUT_G  (18)
#define DESIGN_OUT_H  (19)
#define DESIGN_OUT_I  (20)
#define DESIGN_OUT_J  (21)

void setup()
{
  const bool stop_verification_on_error = VERIFICATION_STOP_ON_ERROR;
  bool tests_passed = true; /* asume the best for the start */

  Serial.begin(SERIAL_BAUDRATE);

  pinMode(DESIGN_IN_A, OUTPUT);
  pinMode(DESIGN_IN_B, OUTPUT);
  pinMode(DESIGN_IN_C, OUTPUT);
  pinMode(DESIGN_IN_D, OUTPUT);
  pinMode(DESIGN_IN_E, OUTPUT);
  pinMode(DESIGN_IN_F, OUTPUT);
  pinMode(DESIGN_IN_G, OUTPUT);
  pinMode(DESIGN_IN_H, OUTPUT);
  pinMode(DESIGN_IN_I, OUTPUT);
  pinMode(DESIGN_IN_J, OUTPUT);

  pinMode(DESIGN_OUT_A, INPUT);
  pinMode(DESIGN_OUT_B, INPUT);
  pinMode(DESIGN_OUT_C, INPUT);
  pinMode(DESIGN_OUT_D, INPUT);
  pinMode(DESIGN_OUT_E, INPUT);
  pinMode(DESIGN_OUT_F, INPUT);
  pinMode(DESIGN_OUT_G, INPUT);
  pinMode(DESIGN_OUT_H, INPUT);
  pinMode(DESIGN_OUT_I, INPUT);
  pinMode(DESIGN_OUT_J, INPUT);

  Serial.println("Testing all input combinations.");

  Serial.print("Stop verification on error? ");
  Serial.println(stop_verification_on_error ? "Yes" : "No");

  /* take a more generic approach;
   * stops on first error; TODO: this could be an option in the autoatic verification generator
   * caution: this is WIP!
   */
  for(uint16_t in_val = 0; ( in_val < (2 << (DESIGN_NUM_USED_INPUTS-1)) ) && 
                            ( !stop_verification_on_error || 
                              (stop_verification_on_error && tests_passed) ); in_val++ )
  {
    set_design_input_val(in_val);

    // wait some time before checking outputs
    delay(VERIFICATION_SETUP_TIME_MS);

    tests_passed &= verify_design_output_val(in_val);

    // wait some time before setting next inputs
    delay(VERIFICATION_HOLD_TIME_MS);
  }

  Serial.println();
  if(tests_passed)
  {
    Serial.println("[PASSED]");
  }
  else
  {
    Serial.println("[FAILED]");
  }
}

void set_design_input_val(uint16_t val)
{
  /* Set logic design inputs at Arduino's output pins;
   * output the bits via serial later, so that we don't add large delay between the different pins!
   */
  if( (DESIGN_NUM_USED_INPUTS-1) >= 0 )
  {
    digitalWrite(DESIGN_IN_A, val & (1 << DESIGN_NUM_USED_INPUTS-1));
  }
  if( (DESIGN_NUM_USED_INPUTS-2) >= 0 )
  {
    digitalWrite(DESIGN_IN_B, val & (1 << DESIGN_NUM_USED_INPUTS-2));
  }
  if( (DESIGN_NUM_USED_INPUTS-3) >= 0 )
  {
    digitalWrite(DESIGN_IN_C, val & (1 << DESIGN_NUM_USED_INPUTS-3));
  }
  if( (DESIGN_NUM_USED_INPUTS-4) >= 0 )
  {
    digitalWrite(DESIGN_IN_D, val & (1 << DESIGN_NUM_USED_INPUTS-4));
  }
  if( (DESIGN_NUM_USED_INPUTS-5) >= 0 )
  {
    digitalWrite(DESIGN_IN_E, val & (1 << DESIGN_NUM_USED_INPUTS-5));
  }
  if( (DESIGN_NUM_USED_INPUTS-6) >= 0 )
  {
    digitalWrite(DESIGN_IN_F, val & (1 << DESIGN_NUM_USED_INPUTS-6));
  }
  if( (DESIGN_NUM_USED_INPUTS-7) >= 0 )
  {
    digitalWrite(DESIGN_IN_G, val & (1 << DESIGN_NUM_USED_INPUTS-7));
  }
  if( (DESIGN_NUM_USED_INPUTS-8) >= 0 )
  {
    digitalWrite(DESIGN_IN_H, val & (1 << DESIGN_NUM_USED_INPUTS-8));
  }
  if( (DESIGN_NUM_USED_INPUTS-9) >= 0 )
  {
    digitalWrite(DESIGN_IN_I, val & (1 << DESIGN_NUM_USED_INPUTS-9));
  }
  if( (DESIGN_NUM_USED_INPUTS-10) >= 0 )
  {
    digitalWrite(DESIGN_IN_J, val & (1 << DESIGN_NUM_USED_INPUTS-10));
  }

  Serial.print("\nWrote input: 0b");
  for(int8_t bit_index = DESIGN_NUM_USED_INPUTS-1; bit_index >= 0; bit_index--)
  {
    Serial.print((val >> bit_index) & 1, BIN);
  }
#ifdef VERIFICATION_PRETTY_PRINT_INPUT_VAL
  VERIFICATION_PRETTY_PRINT_INPUT_VAL(val);
#endif
  Serial.println();
}

bool verify_design_output_val(uint16_t in_val)
{
  // read value from logic design outputs at Arduino's input pins 
  uint16_t val = 0;
  if( DESIGN_NUM_USED_OUTPUTS >= 1 )
  {
    val |= digitalRead(DESIGN_OUT_A);
    val <<= 1;
  }
  if( DESIGN_NUM_USED_OUTPUTS >= 2 )
  {
    val |= digitalRead(DESIGN_OUT_B);
    val <<= 1;
  }
  if( DESIGN_NUM_USED_OUTPUTS >= 3 )
  {
    val |= digitalRead(DESIGN_OUT_C);
    val <<= 1;
  }
  if( DESIGN_NUM_USED_OUTPUTS >= 4 )
  {
    val |= digitalRead(DESIGN_OUT_D);
    val <<= 1;
  }
  if( DESIGN_NUM_USED_OUTPUTS >= 5 )
  {
    val |= digitalRead(DESIGN_OUT_E);
    val <<= 1;
  }
  if( DESIGN_NUM_USED_OUTPUTS >= 6 )
  {
    val |= digitalRead(DESIGN_OUT_F);
    val <<= 1;
  }
  if( DESIGN_NUM_USED_OUTPUTS >= 7 )
  {
    val |= digitalRead(DESIGN_OUT_G);
    val <<= 1;
  }
  if( DESIGN_NUM_USED_OUTPUTS >= 8 )
  {
    val |= digitalRead(DESIGN_OUT_H);
    val <<= 1;
  }
  if( DESIGN_NUM_USED_OUTPUTS >= 9 )
  {
    val |= digitalRead(DESIGN_OUT_I);
    val <<= 1;
  }
  if( DESIGN_NUM_USED_OUTPUTS >= 10 )
  {
    val |= digitalRead(DESIGN_OUT_J);
    val <<= 1;
  }
  /* the last shift is always one too many */
  val >>= 1;

  Serial.print("  Expected output:  0b");
  for(int8_t bit_index = DESIGN_NUM_USED_OUTPUTS-1; bit_index >= 0; bit_index--)
  {
    Serial.print((expected_out_val[in_val] >> bit_index) & 1, BIN);
  }
#ifdef VERIFICATION_PRETTY_PRINT_EXPECTED_OUT_VAL
  VERIFICATION_PRETTY_PRINT_EXPECTED_OUT_VAL(expected_out_val[in_val]);
#endif
  Serial.println();

  Serial.print("  Read back output: 0b");
  for(int8_t bit_index = DESIGN_NUM_USED_OUTPUTS-1; bit_index >= 0; bit_index--)
  {
    Serial.print((val >> bit_index) & 1, BIN);
  }
#ifdef VERIFICATION_PRETTY_PRINT_REAL_OUT_VAL
  VERIFICATION_PRETTY_PRINT_REAL_OUT_VAL();
#endif
  Serial.println();

  if(expected_out_val[in_val] == val)
  {
    Serial.println("  [PASS]");
    return true;
  }
  else
  {
    Serial.println("  [FAIL]");
    return false;
  }
}

void loop()
{
  /* no need to loop as everything is only required once and therefore
   * already done in setup()
   */
}

maehw / wokwi-lookup-table-generator

Add support for automated in-circuit verification #3