Serial 2.0

[soundanalogous]

Serial 1.0 does not scale well. Update to Serial 2.0 but with a different ID and this time allocate separate bytes for command and port.

Current:

0  START_SYSEX      (0xF0)
1  SERIAL_DATA      (0x60)  // command byte
2  SERIAL_CONFIG    (0x10)  // OR with port (0x11 = SERIAL_CONFIG | HW_SERIAL1)
3  baud             (bits 0 - 6)
4  baud             (bits 7 - 13)
5  baud             (bits 14 - 20) // need to send 3 bytes for baud even if value is < 14 bits
6  rxPin            (0-127) [optional] // only set if platform requires RX pin number
7  txPin            (0-127) [optional] // only set if platform requires TX pin number
6|8 END_SYSEX       (0xF7)

Updated:

0  START_SYSEX      (0xF0)
1  SERIAL2_DATA     (0x64)  // command byte
2  SERIAL_CONFIG    (0x00)
3  port             (HW_SERIALn OR SW_SERIALn)
4  baud             (bits 0 - 6)
5  baud             (bits 7 - 13)
6  baud             (bits 14 - 20) // need to send 3 bytes for baud even if value is < 14 bits
7  rxPin            (0-127) [optional] // only set if platform requires RX pin number
8  txPin            (0-127) [optional] // only set if platform requires TX pin number
7|9 END_SYSEX       (0xF7)

Firmata client authors should be able to implement this change without breaking APIs.

[soundanalogous]

Depends on finalization of https://github.com/firmata/protocol/issues/41 in order to query which features are supported (so client will know if firmware implements Serial v1.0 or Serial v2.0)

[soundanalogous]

https://github.com/firmata/protocol/pull/71

[MJLHThomassen]

Would it also be possible to add support for setting # of bits per tranmission, also supporting 9-bit mode and the Multi-processor Communication Mode?

ATmega48A/PA/88A/PA/168A/PA/328/P complete datasheet-- http://www.atmel.com/Images/doc8271.pdf Page 195 (search "Multi-processor Communication Mode"):

20.9 Multi-processor Communication Mode Setting the Multi-processor Communication mode (MPCMn) bit in UCSRnA enables a filtering function of incoming frames received by the USART Receiver. Frames that do not contain address information will be ignored and not put into the receive buffer. This effectively reduces the number of incoming frames that has to be handled by the CPU, in a system with multiple MCUs that communicate via the same serial bus. The Transmitter is unaffected by the MPCMn setting, but has to be used differently when it is a part of a system utilizing the Multi-processor Communication mode. If the Receiver is set up to receive frames that contain 5 to 8 data bits, then the first stop bit indicates if the frame contains data or address information. If the Receiver is set up for frames with nine data bits, then the ninth bit (RXB8n) is used for identifying address and data frames. When the frame type bit (the first stop or the ninth bit) is one, the frame contains an address. When the frame type bit is zero the frame is a data frame. The Multi-processor Communication mode enables several slave MCUs to receive data from a master MCU. This is done by first decoding an address frame to find out which MCU has been addressed. If a particular slave MCU has been addressed, it will receive the following data frames as normal, while the other slave MCUs will ignore the received frames until another address frame is received.

20.9.1 Using MPCMn For an MCU to act as a master MCU, it can use a 9-bit character frame format (UCSZn = 7). The ninth bit (TXB8n) must be set when an address frame (TXB8n = 1) or cleared when a data frame (TXB = 0) is being transmitted. The slave MCUs must in this case be set to use a 9-bit character frame format. The following procedure should be used to exchange data in Multi-processor Communication mode:

1. All Slave MCUs are in Multi-processor Communication mode (MPCMn in UCSRnA is set).
2. The Master MCU sends an address frame, and all slaves receive and read this frame. In the Slave MCUs, the RXCn Flag in UCSRnA will be set as normal.
3. Each Slave MCU reads the UDRn Register and determines if it has been selected. If so, it clears the MPCMn bit in UCSRnA, otherwise it waits for the next address byte and keeps the MPCMn setting.
4. The addressed MCU will receive all data frames until a new address frame is received. The other Slave MCUs, which still have the MPCMn bit set, will ignore the data frames.
5. When the last data frame is received by the addressed MCU, the addressed MCU sets the MPCMn bit and waits for a new address frame from master. The process then repeats from 2. Using any of the 5- to 8-bit character frame formats is possible, but impractical since the Receiver must change between using n and n+1 character frame formats. This makes fullduplex operation difficult since the Transmitter and Receiver uses the same character size setting. If 5- to 8-bit character frames are used, the Transmitter must be set to use two stop bit (USBSn = 1) since the first stop bit is used for indicating the frame type. Do not use Read-Modify-Write instructions (SBI and CBI) to set or clear the MPCMn bit. The MPCMn bit shares the same I/O location as the TXCn Flag and this might accidentally be cleared when using SBI or CBI instructions.

[soundanalogous]

@MJLHThomassen Do you have a specific use case for this? Also the link you provided redirects to the Microchip home page.

[MJLHThomassen]

@soundanalogous I am currently working on an application/board for simulating I2C peripherals. We are using ATmega32u4 chips connected to an I2C bus which itself is connected to the MCU of a hardware device we are producing. Each ATmega32u4 simulates an I2C sensor (accelerometer, light sensor etc) so we can test the MCU's software response to simulated sensor input.

The ATmega32u4's need to recieve commands from a master device (currently also an ATmega32u4 which communicats via serial using Firmata with a host pc sending instructions), but because they are using their I2C bus for communication with our MCU, we can't use that bus for sending commands to the devices. We hooked them all up in Master-Slave Multi Processor Communication Mode with the master device being the one running Firmata.

However, according to the datasheet (another link here: http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-7766-8-bit-AVR-ATmega16U4-32U4_Datasheet.pdf, chapter 18.8.1) a master device can use the 9-bit character frame format to indicate if a message is an adress message or a data message.

For an MCU to act as a master MCU, it can use a 9-bit character frame format (UCSZn = 7). The ninth bit (TXB8n) must be set when an address frame (TXB8n = 1) or cleared when a data frame (TXB = 0) is being transmitted. The slave MCUs must in this case be set to use a 9-bit character frame format.

Firmata wouldn't nessecarily need to implement the slave side, only the master side. I propose the following addition to the serial protocol for this:

Serial Config:

0  START_SYSEX      (0xF0)
1  SERIAL2_DATA     (0x64)  // command byte
2  SERIAL_CONFIG    (0x00)
3  port             (HW_SERIALn OR SW_SERIALn)
4  baud             (bits 0 - 6)
5  baud             (bits 7 - 13)
6  baud             (bits 14 - 20) // need to send 3 bytes for baud even if value is < 14 bits
7  mpcmMaster       (0 or 1) // indicates this device is a multi processor communication mode master
8  rxPin            (0-127) [optional] // only set if platform requires RX pin number
9  txPin            (0-127) [optional] // only set if platform requires TX pin number
7|10 END_SYSEX      (0xF7)

When mpcmMaster is set, the device would operate with 9 bit character frames as described in the "Using MPCMn" section of the avr datasheet.

Serial Write:

0  START_SYSEX      (0xF0)
1  SERIAL_DATA      (0x67)
2  SERIAL_WRITE     (0x01)
3  port             (HW_SERIALn OR SW_SERIALn)
4  data 0 | address (LSB) // address if serial is configured as mpcm master, data otherwise
5  data 0 | address (MSB) // address if serial is configured as mpcm master, data otherwise
6  data 1 | 0       (LSB)
7  data 1 | 0       (MSB)
8  data 2 | 1       (LSB)
9  data 2 | 1       (MSB)
...                 // up to max buffer - 5
n  END_SYSEX        (0xF7)

[soundanalogous]

Is MPCMn library that works across a wide range of Arduino-compatible architectures. I'm reluctant to add an option that is specific only to the ATmega16/32U4.

[MJLHThomassen]

I don't have the full view of which AVR's support MPCMn, but every one with hardware UART I encountered has this mode available.

For the master (which is what i am proposing here), it would be enough to be able to set "9-bit character format". This is something supported by all (hardware) serial on all devices. Perhaps the config bit shouldn't be called "mpcmMaster" but "characterFormat" which, on any AVR, can range from 5 to 9.

0  START_SYSEX      (0xF0)
1  SERIAL2_DATA     (0x64)  // command byte
2  SERIAL_CONFIG    (0x00)
3  port             (HW_SERIALn OR SW_SERIALn)
4  baud             (bits 0 - 6)
5  baud             (bits 7 - 13)
6  baud             (bits 14 - 20) // need to send 3 bytes for baud even if value is < 14 bits
7  characterFormat  (5, 6, 7, 8 or 9) [optional] // indicates the bits per character of the UART, default: 8
8  rxPin            (0-127) [optional] // only set if platform requires RX pin number
9  txPin            (0-127) [optional] // only set if platform requires TX pin number
7|10 END_SYSEX      (0xF7)

Then:

0  START_SYSEX      (0xF0)
1  SERIAL_DATA      (0x67)
2  SERIAL_WRITE     (0x01)
3  port             (HW_SERIALn OR SW_SERIALn)
6  data 0       (LSB)
7  data 0       (MSB) // This can either contain 0 data bits (5, 6 and 7 bits character format), 1 data bit (8 bit character format) or 2 data bits (9 bit character format)
8  data 1       (LSB)
9  data 1       (MSB) // Same
...                 // up to max buffer - 5
n  END_SYSEX        (0xF7)

This way, the user can determine for themself if they view the 9th bit as address or data bit indicator as required by MPCM, or if they just send 9 bits per character to the other device. In 5, 6 or 7 bits character format, the MSB could also be omitted.

[soundanalogous]

Is characterFormat the same as data bits? I could add dataBits, stopBits, and parity.

[MJLHThomassen]

Yes, I was also thinking about adding stopbits and parity, however I don't think we can introduce the 9-bit mode since the default Arduino HardwareSerial does not support it. There have been numerous discussions about it in the past and some (half-baked) implementations but the arduino maintainers can't apparantly figure out how to properly add it.

Anyway, adding dataBits, stopBits and parity to the protocol would help me alot (i can mis-use the 2nd stop bit as the 9th bit for example, which is also mentioned in some MCPM implementations as a way to do it).

As a reference, the arduino HardwareSerial has support for the following modes:

From https://www.arduino.cc/reference/en/language/functions/communication/serial/begin/:

config: sets data, parity, and stop bits. Valid values are

SERIAL_5N1 SERIAL_6N1 SERIAL_7N1 SERIAL_8N1 (the default) SERIAL_5N2 SERIAL_6N2 SERIAL_7N2 SERIAL_8N2 SERIAL_5E1 SERIAL_6E1 SERIAL_7E1 SERIAL_8E1 SERIAL_5E2 SERIAL_6E2 SERIAL_7E2 SERIAL_8E2 SERIAL_5O1 SERIAL_6O1 SERIAL_7O1 SERIAL_8O1 SERIAL_5O2 SERIAL_6O2 SERIAL_7O2 SERIAL_8O2

Where N = no parity, E = even parity, O = odd parity

[soundanalogous]

As a reference, the arduino HardwareSerial has support for the following modes:

Unfortunately that set of constants varies by which architecture the board uses. That list is only valid for AVR. It's a mess. I tried to support it in the original Firmata Serial proposal but eventually gave up. I may need to investigate alternative cross-architecture Arduino-compatible Serial libraries to make it work or try to get Arduino to provide an alternate constructor for Serial.begin that allows specifying the parameters individually.

[MJLHThomassen]

Unfortunately that set of constants varies by which architecture the board uses. That list is only valid for AVR. It's a mess.

Ah I did not know that. I thought it was supported on every Arduino platform. I agree that using Defines for that is super strange and why don't they just use the numbers or defines for individual parameters.

Anyway, I do not know about how things work on non AVR platforms or about alternative serial implementations. I understand you don't want to add something to the protocol that is device-dependant though.

[soundanalogous]

I can add data bits, stop bit and parity to the protocol, it just won't work on Arduino.

[MJLHThomassen]

Allright. I was planning on implementing the 2.0 protocol for Configurable Firmata. Obviously, with the unsupported things for certain boards resulting in either no-ops or a returned error message. Adding it to the protocol is a first step towards achieving that goal.

I don't think it has to be a problem though. Currently the Serial 1.0 implementation of Configurable Firmata also can't use Serial (the "0th" one) since thats already in use for Client <-> Board comms and can only practically be used on Mega AVR's with more then 1 port anyway (which, from the looks of it, all support the modes i mentioned for their other serial ports).

[soundanalogous]

General pattern here in Arduino HAL, for some architectures they map defines to enums, for others they map to uint8-t value, others to uint16_t value:

• AVR
• SAM
• SAMD

At least AVR and SAMD can be resolved with a cast. That covers the majority of Arduino branded boards.

[MJLHThomassen]

I think the solution to that would be something along the lines of:

HardwareSerial* serial = Serial;
if (dataBits == 5)
{
    if (parity == ODD_PARITY) // Could be NO_PARITY 0x00, EVEN_PARITY 0x01, ODD_PARITY 0x02 
    {
        if (stopBits == 2)
        {
            serial.begin(baudRate, SERIAL_5O2);
        }
        else // stopBits == 1 (or default)
        {
            serial.begin(baudRate, SERIAL_5O1);
        }
    }
    else if (parity == EVEN_PARITY)
    {
        if (stopBits == 2)
        {
            serial.begin(baudRate, SERIAL_5E2);
        }
        else
        {
            serial.begin(baudRate, SERIAL_5E1);
        }
    }
    else // parity == NO_PARITY (or default)
    {
        if (stopBits == 2)
        {
            serial.begin(baudRate, SERIAL_5N2);
        }
        else
        {
            serial.begin(baudRate, SERIAL_5N1);
        }
    }
}
else if (dataBits == 6)
{
    // etc...
}
else if (dataBits == 7)
{
    // etc...
}
else // dataBits == 8 (or default)
{
        // etc...
}

Could possibly also be implemented using a LUT, though the fact the config parameter has different types across implementations could be problematic.

The Firmata side would look something like:

0  START_SYSEX      (0xF0)
1  SERIAL2_DATA     (0x64)  // command byte
2  SERIAL_CONFIG    (0x00)
3  port             (HW_SERIALn OR SW_SERIALn)
4  baud             (bits 0 - 6)
5  baud             (bits 7 - 13)
6  baud             (bits 14 - 20) // need to send 3 bytes for baud even if value is < 14 bits
7  dataBits         (5, 6, 7 or 8) [optional] // indicates the bits per character of the UART, default: 8, set to 0 for default
8  stopBits         (1 or 2) [optional] // indicates the nr of stop of the UART, default: 1, set to 0 for default
9  parity           (0 (none), 1(even) or 2(odd)) [optional] // indicates the parity of the UART, default: 0
10 rxPin            (0-127) [optional] // only set if platform requires RX pin number
11 txPin            (0-127) [optional] // only set if platform requires TX pin number
7|12 END_SYSEX      (0xF7)

For software serial, you need to specify RX and TX, but the dataBits, stopBits and parity will be ignored, so setting them to 0 (along with above sample implementation) will result in softwareSerial with the default 8N1 mode.