software serial (and support for additional UARTs)

[soundanalogous]

A number of requests have been made to add software serial support to Firmata. I think instead of just software serial, it would be better to support both software serial and additional UARTs (on a Mega for example: Serial1, Serial2, etc).

For those who are asking for serial support in Firmata, what are your use cases?

• read only serial devices?
• read/write serial devices
• board to board communication over serial?
• anything else?

I need to know this in order to determine how much of the Stream interface to expose. In the simplest example I assume a read buffer and a write buffer would be sufficient - simply echoing the serial data. However are there also use cases that would need the peek() or flush() methods. There are also limitations in buffer sizes (64 bytes incoming... but you have to split every byte so you only get 32 bytes... even less actually after stripping out the command bytes so more like 29 or 30 bytes of actual data).

Software serial also interferes with some timer-based features. So PWM may not work as expected when using Software serial. There are other potential collisions as well. However this can be avoided by using a Leonardo, Mega or other board that has separate hardware serial ports (Serial1, Serial2, etc). It would use the same protocol as SW serial with the main difference being that you'd configure with a port number rather than a pair of pin numbers.

[kamituel]

My use case is reading RFID tags using this module. So it falls into "read-only serial devices" category.

AFAIK I need to access following methods on SoftwareSerial (which inherits from Stream):

• begin()
• available()
• read()

#include <SoftwareSerial.h>

SoftwareSerial rfid = SoftwareSerial(5, 6);
rfid.begin(9600);

while(rfid.available() > 0) {
    c=rfid.read();
    // ...
}

(source)

I'm using Adruino Duemilanove.

Thanks for openining this issue, @soundanalogous.

[jtrinklein]

My use case is board to board communication. I am in the read/write category.

[soundanalogous]

Board to board communication via software serial with one of the boards having a hardware serial (or Ethernet or WiFi) connection to a Firmata client application?

[jtrinklein]

Here is my setup: I have a raspberry pi with serial connection to arduino with one way communication pi writes, arduino reads. Then I have the arduino connected to another arduino with another one way communication setup. Pi [Tx] -> [Rx] arduino1[Tx] -> [Rx] arduino2

[soundanalogous]

So in your setup, arduino1 would be running StandardFirmata. Arduino 2 is connected to Arduino1 via SoftwareSerial and you have a firmata client application running on the Pi that communicates with arduino 1 via Firmata and would be able to communicate with Arduino 2 through Arduino 1 using SoftwareSerial through Firmata?

On Sun, Nov 3, 2013 at 4:20 PM, James Trinklein notifications@github.comwrote:

Here is my setup: I have a raspberry pi with serial connection to arduino with one way communication pi writes, arduino reads. Then I have the arduino connected to another arduino with another one way communication setup. Pi [Tx] -> [Rx] arduino1[Tx] -> [Rx] arduino2

— Reply to this email directly or view it on GitHubhttps://github.com/firmata/arduino/issues/97#issuecomment-27654679 .

[jtrinklein]

Yep, you got it.

[soundanalogous]

Some initial thoughts on a protocol:

// config: creates new SoftwareSerial instance and calls begin(baud)
0  START_SYSEX  (0xF0)
1  SW_SERIAL    (0x60)  // command byte
2  CONFIG       (0x00)  // subcommand byte
3  txPin        (0-127) // restrictions apply per board
4  rxPin        (0-127) // restrictions apply per board
5  baud         (bits 0 - 6)
6  baud         (bits 7 - 13)
7  baud         (bits 14 - 20)
8  END_SYSEX    (0XF7)

// SoftwareSerial instantiation needs to be deferred
#inclde <SoftwareSerial.h>
SoftwareSerial *mySerial = NULL;

void setup() {}

void loop() {
  if (!mySerial) {
    mySerial = new SoftwareSerial(txPin, rxPin);
    mySerial->begin(baud);
  }
}

// read contents of sw serial buffer and send to Firmata client
0  START_SYSEX  (0xF0)
1  SW_SERIAL    (0x60)
2  SW_SERIAL_TX (0x01)
3  txPin        // used to identify port (may remove this)
4  data 0       (LSB)  // each byte is split
5  data 0       (MSB)
6  data 1       (LSB)
7  data 1       (MSB)
...             // up to max buffer - 5
n  END_SYSEX    (0XF7)

// example implementation
if (mySerial->available() > 0) {
  Firmata.write(START_SYSEX);
  Firmata.write(SW_SERIAL);
  Firmata.write(SW_SERIAL_TX);
  Firmata.write(txPin);
  Encoder7Bit.startBinaryWrite();
  while (mySerial->available() > 0) {
    Encoder7Bit.writeBinary(mySerial->read());
  }
  Encoder7Bit.endBinaryWrite();
  Firmata.write(END_SYSEX);
}

// receive serial data from Firmata client, reassemble and
// call mySerial->write(data) for each byte
0  START_SYSEX  (0xF0)
1  SW_SERIAL    (0x60)
2  SW_SERIAL_RX (0x02)
3  rxPin        // used to identify port (may remove this)
4  data 0       (LSB)
5  data 0       (MSB)
6  data 1       (LSB)
7  data 1       (MSB)
...             // up to max buffer - 5
n  END_SYSEX    (0XF7)

// example implementation
handleSWSerialRX(byte argc, byte *argv)
{
  byte data;
  // argv[0] is subcommand
  // argv[1] is rxPin

  // reassemble data bytes and forward to sw serial write buffer
  for (byte i = 2; i < argc; i += 2) {
    data = argv[i] + (argv[i + 1] << 7);
    mySerial->write(data);
  }
}

// call mySerial->flush()
0  START_SYSEX      (0xF0)
1  SW_SERIAL        (0x60)
2  SW_SERIAL_FLUSH  (0x03)
3  txPin            // used to identify port (may remove this)
4  END_SYSEX        (0XF7)

[maerzhase]

anybody realized the firmata working with softserial? i am working on a project that would profit from it aswell

[soundanalogous]

I need to find some time to work on it.

[ClintonKeith]

I'm using the Romeo DF Robot Controller: http://www.dfrobot.com/index.php?route=product/product&filter_name=romeo&product_id=844#.Utnpz3mtvqa

Which is based on the Leonardo. It has a built-in XBee socket connected to the hardware serial (Serial1). This is a perfect platform for Johnny-five! Hopefully this fix will happen.

[soundanalogous]

I've taken an initial pass at an implementation (based on the protocol I drafted above). It is available in my fork in the softserial branch. It compiles in Arduino 1.0.5 (we're currently having issues compiling configurable firmata in Arduino 1.5.x). You'll find SoftSerialFirmata in the examples folder. Compile and upload to an Arduino using Arduino IDE version 1.0.5. I haven't created any client examples yet so this is all still theory at this point until proven through a working example.

[soundanalogous]

This is the protocol for the preliminary implementation. It differs from what is above in that for now I'm limiting to only a single SoftwareSerial instance so the rxPin and txPin are not included in the SW_SERIAL_RX and SW_SERIAL_TX commands.

Also this will only work for software serial (wrapping the Arduino SoftwareSerial library). Hardware Serial (Serial1, Serial2, etc) poses another set of challenges.

// config: creates new SoftwareSerial instance and calls begin(baud)
0  START_SYSEX  (0xF0)
1  SW_SERIAL    (0x60)  // command byte
2  CONFIG       (0x00)  // subcommand byte
3  txPin        (0-127) // restrictions apply per board
4  rxPin        (0-127) // restrictions apply per board
5  baud         (bits 0 - 6)
6  baud         (bits 7 - 13)
7  baud         (bits 14 - 20)
8  END_SYSEX    (0XF7)

// read contents of sw serial buffer and send to Firmata client
0  START_SYSEX  (0xF0)
1  SW_SERIAL    (0x60)
2  SW_SERIAL_TX (0x01)
3  data 0       (LSB)  // each byte is split
4  data 0       (MSB)
5  data 1       (LSB)
6  data 1       (MSB)
...             // up to max buffer - 5
n  END_SYSEX    (0XF7)

// receive serial data from Firmata client, reassemble and
// call mySerial->write(data) for each byte
0  START_SYSEX  (0xF0)
1  SW_SERIAL    (0x60)
2  SW_SERIAL_RX (0x02)
3  data 0       (LSB)
4  data 0       (MSB)
5  data 1       (LSB)
6  data 1       (MSB)
...             // up to max buffer - 5
n  END_SYSEX    (0XF7)

[ClintonKeith]

This is great news. Happy to help out here where I can. I'm on 1.5.5 with a Leonardo processor. I can breadboard my Xbee to get it into a software serial configuration (rather than the built-in Romeo 2 Serial1 socket). I'll keep an eye out here for when 1.5.x works.

[soundanalogous]

In thinking more about this, I may be able to make this work for either HW or SW serial. Would take something like this:

// in interface declaration:
Stream * serialPort

// in implementation:
if (portType == HW_SERIAL1) {
// make sure we have a board with Serial1
#if defined(UBRR1H) || defined(USBCON)
    serialPort = &Serial1;
#endif
} else if (portType == SW_SERIAL) {
  byte txPin = argv[4];
  byte rxPin = argv[5];
  serialPort = new SoftwareSerial(txPin, rxPin);
}

if (serialPort) {
  serialPort->begin(baud);
}

This would change the config portion of the protocol to something like this:

// config: creates new SoftwareSerial instance and calls begin(baud)
0  START_SYSEX  (0xF0)
1  SERIAL       (0x60)  // command byte
2  CONFIG       (0x10)  // subcommand byte (0x10 = CONFIG SW_SERIAL, 0x20 = CONFIG HW_SERIAL1)
3  baud         (bits 0 - 6)
4  baud         (bits 7 - 13)
5  baud         (bits 14 - 20) // need to send 3 bytes for baud even if value is < 14 bits
6  txPin        (0-127) [softserial only] // restrictions apply per board
7  rxPin        (0-127) [softserial only] // restrictions apply per board
8  END_SYSEX    (0XF7)

[ClintonKeith]

Looks good.

[phlegx]

+1

[kraman]

I have found it useful to buffer the serial data on arduino side and send it in a large chunk. I also support sending the output from arduino -> client based on a configurable character. Eg: \n Here is the protocol I ended up using (based on yours):

#define SYSEX_SERIAL 0x60

#define SERIAL_CONFIG 0x10
#define SERIAL_COMM 0x20
#define SERIAL_FLUSH 0x30
#define SERIAL_CLOSE 0x40

#define SW_SERIAL 0x00
#define HW_SERIAL1 0x01
#define HW_SERIAL2 0x02
#define HW_SERIAL3 0x03

// config: creates new Serial1 instance and calls begin(baud)
0  START_SYSEX  (0xF0)
1  SYSEX_SERIAL (0x60)  // command byte
2  CONFIG       (0x11)  // subcommand byte (SERIAL_CONFIG | HW_SRIAL1)
3  baud         (bits 0 - 6)
4  baud         (bits 7 - 13)
5  baud         (bits 14 - 20) // need to send 3 bytes for baud even if value is < 14 bits
6  buffer len   (bits 0 - 6)
7  buffer len   (bits 7 - 13)    
8  buffer len   (bits 14 - 20)
9  term char    (bits 0 - 6)
10 term char    (bits 7 - 13)
6  txPin        (0-127) [softserial only] // restrictions apply per board
7  rxPin        (0-127) [softserial only] // restrictions apply per board
8  END_SYSEX    (0XF7)

// arduino -> client
0  START_SYSEX  (0xF0)
1  SYSEX_SERIAL (0x60)
2  SERIAL_COMM  (0x21) // subcommand byte (SERIAL_COMM | HW_SRIAL1)
3  data[0]      (bits 0 - 6)
4  data[0]      (bits 7 - 13)
...
...
10 END_SYSEX    (0XF7)

// client -> arduino
0  START_SYSEX  (0xF0)
1  SYSEX_SERIAL (0x60)
2  SERIAL_COMM  (0x21) // subcommand byte (SERIAL_COMM | HW_SRIAL1)
3  data[0]      (bits 0 - 6)
4  data[0]      (bits 7 - 13)
...
...
10 END_SYSEX    (0XF7)

// close
0  START_SYSEX  (0xF0)
1  SYSEX_SERIAL (0x60)
2  SERIAL_CLOSE (0x41) // subcommand byte (SERIAL_CLOSE | HW_SRIAL1)
10 END_SYSEX    (0XF7)

[soundanalogous]

can you submit a pull request with your implementation so I can review it?

[kraman]

@soundanalogous Its pretty much based on your changes. My code is here: https://github.com/kraman/go-firmata/commit/e7742e31a69a2e1f9731db22d5cc27b5c994d0a3

Its still a WIP. Ill work on a PR once its a bit more stable.

[leyenda]

Hello, Im still new into firmata and Im trying to understand the code kraman has shared but I still dont understand where do I put the pins Im using as RX and TX in an Arduino UNO. If someone can help me with this I'll be thankful.

[soundanalogous]

@leyenda What are you trying to accomplish? Are you trying to use SoftwareSerial with an Arduino UNO?

[leyenda]

@soundanalogous thanks for your quick reply, I wrote below what I am trying to.

I've followed a tutorial in your blog called "Wireless Firmata using XBees",then I tried to use johnny-five but firmata couldn't start at some point. After failing that attempt I tried a little echo program just using arduino UNO and the xbees, I notice that I can only get data from the arduino but not send to it, this all using the default RX,TX(0,1) that arduino uno has.

So I read about the SoftwareSerial library and tried it with the same echo program in pins 2,3 and it worked like a charm.

Now I know that I can send and get;but the pins are 2,3 and thats my doubt , how to use firmata on those pins as RX and TX.

[soundanalogous]

There are currently no SoftwareSerial firmata client implementations. It looks like kraman's implementation only supports hardware serial so far and only for the go programming language. It will be implemented in other firmata client libraries (johnny-five, Breakoutjs, etc) once the Firmata software and hardware serial implementation is stable.

[leyenda]

Thanks for the info @soundanalogous and keep going with firmata. It's awesome, still pretty abstract for me; but I'm gonna learn more about it.

[garrows]

Hey, Pretty sure what you want was implemented 3 years ago here https://github.com/firmata/arduino/commit/6f11b143ef58c3712dc99d542996da6a577c6ad3#diff-50924cea2c3746607a755a3974b4833fR76

I was using the DFRobot Romeo (same as @ClintonKeith) with a bluetooth MC-06 chip connected to pin 0 & 1 which unlike other Arduino boards, is not Serial but Serial1 instead.

I got it working by changing StandardFirmata on line https://github.com/firmata/arduino/blob/master/examples/StandardFirmata/StandardFirmata.ino#L597 From

Firmata.begin(57600);

to

Serial1.begin(57600);
Firmata.begin(Serial1);

I haven't tried with SoftSerial yet but I think it should work.

Hope this helps.

[soundanalogous]

@garrows what you are referencing is for using additional UARTs or SoftSerial to communicate with the client application. However this thread is about providing Serial access via the Firmata protocol to add additional serial ports in a client application. So you can for example get data from a GPS or other serial device that is attached to an Arduino while you are using the primary UART to communicate with the client application.

[ClintonKeith]

Thanks @garrows. I'll try that. I thought I did and it didn't work...

@soundanalogous, I may have read it wrong, but I thought it included both. You wrote:

A number of requests have been made to add software serial support to Firmata. I think instead of just software serial, it would be better to support both software serial and additional UARTs (on a Mega for example: Serial1, Serial2, etc).

[soundanalogous]

It would but I'm talking about something totally different than what @garrows is describing. What I'm talking about is a protocol for communicating with serial devices attached to an Arduino board rather than changing which UART or SoftSerial configuration you use for the Firmata connection with the client application.

[lyzadanger]

+1 I'm the pretty obvious use case: Adafruit Ultimate GPS Breakout (read-only, obviously). I'd love for Firmata to have SoftwareSerial—currently using a johnny-five/serialport setup on node, and this would be a big boon to me. I even look forward to parsing the NMEA sentences (once I can get at them!). Let me know if I can help out in any way...

[rwaldron]

@lyzadanger heh, you and me both!

[git-developer]

It would be great to have support for additional UARTs in Firmata. I'm currently using ConfigurableFirmata on a Mega2560, and I'd like to attach a serial device to one of its additional (hardware) serial ports. Is kraman's code sufficient for this?

[pooledge]

I guess it's the most suitable topic for the question, let me just leave it here:

I am using RS485 for communication between Arduino and host computer. My RS485 is half-duplex, therefore readout/writing is possible only on demand.

Now I'd like to use one of the pins, let's say DigitalPin3 to act as transmission enabler permanently, being initialized as LOW (listen for incoming signal, like Serial.available() > 0). As soon as any readout query to host has been initiated by Firmata:

1. DigitalPin3 must be set to HIGH to enable transmission
2. Wait until data has been sent over Serial (flush)
3. DigitalPin3 set back to LOW to disable transmission and enter the "listen" mode

Obviously, doing this for every "case" with Firmata.write in it inside sysexCallBack function does nothing. What would be the best way to achieve this?

Thank you in advance and sorry for my English.

[soundanalogous]

Is there a specific reason you are using RS485 rather than the standard USB serial connection between Arduino and the host computer?

[pooledge]

Hi Jeff, thanks for your reply.

Yes, it is. I plan to be able to control a robot in a studio environment, distanced between 100 and 600 m from the host, going along different electrical circuits, so symmetric line is the only choice for me.

I've initially thought about connecting two Arduinos via RS485, would you recommend this way? (a lot untransparenter for me right now)

For my application, there are 5 DC motors to be controlled via PWM, three sensors reading out some distance measurements and another PWM or analog out for the status LED, nothing else.

Thank you

[soundanalogous]

@pooledge This is the wrong thread for this discussion. I've started a new thread: https://github.com/firmata/arduino/issues/159

[Draghtnod]

Hi, I merged the head of @soundanalogous SoftwareSerial branch with the Configurable branch and added it to the reporting feature. I want to read the Data of a GY-GPS6MV2 (@9600 baud) on an ODroid U3 IO-Shield (ATMega328P@16MHz). Everything works like a charm except for one thing:

I had to adjust the timing values in SoftwareSerial.cpp to get stable data, but there are incorrect bits in the middle of the transmissions. Only some characters/bit-pattern are affected every ~15 bytes. The locations of these errors are stable, even if i modify the timing values. So it theoretically shouldn't be a timing issue. The GPS works fine (checked with external UART) and the signal on the input pin is also clear (checked with oscilloscope). It is also no performance issue (removed all components of the Firmata except for SoftSerial and got the exact same error). The reporting frequency has no influence. Increasing the receive-buffer also didn't help.

Could that be an issue with the Firmata part of the SoftSerial?

[soundanalogous]

I never finished the work in the SoftwareSerial branch so it's likely the issue is in the Firmata code.

[Draghtnod]

Ok I'll start searching in the SoftwareSerial code and send you a pull request if i find something.

[9600]

Hi @Draghtnod, thanks for letting me know!!1!

[soundanalogous]

Just know that I'm planning to change the implementation to also accommodate hardware serial (for boards that have multiple uarts). I may have time this weekend to work on it. I've also posted the most recent Firmata Serial protocol proposal here: https://github.com/firmata/protocol/blob/serial/serial-proposal.md

[Draghtnod]

Your proposal looks good. I got three things to mention: 1) What about multiple software serials? The library supports this by switching the RX port with listen(). 2) The software serial buffer is only 64 byte. Even with 9600 baud you would have to read every ~50ms to not miss any data. So adding a reporting feature would be nice to avoid hammering frequent RX requests. 3) The RX and TX pins are interchanged at new SoftwareSerial(txPin, rxPin). That was quite confusing. :) http://arduino.cc/en/Reference/SoftwareSerialConstructor

[Draghtnod]

I solved my bug. It must be somewhere in the HardwareSerial module of Arduino 1.0.1(+dfsg-7 debian arduino-core package). After upgrading to Arduino 1.6.1 everything works fine out of the box.

[soundanalogous]

Good to know it works as is. I had never actually tested the code with an actually serial device, only compiled it with no errors then got busy with other projects.

[soundanalogous]

I plan to change the draft protocol to include both a READ_ONCE and a READ_CONTINUOUS (reporting) mode. READ_CONTINUOUS would dump the serial buffer at the sampling interval (19ms by default). It would be the user's responsibility to ensure data is assembled correctly. Firmata will blindly relay the serial data to the client.

I would be possible to optionally specify a terminal char that could dump the buffer when that char is received (as in @kraman's implementation). This may be helpful for some serial devices where you're continuously reporting the same data structure. However this could just as easily be handled on the client side. In that case an additional buffer in the firmware wouldn't be necessary since HW and SW serial currently have their own buffers.

[rwaldron]

This sounds good to me, looking forward to testing :)

[damellis]

Some comments on the pull request: https://github.com/firmata/protocol/pull/28/files

SERIAL_READ goes from the Firmata client to the board, right?

The sizes of any buffers (whether send or receive) seem like they should be specified in the firmware and communicated to the Firmata client in a capability message of some sort, rather than specified by the client.

It seems like the protocol should support more than one software serial port. Also, there should probably be a way for the client to discover the number and (pin) locations of the hardware serial ports.

For sending serial data to the client, do you really need a way to do read once? Or to buffer until a particular character arrives? It seems like you could get away with just the start reading and stop reading commands, with the firmware sending available bytes while reading is enabled.

One concern might be with the jitter introduced by sending a variable number of serial data bytes each time through the loop. Maybe the host client can specify the number of bytes it wants to get on each message / loop (maybe with an "all" option)? (Then you could balance between getting the bytes quickly and slowing down everything else.)

[soundanalogous]

SERIAL_READ goes from the Firmata client to the board, right?

Correct

It seems like the protocol should support more than one software serial port.

Supporting multiple software serial ports is tricky since I need to dynamically create them rather than declare them all up front. I could declare a certain number up front regardless if the user actually intends to use SoftSerial. I'd have to see how much memory this would consume and at which point I rule out lower ram devices.

Also, there should probably be a way for the client to discover the number and (pin) locations of the hardware serial ports.

I was thinking the configuration query would handle this. There would be a single SERIAL pin type. For the first SoftSerial TX pin, the configuration query will return pin mode: SERIAL, pin resolution: STX0.

For sending serial data to the client, do you really need a way to do read once? Or to buffer until a particular character arrives?

I agree with you here and will remove READ_ONCE for the first version (so keeping just READ_CONTINUOUSLY and STOP_READING for v1). I'll keep the separate byte for specifying READ_MODE since that will enable me to add READ_ONCE in the future if it turns out to be useful.

One concern might be with the jitter introduced by sending a variable number of serial data bytes each time through the loop.

This is why I was debating relaying the serial data on each iteration of loop or at the sampling interval. If I use the sampling interval I'll need a much larger buffer in case the user is interfacing with a 56k or 115k peripheral. The advantage of sending once per loop is I can probably avoid a separate buffer, thus saving memory and keeping the feature compatible with ATmega328 boards. Your idea of enabling the user to set a number of bytes to be sent per iteration is interesting as well. It would probably have to be a specific range of values in order to avoid overflowing the serial buffer (assuming the existing 64k serial buffer, but more flexible with a separate larger buffer).

[damellis]

Yeah, I was assuming that the host couldn't request more bytes per loop than the client can buffer. You might want a way for the host to discover how big the client buffers are (for both sending and receiving). Either way, though, the client can always just send fewer actual bytes than the host has requested (since this is what would happen if fewer bytes were available). I guess you'd need to pad out the message with empty / default values to ensure a consistent message length.

Does a sampling interval help? It might make the jitter / lag less frequent but it will still happen (and probably be longer when it does).

As for multiple software serial ports, it might be good to keep them as an option in the protocol (e.g. have a bit for SW vs. HW) even if the initial implementation only has one.

[soundanalogous]

The way I have currently crafted the protocol, you can use multiple hardware ports and a mix of a single sw serial port and one or more hw serial ports. In order to use multiple software serial ports I need to add another message (SWITCH_SW_SERIAL_PORT) to switch software serial ports (in the Arduino implementation this would call the listen() method).

[soundanalogous]

I have opened a pull request for an implementation of a Firmata Serial protocol that supports both Hardware and Software serial. Please test it against some serial devices and provide any feedback on the implementation: https://github.com/firmata/arduino/pull/205.