search

chipKIT32 / chipKIT-core

Downloadable chipKIT core for use with Arduino 1.6 - 1.8+ IDE, PlatformIO, and UECIDE
http://chipkit.net/
Apache License 2.0
59 stars 53 forks source link

UART hardware TX FIFO too small #326

Open VigibotDev opened 7 years ago

VigibotDev commented 7 years ago

I need to TX a large burst of (lidar) data inside a non blocking project.

The Arduino core code contains a nice little round robin data buffer where you can keep throwing data at it and the arduino code will read the data and process it in order.

HardwareSerial.cpp : #define SERIAL_BUFFER_SIZE 256 And it work.

On PIC32 core I discovered this inside HardwareSerial.cpp :

size_t HardwareSerial::write(uint8_t theChar)
{

    while ((uart->uxSta.reg & (1 << _UARTSTA_UTXBF)) != 0)  //check the UTXBF bit
  {
        //* wait for the transmitter buffer to have room
    }

    uart->uxTx.reg = theChar;
    return 1;
}

A bad blocking while loop freezing my code :(

There is a way to do like Arduino on ChipKit core ?

I found an implementation here : http://phoenix-mindgame.blogspot.fr/2014/09/pic32-uart-transmission-using-tx.html

But I need help to put inside the core code.

Inside pUART.c void PutStrUART(char *pcString)

VigibotDev commented 7 years ago

With framework serial, correct TDD communication is:

ModemRX | $R...Ws........àQÞN | 24 52 BC 00 00 57 73 0A 05 00 00 80 00 80 00 E0 51 DE 4E | 036 082 188 000 000 087 115 010 005 000 000 128 000 128 000 224 081 222 078
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $R...Wc........áQÞ[ | 24 52 BD 00 00 57 63 0A 06 00 00 80 00 80 00 E1 51 DE 5B | 036 082 189 000 000 087 099 010 006 000 000 128 000 128 000 225 081 222 091
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $R...Wã........àRÝR | 24 52 BC 00 00 57 E3 0A 04 00 00 80 00 80 00 E0 52 DD 52 | 036 082 188 000 000 087 227 010 004 000 000 128 000 128 000 224 082 221 082
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $R...Wn........álÝ. | 24 52 BD 00 00 57 6E 0A 06 00 00 80 00 80 00 E1 6C DD 8F | 036 082 189 000 000 087 110 010 006 000 000 128 000 128 000 225 108 221 143
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $R...X.........àlÞh | 24 52 BF 00 00 58 0E 0A 01 00 00 80 00 80 00 E0 6C DE 68 | 036 082 191 000 000 088 014 010 001 000 000 128 000 128 000 224 108 222 104
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $RÀ..W.........àqÜg | 24 52 C0 00 00 57 B5 0A 03 00 00 80 00 80 00 E0 71 DC 67 | 036 082 192 000 000 087 181 010 003 000 000 128 000 128 000 224 113 220 103
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $R...Wg........àeÝ. | 24 52 BF 00 00 57 67 0A 06 00 00 80 00 80 00 E0 65 DD 90 | 036 082 191 000 000 087 103 010 006 000 000 128 000 128 000 224 101 221 144
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $RÀ..W7........àKÞH | 24 52 C0 00 00 57 37 0A 09 00 00 80 00 80 00 E0 4B DE 48 | 036 082 192 000 000 087 055 010 009 000 000 128 000 128 000 224 075 222 072
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $RÀ..X.........àJÞM | 24 52 C0 00 00 58 09 0A 03 00 00 80 00 80 00 E0 4A DE 4D | 036 082 192 000 000 088 009 010 003 000 000 128 000 128 000 224 074 222 077
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $R...Wý........àRÞN | 24 52 BF 00 00 57 FD 0A 03 00 00 80 00 80 00 E0 52 DE 4E | 036 082 191 000 000 087 253 010 003 000 000 128 000 128 000 224 082 222 078
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $R...X.........àWÝ^ | 24 52 BF 00 00 58 04 0A 03 00 00 80 00 80 00 E0 57 DD 5E | 036 082 191 000 000 088 004 010 003 000 000 128 000 128 000 224 087 221 094
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $R...W+........àSÞX | 24 52 BD 00 00 57 2B 0A 09 00 00 80 00 80 00 E0 53 DE 58 | 036 082 189 000 000 087 043 010 009 000 000 128 000 128 000 224 083 222 088
ModemTX | $S.ø.ÿ....ì.. | 24 53 01 F8 01 FF 03 80 80 80 EC 05 0A | 036 083 001 248 001 255 003 128 128 128 236 005 010
ModemRX | $R...Wi........àQÞ_ | 24 52 BC 00 00 57 69 0A 06 00 00 80 00 80 00 E0 51 DE 5F | 036 082 188 000 000 087 105 010 006 000 000 128 000 128 000 224 081 222 095
majenkotech commented 7 years ago

Any chance you could provide an MCVE for me to test with?

VigibotDev commented 7 years ago

To do a good MCVE we need a RF modem like mine. This is a HM-TRP (used on 3DR Radio) with a custom firmware from me.

The firmware wait for UART data and transmit over the air if there is data inside the buffer and no more data coming for more than a millisecond

It's easy to make a mock-modem with another board like Arduino.

VigibotDev commented 7 years ago

Why there is a hole from >1ms inside your TX ?

majenkotech commented 7 years ago

A hole? how do you mean?

VigibotDev commented 7 years ago

On the modem :

if(serial_len && timer2_tick() - serial_time > SERIAL_TIMEOUT_TICKS) { transmit...

VigibotDev commented 7 years ago

On the PIC32 :

void writeModem() {
 uint8_t v8;
 uint16_t theta16;
 uint16_t odometriex16;
 uint16_t odometriey16;

 v8 = map(max(v, VBATMIN), VBATMIN, VBATMAX, 0, 255);
 theta16 = int(theta / (2.0 * PI) * 65536.0);
 odometriex16 = int(odometriex * 1000.0) + 0x8000;
 odometriey16 = int(odometriey * 1000.0) + 0x8000;

 MODEM.write("$R");
 MODEM.write(v8);
 MODEM.write(pwmMax);
 MODEM.write(erreurMax);

 for(uint8_t i = 0; i < NBMESURESTX; i++) {
  MODEM.write(angles_q6[i] >> 8);
  MODEM.write(angles_q6[i] & 0xFF);
  MODEM.write(distances_q2[i] >> 8);
  MODEM.write(distances_q2[i] & 0xFF);
 }

 MODEM.write(theta16 >> 8);
 MODEM.write(theta16 & 0xFF);
 MODEM.write(odometriex16 >> 8);
 MODEM.write(odometriex16 & 0xFF);
 MODEM.write(odometriey16 >> 8);
 MODEM.write(odometriey16 & 0xFF);
 MODEM.write(TXCHANNEL);
 MODEM.write(RXCHANNEL);
}
VigibotDev commented 7 years ago

MODEM.write for the framework serial. I replace MODEM.write("$R"); with txBuffer.write('$'); txBuffer.write('R'); etc...

VigibotDev commented 7 years ago

I think there is a 1 byte missing or lasting from previous TX, when it do a new TX. You can't show it because I think you use Arduino terminal with hidden return invisible char. We must test your code with binary data and RX with a hex editor.

majenkotech commented 7 years ago

Nope, not seeing it at all. And no, I'm not using the Arduino IDE's shoddy excuse for a serial terminal. I'm using a logic analyser to interrogate the waveform (like Trump waterboarding an innocent Muslim).

noloss

There I am sending the same $R... packet (from your example output above) over and over again with a 2ms delay between them - using the example code I posted above.

It could be that your TX interrupt is being interrupted by another interrupt (UART4 has a lower "natural" priority than UARTs 1-3), so you could try increasing the priority from 2 (which all UARTs are on by default) to something higher (up to 7, which is maximum priority).

VigibotDev commented 7 years ago

I make a lot of test with and without modem. The bug is here only with my modem.

VigibotDev commented 7 years ago

I get the same problem with UART4 priority of 7 !!!

VigibotDev commented 7 years ago

Also there is never 1ms blocking inside my code

majenkotech commented 7 years ago

If I remove the delay so it completely spams and floods the txBuffer I still get perfectly formed smooth transmissions. No delay at all between packets, no corruption, everything nice and tight and perfect. One packet takes 1638µs to transmit at 115200 baud, and the next is sent immediately with no extra or missing bytes.

What is this mystery modem you are using?

VigibotDev commented 7 years ago

Clone of HM-TRP from HopeRF http://www.hoperf.com/upload/rf/HM-TRP.pdf This clone do 500mW vs. 100mW for the hopeRF one. The firmware is my fork from 3DR radio (dronecode) but with a lof of simplification. Two last byte set the TX and RX channel, not showed trace.

VigibotDev commented 7 years ago

I don't understand why I never get buggy with framework serial on PIC32 or Arduino

majenkotech commented 7 years ago

Nor me. I get exactly the same results regardless of which method I use - precisely the same timing and everything.

VigibotDev commented 7 years ago

This is not the modem there is a missing byte inside the frame !!! I transmit 17 bytes I get only 16 byte !!! 24 52 A9 00 00 00 00 00 00 CC 80 00 7F EA 0A 05 Last two byte are mandatory and untouched this is why the problem is difficult to understand.

There is a problem when I use Pipe ! I have only to find the problematic line in writeModem()

majenkotech commented 7 years ago

I see it! You are not alone!

I am transmitting 

....0x05 0x00 0x00 0x80 0x00 0x80 0x00....

Yet I am seeing:

....0x05 0x00 0x80 0x00 0x80 0x00....
majenkotech commented 7 years ago

I don't think it's pipe - I think it's the TX interrupt and the "kick" function getting confused.

Change kick() so it reads:

void kick() {
    if (!txRunning) {
        txRunning = true;
        IFS2bits.U4TXIF = 1;
        setIntEnable(_UART4_TX_IRQ);
    }    
}

Instead of sending a byte on the first kick we just turn the interrupt on and tell it to trigger - that then makes the interrupt the only place that ever reads from the pipe.

VigibotDev commented 7 years ago

you get it. this is a vicious byte lost in the middle of frame that explain all. I test it

VigibotDev commented 7 years ago

It run perfectly now.

majenkotech commented 7 years ago

Yay! That really was, as you say, vicious. I should have noticed there were only 18 bytes in my output instead of 19... I only looked at the first and last bytes and saw they were the same... :/

VigibotDev commented 7 years ago

LOL :D now my home robot do live Lidar mapping here : http://www.serveurperso.com/?page=robot

majenkotech commented 7 years ago

Oh that's amazing! I love it!!!!

VigibotDev commented 7 years ago

There are still tons of things to do to improve it :)

VigibotDev commented 7 years ago

Now UART communication is good I just need a working I2C to add IMU on yaw odometry !!!!

majenkotech commented 7 years ago

You need to get to grips with DTWI for that. It's possible to get that going completely non-blocking, but it's a bit of a pig to decipher Keith's examples.

VigibotDev commented 7 years ago

for performance I2C is crap SPI and UART rulez !

VigibotDev commented 7 years ago

I get few strange bug on my robot. very short but dangerous random problem on many I/O

majenkotech commented 7 years ago

Interference on signals?

VigibotDev commented 7 years ago

I get very rare, very short, and violent reaction on motors. I use core timer interrupt to read encoders.

majenkotech commented 7 years ago

I've always been a bit iffy about using the CT. I prefer to configure a separate timer instead so it's a completely distinct interrupt from anything else with predictable properties and results.

If you want to try that I (of course) have a library ;)

VigibotDev commented 7 years ago

I try it to replace my CT

VigibotDev commented 7 years ago

I want run this at 20KHz

uint32_t encodeurs(uint32_t time) {
 uint8_t avg;
 uint8_t arg;
 uint8_t avd;
 uint8_t ard;
 static uint8_t oldAvg = 0;
 static uint8_t oldArg = 0;
 static uint8_t oldAvd = 0;
 static uint8_t oldArd = 0;
 static const int8_t encodeurs[] = {
   0, -1,  1,  0,
   1,  0,  0, -1,
  -1,  0,  0,  1,
   0,  1, -1,  0
 };

 avg = ENCODEURSPIN & 3;
 arg = ENCODEURSPIN >> 2 & 3;
 avd = ENCODEURSPIN >> 4 & 3;
 ard = ENCODEURSPIN >> 6 & 3;

 encodeurAvg += encodeurs[(oldAvg << 2) + avg];
 encodeurArg += encodeurs[(oldArg << 2) + arg];
 encodeurAvd += encodeurs[(oldAvd << 2) + avd];
 encodeurArd += encodeurs[(oldArd << 2) + ard];

 oldAvg = avg;
 oldArg = arg;
 oldAvd = avd;
 oldArd = ard;
}
VigibotDev commented 7 years ago

I forgot to "volatile" my shared vars !

VigibotDev commented 7 years ago

Your lib is adopted easy and work perfectly.

I discovered the problem : there is rare RX data corruption with Piped UART

majenkotech commented 7 years ago

Do you have the latest pipe version that has both head and tail as volatile? Or are you just overrunning the buffer with to much data?

On Sun, 29 Jan 2017, 13:29 Pascal, notifications@github.com wrote:

Your lib is adopted easy and work perfectly.

I discovered the problem : there is rare RX data corruption with Piped UART

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/chipKIT32/chipKIT-core/issues/326#issuecomment-275913753, or mute the thread https://github.com/notifications/unsubscribe-auth/ADouHCg0LJqnOOgblMPh0gyOBtx3a9Djks5rXJQggaJpZM4LwgiM .

-- Matt Jenkins Majenko Technologies http://stores.ebay.co.uk/Majenko-Technologies

VigibotDev commented 7 years ago

I cloned from your repo since my first test of your example I have the "Made head volatile" commit

VigibotDev commented 7 years ago

overrunning my RX buffer is not possible, I "don't need" buffer for RX because I do asynchronous reading very very often :

bool readModem() {
 uint8_t current;
 static uint8_t n = 0;
 static uint8_t frame[RXFRAMESIZE];

 while(rxBuffer.available()) {
  current = rxBuffer.read();

  switch(n) {

   case 0:
    if(current == '$')
     n = 1;
    break;

   case 1:
    if(current == 'S')
     n = 2;
    else
     n = 0;
    break;

   case 2: case 3: case 4: case 5:
   case 6: case 7: case 8: case 9:
    frame[n++] = current;
    break;

   case 10:
    z = frame[6];
    //servosSelect = z == CAMOUTIL1 || z == CAMOUTIL2;
    servosSelect = 0;
    x[servosSelect] = (frame[2] << 8) + frame[3];
    y[servosSelect] = (frame[4] << 8) + frame[5];
    vx = frame[7];
    vy = frame[8];
    vz = frame[9];
    flags = current;
    n = 0;
    return true;

  }
 }

 return false;
}
majenkotech commented 7 years ago

Can you identify the form of the corruption?

On Sun, 29 Jan 2017, 13:41 Pascal, notifications@github.com wrote:

overrunning my RX buffer is not possible, I "don't need" buffer for RX because I do asynchronous reading very very often :

bool readModem() { uint8_t current; static uint8_t n = 0; static uint8_t frame[RXFRAMESIZE];

while(rxBuffer.available()) { current = rxBuffer.read();

switch(n) {

case 0: if(current == '$') n = 1; break;

case 1: if(current == 'S') n = 2; else n = 0; break;

case 2: case 3: case 4: case 5: case 6: case 7: case 8: case 9: frame[n++] = current; break;

case 10: z = frame[6]; //servosSelect = z == CAMOUTIL1 || z == CAMOUTIL2; servosSelect = 0; x[servosSelect] = (frame[2] << 8) + frame[3]; y[servosSelect] = (frame[4] << 8) + frame[5]; vx = frame[7]; vy = frame[8]; vz = frame[9]; flags = current; n = 0; return true;

} }

return false; }

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/chipKIT32/chipKIT-core/issues/326#issuecomment-275914448, or mute the thread https://github.com/notifications/unsubscribe-auth/ADouHHVJufAB1jE85uOHOV5RaT5Y6D6Nks5rXJcggaJpZM4LwgiM .

-- Matt Jenkins Majenko Technologies http://stores.ebay.co.uk/Majenko-Technologies

VigibotDev commented 7 years ago

This is very hard because when I get one corrupt frame the next frame is good at 20Hz. And it's very rare. RX frame are short.

VigibotDev commented 7 years ago

The radio layer is secured with CRC16 and never provide a bad frame on serial side. And I never and never get 1 frame of bad data with framework serial.

If there is a bad frame my robot can power off/on all subsystems (1 byte flag for 8 subsystems power supply) and also move dynamixel servo (secured with software clamps)

I understood the problem by seeing the servo motors move quickly and return to their place.

VigibotDev commented 7 years ago

I use priority 2 for interrupt like your code.

majenkotech commented 7 years ago

Instead of piping the RX bytes you could directly interpret them in the interrupt routine...

On Sun, 29 Jan 2017, 13:48 Pascal, notifications@github.com wrote:

The radio layer is secured with CRC16 and never provide a bad frame on serial side. And I never and never get 1 frame of bad data with framework serial.

If there is a bad frame my robot can power off/on all subsystems (1 byte flag for 8 subsystems power supply) and also move dynamixel servo (secured with software clamps)

I understood the problem by seeing the servo motors move quickly and return to their place.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/chipKIT32/chipKIT-core/issues/326#issuecomment-275914867, or mute the thread https://github.com/notifications/unsubscribe-auth/ADouHNwkYRjOBCZJzCw5Mf3mXYJy1e_Eks5rXJizgaJpZM4LwgiM .

-- Matt Jenkins Majenko Technologies http://stores.ebay.co.uk/Majenko-Technologies

VigibotDev commented 7 years ago

Yes I don't need the piping because hardware buffer is good for my implementation.

VigibotDev commented 7 years ago

After reflection I prefer to keep the pipe ! The problem won't appear with your timer lib

VigibotDev commented 7 years ago

you can update your UART sample on Pipe because it's a must have on pic32.

majenkotech commented 7 years ago

I shall do. I will also morph it into a powerful UART library.

On Sun, 29 Jan 2017, 14:03 Pascal, notifications@github.com wrote:

you can update your UART sample on Pipe because it's a must have on pic32.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/chipKIT32/chipKIT-core/issues/326#issuecomment-275915773, or mute the thread https://github.com/notifications/unsubscribe-auth/ADouHOTI2-TcvbmL1dvWnssvRtf1m9raks5rXJwrgaJpZM4LwgiM .

-- Matt Jenkins Majenko Technologies http://stores.ebay.co.uk/Majenko-Technologies

VigibotDev commented 7 years ago

Oops I just get 1 RX corrupt frame