Data arriving out of order

[DanielJoyce]

Investigating the use of SPJS for talking to serial port RFID/Barcode readers. The default buffer flow is causing data to arrive out order, leading to jumbled scans.

The reader works fine when talking to cutecom over its serial port. So something is happening in SPJS somewhere.

[chilipeppr]

That is network congestion. You need to solve that in a higher layer like how chilipeppr does it by looking at per line id's

On Apr 25, 2017 12:39 PM, "Daniel" notifications@github.com wrote:

Investigating the use of SPJS for talking to serial port RFID/Barcode readers. The default buffer flow is causing data to arrive out order, leading to jumbled scans.

The reader works fine when talking to cutecom over its serial port. So something is happening in SPJS somewhere.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/chilipeppr/serial-port-json-server/issues/34, or mute the thread https://github.com/notifications/unsubscribe-auth/AHidbfrBcqt7-kyS_tPmlQUsF8y-wtQGks5rzkvWgaJpZM4NH9WB .

[DanielJoyce]

Its purely localhost so network congestion is nearly impossible.

Its not only out of order data, but dropped data as well, which should be impossible.

[DanielJoyce]

Hmm, I modified the timed buffer flow to increase the timeout to 500 millis and used that. No corruption. This looks like some kind of race condition somewhere. I don't think its network traffic issue as I am on localhost, and WS runs over TCP which enforces single arrival and sequencing.

[DanielJoyce]

Just confirmed data is garbled on GO side.

Problem appears to be here: Lines 157-175 of serialport.go

            if p.bufferwatcher.IsBufferGloballySendingBackIncomingData() == false {
                //m := SpPortMessage{"Alice", "Hello"}
                m := SpPortMessage{p.portConf.Name, data}
                //log.Print("The m obj struct is:")
                //log.Print(m)

                //b, err := json.MarshalIndent(m, "", "\t")
                b, err := json.Marshal(m)
                if err != nil {
                    log.Println(err)
                    h.broadcastSys <- []byte("Error creating json on " + p.portConf.Name + " " +
                        err.Error() + " The data we were trying to convert is: " + string(ch[:n]))
                    break
                }
                log.Print("Printing out json byte data...")
                log.Print(string(b))
                h.broadcastSys <- b
                //h.broadcastSys <- []byte("{ \"p\" : \"" + p.portConf.Name + "\", \"d\": \"" + string(ch[:n]) + "\" }\n")
            }

commenting out the surrounding if test fixes the issue for the case where the default buffer protocol is used.

//          if p.bufferwatcher.IsBufferGloballySendingBackIncomingData() == false {
                //m := SpPortMessage{"Alice", "Hello"}
                m := SpPortMessage{p.portConf.Name, data}
                //log.Print("The m obj struct is:")
                //log.Print(m)

                //b, err := json.MarshalIndent(m, "", "\t")
                b, err := json.Marshal(m)
                if err != nil {
                    log.Println(err)
                    h.broadcastSys <- []byte("Error creating json on " + p.portConf.Name + " " +
                        err.Error() + " The data we were trying to convert is: " + string(ch[:n]))
                    break
                }
                log.Print("Printing out json byte data...")
                log.Print(string(b))
                h.broadcastSys <- b
                //h.broadcastSys <- []byte("{ \"p\" : \"" + p.portConf.Name + "\", \"d\": \"" + string(ch[:n]) + "\" }\n")
//          }

[DanielJoyce]

This seems weird to me because the default buffer protocol has that test set to false, so the body should be working...

[DanielJoyce]

Ok that does seem to seriously improve things in the default bufferflow case. Logging the data looks correct. And the data looks nearly entirely correct on the WS socket end.

Still seeing weird data client side, but its extra 'noise', not dropped or reordered ws frames.

It may be the websocket wrapping lib I am using.

[DanielJoyce]

Okay, pulled out WS lib, went to raw websocket, looks like I am getting sent the trailing piece of the last message plus a little garbage.

As this comes inside a json {"P":"","D":""} object, this is obviously being sent by SPJS from somehwere

[DanielJoyce]

Weird, don't know what to believe anymore. Now its working all the time now. I even backed out my changes and it working fine now. I don't know if spjs or serial port somehow got into a bad state?

But when it was misbehaving, I restarted and rebuilt and edited the sources many many times, and it was acting wonky for a while. Like a stale file somwhere...

[DanielJoyce]

Went and did a full clean and rebuild, and even with none of the changes above, still works fine...