Best practice for use serial data over socket interface

Joey-1970 commented 6 years ago

I hope for a good idea: ;-)

...I don' t know what is the best way to read received data from serial interfaces at the right time. I can send data to the serial device, data get back from it to the serial buffer. If I know this I can get it.

But what is the best practice to know it?

I try to use the notification on GPIO 15 (RxD), if there is some activity I can look if there are some data in the serial buffer - but there is very much activity on the notification and I only need one sign - I think this can't be the best practice...

I try with the watchdog, but for this the notification is needed too, I read that the function "event" will need the same as the base of work.

But what is the best way? If I send data to the serial device I can look if there is an answer, but what should I do if I know there can be data received from the serial device without a trigger from "my" side? I think to poll every seconds cound not the best way...

Joachim

btw: what is the difference between an notification, an event an a watchdog (only the time after activity)?

Joey-1970 commented 6 years ago

...something like gpioSetISRFunc would maybe the right function for that? But it is not available for the socket interface?

Joachim

joan2937 commented 6 years ago

gpioSetISRFunc is useful from C when the absolute minimal latency is required. From the point of view of the socket interface it's pretty meaningless, any time saving would be swamped by the socket message overhead.

A notification is generated by the GPIO sampling activity. When pigpio notices a GPIO has changed state it sends a notification to each interested party.

A watchdog is also generated by the GPIO sampling activity. When pigpio notices a GPIO has not changed state for a set number of milliseconds it sends a notification to each interested party.

An event is user defined. Code has to be written in C to generate the event.

Joey-1970 commented 6 years ago

Hello Joan,

thank you for this discription. So after this, I think that use of the WatchDog will be helpful. After transfer of serial data on GPIO 15 I get a WatchDog Notification and can look if there are new data? Will be very goog if this works! I try it!

Joachim

Joey-1970 commented 6 years ago

Hello Joan,

I trry to test it with GPIO 15 with WatchDog and GPIO 17 and 27 with "normal" notification:

NOIB, then NB with handle and Bits pow(2,17) + pow(2,27), WDOG for GPIO 15 But I get no WatchDog-Event...
NOIB, then NB with handle and Bits pow(2,17) + pow(2,27) + pow(2,15), WDOG 100ms for GPIO 15 Now I get a Watchdog-Event, but the notification for GPIO 15 with the tons of level changes too...

Can't I get the WatchDog-Event only? I actually only need this one Event for a serial data transmission...

Joachim

Joey-1970 commented 6 years ago

...I don't really understand what happend there... If I set the watchdog-Timeout to 150ms, I get a mass of notifications for GPIO 15 with the flags for the watchdog-timeout too after the "normal" notifications for all the byts that has been reached from the device. I try to set the watchdog-timeout to 500ms and now I get no notifications...

Joachim

Joey-1970 commented 6 years ago

...you has predefined an event for BSC Slave-Activity on 31. It is possible predefined a event for "SERDA", when bytes are available to be read?

Joachim

joan2937 commented 6 years ago

You will need to alter pigpio to get a "SERDA" event.

I'm not sure if you are saying you have discovered a bug in the watchdog software. If you have I will need an example script which shows the failure.

Joey-1970 commented 6 years ago

Hello Joan,

thanks again for your answer!

My PIGPIO-projects (modules for ip-symcon) are now a few thousends line big (https://github.com/Joey-1970/SymconModules , https://github.com/Joey-1970/GeCoS-Modules). I have no idea how can show you in a "short" way what happend in this case....

The watchdog work in a another way as I expected (I get every x ms a watchdog-notification, I expected I get only one a the next x ms after next actifity - my mistake)

I see two ways for an efficient pratice to get information that serial data arrived:

get a PIGPIO-event if there are data in the serial buffer
something like a "interrrupt" if some activity on a special gpio, some "event" will inform with ONE notification until the user set it back manuell

Unfortunately I don't know how I can do that...

Joachim

Joey-1970 commented 6 years ago

Hello Joan,

I'm still searching for a efficient way to read the serial buffer. Maybe you please show me a example to create a suitable event for this case?

Joachim

guymcswain commented 6 years ago

@Joey-1970 , I too would like a more efficient method for reading serial data. I'm using the socket interface so efficiency from that perspective is minimizing network traffic (ie polling). May I offer a possible solution without being considered presumptuous (I'm no expert in pigpio).

From reading the documentation, it appears possible to generate your own "serial data available" (SERDA?) event using a pigpio script. Using pigs, to remain application language agnostic, define a script like the following: pigs proc tag 100 wait p0 mils p1 evt p2 jmp 100 where the parameters are: p0 - the gpio bit mask for the serial RX pin p1 - the time to wait for data to accumulate in the circular buffer p2 - the event number (0-30) that you chose for SERDA

After obtaining the script ID from the above command you would run the script: pigs procr sid, 0x100, 25, 1

This would generate 'event 1' 25 milliseconds after activity on GPIO8.

I have not tested this script but it is in my plan once I complete implementation for serial writes. Hopefully joan2937 will let us know if this idea is workable.

Joey-1970 commented 6 years ago

Hello guymcswain,

thanks for your answer! I thing it is the better idea to look for activity on GPIO15 (TxD, Pin 10) because there you see if there is some incoming data. I'm not sure, if the script do want you expect. Look for me like a watchdog? The TxD-line is normally high level, if there is traffic it goes many times high/low (for each bit that received). I think, we need one signal, when it goes the first time to low (start of script activity) and it is expected ready when it goes to high-level again for x ms (end of activity). If at that time there is a event, that maybe would be perfect! ;-)

Joachim

guymcswain commented 6 years ago

Hello Joachim,

Yes, use the gpio pin where you have incoming serial data. For GPIO15 the bit mask is 0x10000. 'wait 0x10000' looks for any transition on GPIO15 then the script goes to sleep for 25 milliseconds (mils 25). My assumption is that the socket interface can have the notifications on GPIO15 masked/blocked so that the only network traffic would be for 'event 1' every 25 milliseconds. This remains to be tested to see if pigpio behaves accordingly.

To have only one signal might be possible using a modified script but you would need to sure that in your application you would never overflow the incoming circular buffer that pigpio uses to hold data before it is read: For example, if your incoming messages never exceed a certain length before the serial bus goes idle.

By the way, does anyone know how big the circular buffer is?

Joey-1970 commented 6 years ago

Hello guymcswain,

but when there is a signal every 25 ms - this is like use the watchdog?

Once a watchdog has been started monitors of the GPIO will be triggered every timeout interval after the last GPIO activity. The watchdog expiry will be indicated by a special TIMEOUT value. <

Joachim

guymcswain commented 6 years ago

My understanding is that watchdog continues to fire as long as the bus remains idle whereas the script only fires when there is new/more activity on the bus. I could be wrong.

Example void aFunction(int gpio, int level, uint32_t tick) { printf("GPIO %d became %d at %d", gpio, level, tick); } // call aFunction whenever GPIO 4 changes state gpioSetAlertFunc(4, aFunction); // or approximately every 5 millis gpioSetWatchdog(4, 5);

In addition, you need to have an alert function registered (or a notification channel open for that gpio) to receive the watchdog. That implies you will get noise from all the serial signal edges - a problem you and I are trying to avoid.

Joey-1970 commented 6 years ago

Hello guymcswain, I'm not sure about exact function... You try your script? Do it work? One other idea: If there is aktivity (first falling edge) GPIO15, set onother GPIOx to 0. The user get the serial data from the buffer and set this GPIOx to 1 again? Or: If there is aktivity (first falling edge) GPIO15, set onother GPIOx to 0. If there is x ms no activity set GPIOx to 1? Then you can use the "normal" notification on GPIOx....

Joachim

guymcswain commented 6 years ago

I'm working on implementing the event monitoring and script APIs now. Once that is done I will begin testing the script for 'serial data available.' It will be a few days before I have any results to report. Guy

Joey-1970 commented 6 years ago

Hello Guy,

are you sure that the bit mask for GPIO15 is 0x10000? I think it is 0x8000 (2^15)?

I can set the script without a error over the socket interface (PROC t)
I can start the script over the socket interface without a error (PROCR sid par)
I can set the event to the monitor without a error (EVM h bits) ...but I get no notification. Not if there is data from the serial device, not when I Trigger the script...

Do I forget something?

Joachim

guymcswain commented 6 years ago

Hi Joachim, You are correct, my mistake on GPIO15 bit mask. But, if you cannot trigger the event programmatically as a test then something else is wrong. I've just implemented events api in my library where my first test was to verify triggering an event and receive it through notification channel. This works for me. I'm still a day or two away from implementing the 'serDA' script. Guy

joan2937 commented 6 years ago

There is an error within the pigpio scripts code which will prevent the use of event trigger (EVT). The code checks for internal/external commands by testing against command number 100 (internal commands start at 800). The constant 100 was never a great choice but didn't seem so bad when the maximum command number was in the 90s.

A temporary fix is to change a line in the pthScript function in pigpio.c (currently line 6548).

Change if (instr.p[0] < 100) to if (instr.p[0] < PI_CMD_SCRIPT).

When I have time I will tidy up scripts a little by preventing commands which have no useful functions within the script.

Joey-1970 commented 6 years ago

Hello Guy,

before I make a basic failure: What is the right way to use the noticitation (and get the "right" handle) by the socket interface:

Use NOIB (99), take the handle and set with this NB(19)? (in documention NOIB give not back the handle but it works) or
Use NOIB (99) (to get a "keep alive signal"), NO (18) and take this handle to start notification with NB (19)

Until now I use the first way...

Second question: If I send EVM h bits, it seems that I get no answer from PIGPIO, but in the socket-interface-documention (http://abyz.co.uk/rpi/pigpio/sif.html) it should be...

Joachim

guymcswain commented 6 years ago

@Joey-1970 I set up my notifications using a second network socket similar to how the pigpio Python lib works. Steps:

Open second socket (aka, the notification socket)
Issue the NOIB command on this socket and obtain the handle from response
Now send the EVM h bits command through the usual command socket
Trigger event by sending EVT eventNum to command socket

@joan2937 Thank you for this information. I'm a bit noobish building C programs outside of simple Arduino environments so I need a few pointers. I see these instructions on your download page:

To compile, link, and run a C program gcc -Wall -pthread -o foobar foobar.c -lpigpio -lrt sudo ./foobar

What else is there to know about the process? I figure I need build-essentials installed and to run all of this on the Raspberry Pi ARM cpu. Any additional help is appreciated.

I plan to do this:

sudo rm -rf PIGPIO
wget abyz.co.uk/rpi/pigpio/pigpio.tar
tar xf pigpio.tar
cd PIGPIO
edit teh codez
make
sudo make install

guymcswain commented 6 years ago

A temporary fix is to change a line in the pthScript function in pigpio.c (currently line 6548). Change if (instr.p[0] < 100) to if (instr.p[0] < PI_CMD_SCRIPT).

Found it on line 6541. Very easy build process! Now onward to testing.

Just ran a script that was simply tag 100 mils p0 evt p1 halt with parameters 100, 1. My test listens for event 1. Success!

Joey-1970 commented 6 years ago

Hello guy,

current I have the problem that I can't start the script - I always get back an error (something like "PI_BAD_SCRIPT_ID"). :-( Until now I don't understand way... I get back from PROC the ScriptNumber 0, but when I want to start it with this ScriptID I get this error.

This "new" script is only to test the event?

Joachim

guymcswain commented 6 years ago

@joan2937 🥇 I'm able to run my pigpio script and trigger events. I have a question, however. At some point I stop the script using PROCD sid then immediately follow with PROCP sid which returns the status of the script as running (2). Do you expect the status to be halted (1)? It's a really not an issue for me but just wanted to point this out to you.

guymcswain commented 6 years ago

@joan2937 , When I run my test with notifications turned on, the results seem as expected. But when I turn off notifications, leaving only the events registered, I'm seeing unexpected results.

This is my console when running both notifications and events:

request= 4 0 0 0 25 0 0 0 1 0 0 0 0 0 0 0 #write gpio25 to 1 response= 4 0 0 0 25 0 0 0 1 0 0 0 0 0 0 0 notification received: chunk size = 12 GPIO25=1 at 1340967029 notification received: chunk size = 12 got event #bit 7 of flags is set serDA event fired! #listener attached to this event request= 4 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 #write gpio25 to 0 response= 4 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 notification received: chunk size = 12 GPIO25=0 at 1341072824 notification received: chunk size = 12 got event serDA event fired! request= 41 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 #ending my test with stop script response= 41 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

... and this is the console running only events:

request= 4 0 0 0 25 0 0 0 1 0 0 0 0 0 0 0 response= 4 0 0 0 25 0 0 0 1 0 0 0 0 0 0 0 notification received: chunk size = 12 got event serDA event fired! notification received: chunk size = 24 got event serDA event fired! got event serDA event fired! request= 4 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 response= 4 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 notification received: chunk size = 12 got event serDA event fired! notification received: chunk size = 12 got event serDA event fired! request= 41 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 response= 41 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Recall the serDA script is tag 100 wait p0 mils p1 evt p2 jmp 100 and I'm running it with procr sid 0x2000000 25 1 and since the edges on gpio25 are spaced 100 milliseconds apart, I easily expect to see an event for each edge.

guymcswain commented 6 years ago

@joan2937
Additional observations with script parameter p1 >> gpio pulse width:

wait x behaves as if it is level sensitive in the absence of notifications enabled. For a narrow width negative pulse to gpio pin, start receiving events at p1 intervals before and after the pulse. For the case of a positive pulse, only receive a single event as expected.
With notifications and events enabled, I observe one event for both positive and negative pulses. But interestingly, on the positive pulse, only the falling edge receives a notification.

Again, the desired outcome is to not have any notifications but then wait x misbehaves without having them turned on.

guymcswain commented 6 years ago

Am I to be using email to discuss this script issue with you and Joachim instead of the GitHub issues thread? I posted some more observations from my testing.

On Thu, Sep 7, 2017 at 11:29 AM, joan2937 notifications@github.com wrote:

A script enters halted state when it executes the last instruction or it executes the halt command. It will be halted until it is started again. I think your script is a continuous loop so it will be running but waiting for a stimulus.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/joan2937/pigpio/issues/148#issuecomment-327851103, or mute the thread https://github.com/notifications/unsubscribe-auth/AcZ5jJpNQ-qPQlmeeyNJkpF6kKosXbscks5sgBhFgaJpZM4O8Trc .

joan2937 commented 6 years ago

@guymcswain Here is best, in the sense that I will check here. I purge e-mail regularly whether responded to or not. Neither method will get a guaranteed response, it depends on what I am doing, and at the moment that isn't pigpio.

Joey-1970 commented 6 years ago

Hello guy,

I had to fix a little timing-problem...

So I start:

Script = "tag 999 wait p0 mils p1 evt p2 jmp 999";
PROC -> get back ScriptID
Parameter = array(32768, 50, 1) -> GPIO 15 (TxD), 50ms, Event 1)
PROCR -> get back status (being initialised)
EVM -> Handle from NOIB, 1, 0 Look like it all work good, but I get no Event when there is traffic on GPIO 15. When I use "pigs evt 0" via SSH I get a event with Event Number 0... (Bit 0-4) in my script. What is wrong? (I don't make the change that joan posted)

Joachim

guymcswain commented 6 years ago

Hi Joachim, according to Joan you don't stand a chance of getting the script to work unless you make the change to file pigpio.c. This is very easy to do just follow the steps outlined on http://abyz.co.uk/rpi/pigpio/download.html

The problem now, however, is the script will continuously issue events while the serial line is in the high state and idle. If, somehow, you can provide the inverted serial signal to pigpio, then my testing shows the script should work.

Joey-1970 commented 6 years ago

Okay, that is an important information! ;-)

Your script work now like a watchdog? When the end of the script is not a "jmp" and we would stop it after the event and restart it after get data from the serial buffer?

Joachim

guymcswain commented 6 years ago

The serial data available script, in its current state does not work. I need to investigate the wait x command further.

guymcswain commented 6 years ago

@joan2937 Just fyi as I know you are busy. In looking at pigpio.c, in functions scrWait()and pthAlertThread() I'm seeing a potential conflict in the shared global monitorBits. I'm thinking there may need to be a mutex lock on this variable. But I'm not confident enough in my knowledge of pigpio.c or C in general to offer a patch.

guymcswain commented 6 years ago

@Joey-1970 Hello Joachim,

I have abandoned the serda script effort, at least for now. Instead, I decided to look at implementing the serial data available functionality within pigpio.c. I was pleased to see how cleanly this can be done.

So, I forked the pigpio repo and made changes to provide a 'SERDA' event. I ran my tests on it and it is passing. I ran the pigpio 'x_*' tests and all pass so don't think I broke anything. I'm requesting that you try it and let me know if it works in your environment. Follow these steps:

sudo killall pigpiod
cd ~ #or move to some other parent directory
git clone https://github.com/guymcswain/pigpio.git
cd pigpio
git checkout serda
make
sudo make install
sudo pigpiod

You'll need to modify your client application - showing pigs pseudo code here:

EVM h (1<<gpio) # the same gpio you will open for serial read
attach your serial read listener/callback function to event number gpio
SLRO gpio baud dataBits
send data to the gpio pin

Guy

Joey-1970 commented 6 years ago

Hello Guy,

many thanks for sharing this and to let me part of it! I hope I can try it this evening and will give a feedback.

Works this only with SLRO or with the "normal" SERO (GPIO 14/15) too?

Joachim

guymcswain commented 6 years ago

Yes, SLRO only.

I have not used SERO but SERDA is available if you are using the hardware device. SERO and related functions are just wrappers around the ioctl function calls. So there is not much that pigpio can do with those calls. Perhaps you hook an interrupt in this case? I don't really know.

Joey-1970 commented 6 years ago

Hello Guy,

yesterday I only install "your" PIGPIO-Version, looks to be working as usually. ;-) The changes in my applications are bigger then I first expected, so I will need a little time, but I see a advantage for the user of my application too.

I hope, that after a positiv test, that Joan will take it in the stable version?

Joachim

guymcswain commented 6 years ago

You can limit your changes to just the receive side using SLRO. For example, if you wanted to loop-back read the data you have written on TXD (GPIO14), by the SERW method, just transmit the usual way but set up the read using SLRO 14 baud dataBits. I will test this on my side as well.

I hope, that after a positiv test, that Joan will take it in the stable version?

That is the goal. But after extensively testing. I would then need help from Joan to fix a compiler warning.

Joey-1970 commented 6 years ago

Hello Guy,

I finally change some functions in my application and I get some events when there is traffic - but a lot of events... I thought there is one event when there is traffic until a fetch the buffer. When should it stop?

Joachim

guymcswain commented 6 years ago

Hello Joachim, Thanks for the feedback.
I just found the same problem when sending large(r) amounts of data. I didn't see this in my small test. Maybe I was ending my test too early. I'm working to make a patch as soon as possible. Guy

Joey-1970 commented 6 years ago

Hello Guy,

my "favorite" would be, if

there is a "internal" notification in your function that when there is traffic on RxD (normal level 1 went to 0)
when there is x ms a "stable" level 1 again, then the RxD broadcast is ready and then send ONE (external) event...(Now the user can get fetch the data in buffer)
the loop started new, if there is traffic on RxD again

Would this be possible?

Joachim

guymcswain commented 6 years ago

There are several considerations that I believe are important:

Buffer overrun. If incoming message is large we must trigger the 'ready' event before this happens. I'm using a high water mark of 1/2 the buffer size for this trigger. (Currently 4096 bytes)
First byte latency. Applications that rely on short messages but require a timely response. (Maybe this is how your application is characterized?)
Minimize resources and performance impact. Any logic added for generating the 'ready' event will add a parasitic load that will slow pigpio's performance. My implementation already uses one timer. If more features are desired this may require more timers!

To satisfy both 1 and 2 requires that your software must handle more than a single 'ready' event. Also, I don't believe pigpio should handle everything that is 'inconvenient' for the application.

guymcswain commented 6 years ago

Hello Joachim, I just pushed an updated serda branch with fixes to pigpio.c. There are performance issues - works with baud rates up to 57600, fails at 115200. Let's continue our discussion here until we get satisfactory results. Guy

Joey-1970 commented 6 years ago

Hello Guy,

I could do aa little test of the new version. I install it but the result suprise me. I use a nextion display at 9600 Baud as device. At the start I send a lot of data to the display from my applcation to configure and to read some device-information from the display, but for this I see no event. As I test a touch on the display, I see a result - ONE result!;-) But why this differences? (Don't know where we can discuss in your link...) Joachim

guymcswain commented 6 years ago

Its probably a bug 🐛 ! Tell me more: At first you write then read without waiting for event. Then later you wait for the event before read?

(Apparently I don't know how to use GitHub to make an issues list on a forked repo.)

Found it! You can comment on a commit to the forked repo here. (Bottom of page)

Joey-1970 commented 6 years ago

Hello Guy, sorry for delay... First: I think your function work very well! If I touch the display or do something there, I get a event and get the data.

But: I have some problems in my application to send serial data to the Nextion Display. I send it with WVAS, I get no error from PIGPIO but no reaction/response for it from the device. Musst I do something more to send data? Do you use WVAS too? Must I configure the GPIO to Output or something else? I try sending data now for the last days - but without response...

Joachim

guymcswain commented 6 years ago

Hello Joachim, Please see my response here. Guy

guymcswain commented 4 years ago

Will close this for now until an experienced pigpio developer volunteers to take a shot at implementing.

joan2937 / pigpio

Best practice for use serial data over socket interface #148