[FEATURE] Integrate ble-receiver into live plotting

[xjjak]

The new firmware with pure BLE binary data transmission can be used with the ble-receiver script on the other end, which translates the binary values into a string. While this happens "live", unlike the ble-serial receiver it does not open a separate file/port, but prints directly to stdout. As such, it is not compatible with the current version of the live-plotting software, which is reliant on ble-serial or at least a program opening a port on /tmp/ttyBLE.

(Side note: I completely forgot about ble-serial opening up a separate port and thought a receiver like this could serve as a sort of drop-in replacement)

There are two ways to reconcile this:

1. Modify the live-plotting code to be able to plot values coming from the stdout of the process which initiates the connection
2. Modify the ble-receiver program to also open up an "endpoint" which it writes to instead or in addition to stdout. This could also be /tmp/ttyBLE maintaining more compatibility with the current version of live-plotting

[xjjak]

I tried out some glue, by modifying the live-plotting code to start the script to python ble-receiver.py -d \"~a\" > /tmp/ttyBLE. This works, but the plot is very laggy. The bottleneck may be the pipe though.

Way 2.is probably easier to do, especially by me, but it's still an open question whether a buffer like /tmp/ttyBLE should be used or the data received directly via stdout?

[xjjak]

I redirected the print in ble-receiver.py to /tmp/ttyBLE, which works well, but the plot is still jittery/laggy and shows a "parsing error". This may have to do with the transferring of the angle values as whole values instead of floats. Or is the frequency is actually too high?

[xjjak]

After adding flush=True to the print statement of the incoming data as a string, the plot seems to be more stable and the lags are gone. Without it, the prints are most likely buffered before actually being written to the disk until it's "worthwhile" to perform an actual write operation. The screen is still flashing a lot, but that is probably just the plot being redrawn.

• [ ] The parsing error remains to be fixed

[xjjak]

Regarding the parsing error: I have found the point of "error" to be:

(define (parse str)
  (cond
    [(eof-object? str)
     #f]

The incoming string is an end of file object. I don't know why that happens or why it happens at irregular intervals.

[palisn]

After adding flush=True to the print statement of the incoming data as a string, the plot seems to be more stable and the lags are gone. Without it, the prints are most likely buffered before actually being written to the disk until it's "worthwhile" to perform an actual write operation. The screen is still flashing a lot, but that is probably just the plot being redrawn.

* [ ]  The parsing error remains to be fixed

I don't know how exactly the operating system handles this kind of files, but it would probably be best to try to avoid writing to the disk all the time. I can't imagine that the disk would like that very much.

A further alternative might be to use another intermediary point that we have more control over, or we just implement the binary parsing in racket which should also not be to hard I'd assume but I've not worked with binary data in racket yet.

Regarding the parsing error: I have found the point of "error" to be:

(define (parse str)
  (cond
    [(eof-object? str)
     #f]

The incoming string is an end of file object. I don't know why that happens or why it happens at irregular iintervals.

No idea why this would happen. Sound like some improper file handling is happening but I don't know where or why.

[xjjak]

After adding flush=True to the print statement of the incoming data as a string, the plot seems to be more stable and the lags are gone. Without it, the prints are most likely buffered before actually being written to the disk until it's "worthwhile" to perform an actual write operation. The screen is still flashing a lot, but that is probably just the plot being redrawn.

* [ ]  The parsing error remains to be fixed

I don't know how exactly the operating system handles this kind of files, but it would probably be best to try to avoid writing to the disk all the time. I can't imagine that the disk would like that very much.

You're right, this is not optimal. I could also imagine that this is actually what's somehow causing the eof error. I need to see if there is a difference to a file on disk and a "port" that appears as a file, some sort of a "stream" I guess, which then perhaps wouldn't need/contain any end-of-file.

[xjjak]

A further alternative might be to use another intermediary point that we have more control over, or we just implement the binary parsing in racket which should also not be to hard I'd assume but I've not worked with binary data in racket yet.

This could be done of course, decreasing the amount of data needed to be transferred between programs, but I think some external program will still be required, as I don't think there is a BLE client yet for racket.

[palisn]

This could be done of course, decreasing the amount of data needed to be transferred between programs, but I think some external program will still be required, as I don't think there is a BLE client yet for racket.

I think there is something that could probably work for working with Bluetooth, but I wouldn't want to rely on that either. So we should probably keep the Bluetooth part in Python.

Also, I just thought of a nice way to handle the communication between Racket and Python: We could just start the Python script inside of Racket and let Racket handle the stdout directly. Or is there any technical reason this might not work, @xjjak?

[xjjak]

That should work fine. I just tried to avoid that, so the racket code wouldn't need to be modified as much. Or in ways that I could do... But I guess it's easier to just do that, reduces complexity in both programs. I'm more used to lisp style code now, so I might actually be able to do it myself if you'd like. Just wondering if the live-plotting-dev branch is up-to-date currently?

[palisn]

That should work fine. I just tried to avoid that, so the racket code wouldn't need to be modified as much. Or in ways that I could do... But I guess it's easier to just do that, reduces complexity in both programs. I'm more used to lisp style code now, so I might actually be able to do it myself if you'd like.

It should be rather straight forward to implement. If you want to you try implementing it, but if not I'll try to do it as soon as possible, might even be able to do it today.

Just wondering if the live-plotting-dev branch is up-to-date currently?

I'm currently not sure of that either :sweat_smile:, I started reorganizing the code a bunch a while ago. But I'd probably just stash that for now and work on the current live-plotting branch. As far as I remember, there shouldn't be any major changes that I didn't push.

[xjjak]

Well, I looked at the code of ble-serial again, and I think you doing it would be much quicker... 😬

[palisn]

Well, I looked at the code of ble-serial again, and I think you doing it would be much quicker... 😬

Sure, the output of the script is already stdout, right? Also, do we need to pass any command line arguments through the racket program or should there be any GUI forms? There isn't any scanning yet, right?

[xjjak]

Yeah, stdout is default. Only the MAC device address needs to be passed. It's the same as with ble-serial (-d). There is no inbuilt scan functionality yet, yes. Coming soon though...

[palisn]

If it's the same MAC address, can we just use the scanning functionality from ble-scan for now?

[xjjak]

Yes. For my testing I just replaced ble-serial with python ble-receiver.py in the currently used command string. The MACi is compatible. So I guess the scanning part and all can stay the same...