search

micropython / micropython-esp32

Old port of MicroPython to the ESP32 -- new port is at https://github.com/micropython/micropython
MIT License
678 stars 218 forks source link

Implement Bluetooth BLE -- RFC on API #68

Closed MrSurly closed 7 years ago

MrSurly commented 7 years ago

Putting here to track BLE support, and to gather comments for API implementation.

MrSurly commented 7 years ago

What I have so far:

In general, the default values are shown in these examples.

Start the BT stack

import network
b = network.Bluetooth()

Create a service, add characteristic

service_uuid = bytearray(range(16))
characteristic_uuid = bytearray([x * 3 for x in range(16)])

s = b.service(service_uuid, isPrimary = True)
s.add_characteristic(
    characteristic_uuid,
    permissions = b.PERM_READ | b.PERM_WRITE,
    properties = b.PROP_READ | b.PROP_WRITE | b.PROP_NOTIFY)
Permissions for characteristics:

These are constants found in class Bluetooth

Properties for characteristics

These are constants found in class Bluetooth

Begin advertising

b.ble_settings(
    int_min = 1280,
    int_max = 1280,
    type b.ADV_TYPE_IND,
    own_addr_type = b.BLE_ADDR_TYPE_PUBLIC,
    peer_addr = bytearray([0 for x in range(6)]),
    peer_addr_type = b.BLE_ADDR_TYPE_PUBLIC,
    channel_map = b.ADV_CHNL_ALL,
    filter_policy = b.ADV_FILTER_ALLOW_SCAN_ANY_CON_ANY,
    adv_is_scan_rsp = False,
    adv_dev_name = None,
    adv_man_name = None,
    adv_inc_txpower = False,
    adv_int_min = 1280,
    adv_int_max = 1280,
    adv_appearance = 0,
    adv_uuid = None,
    adv_flags = 0)

b.adv_enable(True)

Still working on:

MrSurly commented 7 years ago

@dpgeorge @mahanet

pfalcon commented 7 years ago

"network" module is unlikely a good place for native Bluetooth stuff. While not spelled explicitly, "network" is intended for IP-based networks. So, it may be a good place for 6LoWPAN support over BLE, but stuffing everything at all there doesn't sound right.

On the other hand, using "socket" paradigm for Bluetooth communication sounds interesting, but going that way would require extensive review of the prior art (e.g. how it works in Linux) to design good API.

MrSurly commented 7 years ago

Current progress at: https://github.com/MrSurly/micropython-esp32/tree/dev-bluetooth

It's a mess; still sorting out learning µPy, and BT.

mahanet commented 7 years ago

@MrSurly your progress gives me hope :+1:

So yeah a mess but couldn't help trying it out. After flashing I tried the example (I believe you were suggesting a concept more than a working example). I think those are some typos- Service instead of service equal sign after type= (in call to ble_settings) adv_enable is present as ble_adv_enable

Apart from these, add_characteristic didn't accept keyword args.

MrSurly commented 7 years ago

Yeah, it's pretty broken; I'm only pushing commits for the sake of off-site backup.

I'm working on it right now.

On Tue, Apr 18, 2017 at 3:05 PM, mahanet notifications@github.com wrote:

@MrSurly https://github.com/MrSurly your progress gives me hope 👍

So yeah a mess but couldn't help trying it out. After flashing I tried the example (I believe you were suggesting a concept more than a working example). I think those are some typos- Service instead of service equal sign after type= (in call to ble_settings) adv_enable is present as ble_adv_enable

Apart from these, add_characteristic didn't accept keyword args.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/micropython/micropython-esp32/issues/68#issuecomment-294998003, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYHCPvnfSD9SS-PEZxvNvsfESqqhE0xks5rxTPDgaJpZM4M3aEm .

filipeaguiar22 commented 7 years ago

Hi MrSurly,

I am interested in your progress on the Classic Bluetooth. I would like to contribute for the effort, but I will need some guidance. Would you be interested in having some help? Filipe

MrSurly commented 7 years ago

I am interested in your progress on the Classic Bluetooth. I would like to contribute for the effort, but I will need some guidance. Would you be interested in having some help?

@filipeaguiar22

Sure. Right now, I'm just focusing on BLE GATTS. How would you like to proceed? Have you already begun working on BT Classic?

MrSurly commented 7 years ago

Update: (comments welcome)

Right now, you can use the Bluetooth (singleton) object to create Service objects.

Service objects have an associated internal list/tuple called chars, which is the characteristics associated with the service. Char(acteristic) objects can be created separately, and attached to a Service by updating Service.chars. You need to call Service.stop() then Service.start() if you update characteristics.

Current problem

There is a problem in the implementation detail. The IDF works on callbacks, so when you start a service, in the callback, the correct Service object must be referenced, to update the handle, etc.

I could easily keep an internal reference cache to all the created Service objects, so that I can look them up in the callback, but what if the Service object in question is no longer in use in user-land Python code? This led to my recent question in the forum: "weak" reference to python object?, since weak references would solve this problem neatly.

Currently working, at an API level, if not quite yet internally:

import network as n
b = n.Bluetooth()
s1 = b.Service(uuid=0xaabb)
s2 = b.Service(uuid=0xaacc)
c1 = b.Char(0xcc)
c2 = b.Char(0xdd)
s1.chars = [c1]
s2.chars= [c2]
[s.start() for s in (s1,s2) ]
pfalcon commented 7 years ago

@MrSurly : You're trying to create an API and its usage model foreign to MicroPython principles. MicroPython uses simple and obvious explicit resource management model, where resource is allocated in constructor and freed in an explicit method call, usually named .close(). If users don't call .close() when they finished working with a resource, they want a resource leak, and of course, they get what they want. Weak references, singletons for dynamically allocated objects - nothing of this is needed. The only programmer's responsibility is to make sure that object registered with external services is referenced from the memory area GC-scanned by uPy. When you can't ensure scanning of external services' memory, you indeed need to keep a container for the sole purpose of keeping the objects live, but the simplest container is a list, not a set.

Weakref support "discussion" (which is again, completely unneeded for the usecase) is at the expected place (which is search results in the main project): https://github.com/micropython/micropython/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20weakref

pfalcon commented 7 years ago

Well, uPy supports destructors (__del__ method) and its usage is actually a good alternative to keeping an explicit "liveness" container. So, instead of keeping reference to unclosed object forever, we actually close it automatically when it's GCed. That's how modlwip works - I almost had that "d'oh" moment wondering if we had liveness issues with it all this time.

But that's again object liveness management measure, not resource management measure - for I/O resources, relying on desctructors is a bad idea because it's not guaranteed when it will be called (or that it will be called at all). Again, explicit .close() method is the only viable solution.

MrSurly commented 7 years ago

You're trying to create an API and its usage model foreign to MicroPython principles. MicroPython uses simple and obvious explicit resource management model, where resource is allocated in constructor and freed in an explicit method call, usually named .close(). If users don't call .close() when they finished working with a resource, they want a resource leak, and of course, they get what they want. Weak references, singletons for dynamically allocated objects - nothing of this is needed.

For BLE services, the IDF has a callback. The callback must have access to the service object data to properly manage the BLE stack. Here are two implementations:

  1. Allow creation of objects that are returned (i.e. Bluetooth.Service() ctor) to the user.
  2. Keep all the data internal to the Bluetooth singleton, and manage via methods attached to Bluetooth

Number 1 requires that the Bluetooth singleton have access to the created objects when the callback is called. Calls to a close() method for a Service class would have to inform the Bluetooth singleton that the object is no longer viable, and remove the internal reference; this is entirely do-able. Service objects have to have a singleton nature for a given UUID, since the IDF treats them this way. Thus, creating a new Service (with a given UUID) object when you haven't called close() on an existing one (with the same UUID) would be a runtime error.

Number 2 seems overly clunky, as it makes sense to use an object heirarchy (Bluetooth --> Service --> Characteristic), rather than having a Bluetooth god object.

I'm sure there are more implementation options I haven't considered. I welcome any concrete API examples you'd care to suggest, if it can work within the reality of how BLE & the ESP IDF operate.

IDF Constraints:

pfalcon commented 7 years ago

I welcome any concrete API examples you'd care to suggest,

Above is the pretty concrete example - modlwip, a module for native lwIP API. It has all traits of a mundane callback-based API: set up (like creation of objects) is initiated by client, but some events are delivered back to it via callbacks. Of course, callbacks have "user data" parameter, and that's what used to pass the original socket mp_obj_t into callbacks.

if it can work within the reality of how BLE & the ESP IDF operate.

Well, now if IDF does something completely different to the normal process describe above, then you please point to the definition of these callbacks in IDF headers.

If you really hint what you do, then perhaps you're indeed trying to over-engineer and option 2 is the most natural solution for an initial prototype. I personally can appreciate desire to build a Service object where there's none actually, but doing that may indeed require too much "impedance matching". Again, studying prior art of (apparently) simpler implementation like modlwip would show the idea how that should work and help to decide whether callbacks and overall API of IDF is suitable for something like that.

MrSurly commented 7 years ago

Above is the pretty concrete example - modlwip, a module for native lwIP API. It has all traits of a mundane callback-based API: set up (like creation of objects) is initiated by client, but some events are delivered back to it via callbacks. Of course, callbacks have "user data" parameter, and that's what used to pass the original socket mp_obj_t into callbacks.

The IDF has a single C callback for all GATTS, which includes events for creating services. It does not support a "user data" parameter. For service creation, it will return the service id (which includes UUID), and a handle. For other events related to services, it gives you this same handle (which is the BLE handle). Thus, you need to keep the original object ... somewhere.

pfalcon commented 7 years ago

Then call @projectgus and @igrr and pass them dearest hellos - they did it again!

Note that with

esp_err_t esp_ble_gatts_create_service(esp_gatt_if_t gatts_if,
                                       esp_gatt_srvc_id_t *service_id, uint16_t num_handle);

it would work if only instead of

    /**
     * @brief ESP_GATTS_CREATE_EVT
     */
    struct gatts_create_evt_param {
        esp_gatt_status_t status;       /*!< Operation status */
        uint16_t service_handle;        /*!< Service attribute handle */
        esp_gatt_srvc_id_t service_id;  /*!< Service id, include service uuid and other information */
} create;

it was:

struct gatts_create_evt_param {
        esp_gatt_status_t status;       /*!< Operation status */
        uint16_t service_handle;        /*!< Service attribute handle */
        esp_gatt_srvc_id_t *service_id;  /*!< Service id, include service uuid and other information */
} create;

i.e. if only event delivered via callback had a pointer to esp_gatt_srvc_id_t, as passed in from esp_ble_gatts_create_service(). Then you could subclass esp_gatt_srvc_id_t and add any additional data to it. But nope, doing it the most obvious way is not how Espressif folks do stuff.

So, @MrSurly, sure, there's no choice, you'll need to maintain a mapping from UUID to Service object, that will also take care about singleton'ness.

projectgus commented 7 years ago

Hi @MrSurly ,

I'm not going to engage with Paul's comment directly because I don't think that attitude is conducive to a constructive or professional working relationship.

However, there's no reason why we can't expand the GATTS API to support the use case you have. I don't work on BLE directly, but if I understand correctly it may be enough to add a void *user_data member to the esp_gatt_srv_id_t structure? This way user data specified when the service is created will be accessible in the callbacks.

If you have a request of this kind, or a different suggestion, feel free to open an issue on the ESP-IDF github project - our Bluetooth developers are active on there and they may have other feedback or suggestions. Or pull requests are also welcome, if you have a modification to IDF that works for you.

Angus

MrSurly commented 7 years ago

Hi @projectgus,

Thanks for the heads up on modifying the IDF; didn't know that that was a likely possibility.

In any case, I need to keep an active μPy object because of the issues outlined above (especially singleton-ness), so in this specific case, there's no need. I think I have a good basis for going forward. Enough "analysis paralysis" =)

Having a userdata pointer might be useful for other code bases, but I suspect most of them are going to be more like the hard-coded service tables seen in the BT GATTS example in the IDF.

pfalcon commented 7 years ago

but if I understand correctly it may be enough to add a void *user_data member to the esp_gatt_srv_id_t structure?

@projectgus , thanks for your kind words. One of these days, someone with "constructive or professional working relationship" should make sure that every callback of something which aspires to be more or less general API, accepts a user data parameter, this is a very well known principle with no exceptions, and there's absolutely no need for threads like this to happen to prove it again and again.

MrSurly commented 7 years ago

Okay, V2.0:

Comments welcome

Start the BT stack

import network
b = network.Bluetooth()

Services & Characteristics

Service constructor
service = b.Service(uuid, primary = True)

Services with the same UUID are the same object. Constructing a second service with the same UUID will return the same service object. The exception are services that have been closed (via the .close() method) -- this deallocates the service from the BLE stack.

Service callback
service.callback(callback_func, user_data = None)
Characteristic constructor
char service.Char(uuid, value = None, permissions = b.PERM_READ | b.PERM_WRITE, properties = b.PROP_READ | b.PROP_WRITE | b.PROP_NOTIFY)

Characteristics are associated with services.

Start or stop service
service.start()

# stuff
service.stop()

# services can be restarted:
service.start()
Closing a service
service.close() # Will call stop()

service.start() # Runtime error; service is no longer usable

Closed service objects are completely "forgotten" by the BLE stack, you needn't keep reference. You will no longer receive callbacks for this service object or it's characteristics.

Get a list of current services
currentServices = b.services()
Get a list of service characteristics
chars = service.chars()

Advertising

BLE settings
b.ble_settings(
    int_min = 1280,
    int_max = 1280,
    type b.ADV_TYPE_IND,
    own_addr_type = b.BLE_ADDR_TYPE_PUBLIC,
    peer_addr = bytearray([0 for x in range(6)]),
    peer_addr_type = b.BLE_ADDR_TYPE_PUBLIC,
    channel_map = b.ADV_CHNL_ALL,
    filter_policy = b.ADV_FILTER_ALLOW_SCAN_ANY_CON_ANY,
    adv_is_scan_rsp = False,
    adv_dev_name = None,
    adv_man_name = None,
    adv_inc_txpower = False,
    adv_int_min = 1280,
    adv_int_max = 1280,
    adv_appearance = 0,
    adv_uuid = None,
    adv_flags = 0)

b.adv_enable(True)
MrSurly commented 7 years ago

Okay, now having a weird problem

The function below is used to find an existing service by UUID, for the reasons outlined above. It works fine, except when called from the gatts callback registered using esp_ble_gatts_register_callback. The call to mp_iternext fails with the exception seen below.

Is anyone aware of any possible problems of doing this stuff from the callback? The documentation doesn't mention anything, that I've found.

@projectgus Is there someone on the IDF team that might be able to comment?

Original function
STATIC network_bluetooth_service_obj_t* find_service(esp_bt_uuid_t* uuid) {
    NETWORK_BLUETOOTH_DEBUG_PRINTF("find_service\n");
    network_bluetooth_obj_t* bluetooth = network_bluetooth_get_singleton();
    NETWORK_BLUETOOTH_DEBUG_PRINTF("bluetooth = %p\n", bluetooth);
    NETWORK_BLUETOOTH_DEBUG_PRINTF("uuid = %p\n", uuid);

    mp_obj_t iterable = mp_getiter(bluetooth->services, NULL);
    NETWORK_BLUETOOTH_DEBUG_PRINTF("iterable = %p\n", iterable);
    mp_obj_t item;
    while ((item = mp_iternext(iterable)) != MP_OBJ_STOP_ITERATION) {
        NETWORK_BLUETOOTH_DEBUG_PRINTF("inside while\n");
        network_bluetooth_service_obj_t* service = (network_bluetooth_service_obj_t*) item;
        NETWORK_BLUETOOTH_DEBUG_PRINTF("before cmpare\n");
        if(uuid_eq(uuid, &service->service_id.id.uuid)) {
            return service;
        }
    }
    return MP_OBJ_NULL;
}
Function modified to narrow down issue, called from callback
STATIC network_bluetooth_service_obj_t* find_servicex(esp_bt_uuid_t* uuid) {
    NETWORK_BLUETOOTH_DEBUG_PRINTF("find_servicex\n");
    network_bluetooth_obj_t* bluetooth = network_bluetooth_get_singleton();
    NETWORK_BLUETOOTH_DEBUG_PRINTF("bluetooth = %p\n", bluetooth);
    NETWORK_BLUETOOTH_DEBUG_PRINTF("uuid = %p\n", uuid);

    mp_obj_iter_buf_t iter_buf;
    mp_obj_t iterable = mp_getiter(bluetooth->services, &iter_buf);
    NETWORK_BLUETOOTH_DEBUG_PRINTF("iterable = %p\n", iterable);
    mp_obj_t item;
    NETWORK_BLUETOOTH_DEBUG_PRINTF("before mp_iternext\n");
    item = mp_iternext(iterable);
    NETWORK_BLUETOOTH_DEBUG_PRINTF("after mp_iternext\n");

    return MP_OBJ_NULL;
}
Exception
network_bluetooth_service_start()
network_bluetooth_gatts_event_handler(event = 07 / CREATE, if = 04, param = (status = 00 / OK, service_handle = 0028, service_id = (id = (uuid = AA 00 , inst_id = 00), is_primary = true))
ESP_GATTS_CREATE_EVT
before find_service
find_servicex
bluetooth = 0x3ffc093c
uuid = 0x3ffe8706
iterable = 0x3ffe86b0
before mp_iternext
Guru Meditation Error of type LoadProhibited occurred on core  0. Exception was unhandled.
Register dump:
PC      : 0x400dd086  PS      : 0x00060830  A0      : 0x800dd0b2  A1      : 0x3ffe85e0  
A2      : 0x3ffd99fc  A3      : 0x3f440265  A4      : 0x3ffe8660  A5      : 0x3ffd99a4  
A6      : 0x00000000  A7      : 0x00000000  A8      : 0x800dd086  A9      : 0x3ffe85c0  
A10     : 0x00000000  A11     : 0x00000004  A12     : 0x00000001  A13     : 0x00000080  
A14     : 0x00000001  A15     : 0x00000000  SAR     : 0x00000004  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x0000000c  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xfffffffb  

Backtrace: 0x400dd086:0x3ffe85e0 0x400dd0b2:0x3ffe8610 0x400dc40e:0x3ffe8630 0x400f1632:0x3ffe86b0 0x40105325:0x3ffe86e0 0x40105a60:0x3ffe8700 0x401030cb:0x3ffe8740

Rebooting...
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0x00
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0008,len:8
load:0x3fff0010,len:2372
load:0x40078000,len:6708
load:0x40080000,len:252
entry 0x40080034
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0x00
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0008,len:8
load:0x3fff0010,len:2372
load:0x40078000,len:6708
load:0x40080000,len:252
entry 0x40080034
dpgeorge commented 7 years ago

@MrSurly good to see you are making progress! The crash may be a problem with the callback code not being in iRAM. A lot of the IDF callbacks happen within an IRQ context and can only run from iRAM. I think you can configure this though so the callback is called after the IRQ returns, when flash is accessible again; you configure it when you set the callback.

Also note that mp_iternext may raise an exception (but not StopIteration, it'd be something else) and that would probably lead to a crash. To guard against that you'd need to wrap all the code in an nlr block; see py/runtime_utils.c for an example of how to do that.

dpgeorge commented 7 years ago

If the list of services is always a list and not a generic iterable (I think that's a sensible restriction) then it's much easier to use mp_obj_get_array() to extract the items from the list.

MrSurly commented 7 years ago

@MrSurly good to see you are making progress! The crash may be a problem with the callback code not being in iRAM. A lot of the IDF callbacks happen within an IRQ context and can only run from iRAM. I think you can configure this though so the callback is called after the IRQ returns, when flash is accessible again; you configure it when you set the callback.

I don't really have that option: esp_err_t esp_ble_gatts_register_callback(esp_gatts_cb_t callback)

There isn't anything in the documentation about this code needing to be in iRAM, and none of the example code indicates that you need to do that.

I want to do more investigation -- I think it's more likely I'm just doing something wrong, and less likely it's something with the IDF/docs/examples. Although, the exact same code works fine when called outside the callback (i.e. when you try to add a duplicate service UUID).

I'm considering getting a JTAG -- have you used one with the ESP32?

I think a plan might be:

Also note that mp_iternext may raise an exception (but not StopIteration, it'd be something else) and that would probably lead to a crash. To guard against that you'd need to wrap all the code in an nlr block; see py/runtime_utils.c for an example of how to do that

I'll try it.

Thanks!

MrSurly commented 7 years ago

If the list of services is always a list and not a generic iterable (I think that's a sensible restriction) then it's much easier to use mp_obj_get_array() to extract the items from the list.

It's always a list. I'm mostly learning by looking at existent code, and I saw the mp_getiter idiom often employed.

projectgus commented 7 years ago

I believe that callback runs in the context of the Bluedroid task, not an ISR, so it shouldn't be a problem having it in flash cache not IRAM. (Side note: as it happens, in IDF most ISRs don't need to be in IRAM any more, see here.)

Looking at the exception dump, this looks to me like a null pointer dereference (exception type LoadProhibited, EXCVADDR 0xd - implies the pointer was to 0x0 and it's trying to read a structure member at offset 0xd).

You can decode the crashing address and the "Backtrace" line, to find where in the source this happens. The simplest way is to use the idf_monitor tool (which IDF runs when you invoke the "make monitor" target). There's some documentation here: http://esp-idf.readthedocs.io/en/latest/idf-monitor.html

I don't know how compatible that approach is with Micropython. Alternatively you can paste the backtrace line into addr2line. There's a sample command line at the documentation link above.

I'm considering getting a JTAG -- have you used one with the ESP32?

Regarding JTAG, there are still some quirks with openocd on ESP32 which makes it a little awkward to use sometimes. A simpler - and often just as workable - solution is to enable the "gdbstub on panic", where the ESP32 speaks the gdb protocol over serial after a fatal exception of this kind. This allows you to poke around with gdb, similar to loading a core dump on the desktop. You can set this option in make menuconfig under "ESP32-Specific" -> "Panic Handler Behaviour". (I know Micropython doesn't use menuconfig directly, but that's the best explanation I have for how to find it.)

If you enable the serial gdbstub and use idf_monitor/"make monitor" then gdb will automatically be started when the gdbstub starts up on the ESP32. If you use a different serial program then you have to close the serial port and then run gdb manually. The manual gdb command can also be found in the documentation link above.

(If you do want a JTAG adapter, we mostly use FT2232H based ones like the "Tiao USB JTAG". I've also heard of people using J-Link. Any adapter supported by openocd in "JTAG mode" (ie not ST-Link, not CMSIS) should theoretically work.

Once you have a source code stack trace I'm happy to chase this up further on the IDF side. :)

MrSurly commented 7 years ago

@dpgeorge

JFYI, I've had to severely modify esp32/Makefile for BT. There are a lot of places where headers clash (e.g. too many incompatible config.h files), so includes have been split up. Let me know if I'm going about this the wrong way.

Also, some parts of the IDF BT don't compile cleanly (usually strict aliasing), and I've added -Wno-error flags for the files in question.

MrSurly commented 7 years ago

Status

Need:

Comments are very welcome. @mahanet, what is/are your use case(s)?

Working:

(As defined in the last RFC, above)

Coming soon:

To Do:

Not anytime soon (?):

projectgus commented 7 years ago

Also, some parts of the IDF BT don't compile cleanly (usually strict aliasing), and I've added -Wno-error flags for the files in question.

FWIW, this was recently fixed in IDF master branch (as of espressif/esp-idf#518).

Bluetooth Classic -- not even sure if the IDF supports this?

IDF master has support for A2DP as of last week. I believe other BT Classic profiles are coming.

MrSurly commented 7 years ago

@dpgeorge wrote:

The crash may be a problem with the callback code not being in iRAM. A lot of the IDF callbacks happen within an IRQ context and can only run from iRAM. I think you can configure this though so the callback is called after the IRQ returns, when flash is accessible again; you configure it when you set the callback.

I'm past where I was, but I've narrowed it down a bit. For my test, from within the callback, I'm simply doing:

mp_obj_print(mp_const_none, PRINT_REPR);

This crashes spectacularly, but if I modify obj.c and remove the call to MP_STACK_CHECK(); from mp_obj_print_helper, then the callback prints "None" as expected.

Looks like there's something about the µPy stack / threads that causes this crash when accessed from the callback.

Any thoughts?

MrSurly commented 7 years ago

I believe that callback runs in the context of the Bluedroid task, not an ISR, so it shouldn't be a problem having it in flash cache not IRAM. (Side note: as it happens, in IDF most ISRs don't need to be in IRAM any more, see here.) ...

@projectgus Thank you, this is very useful information.

dpgeorge commented 7 years ago

JFYI, I've had to severely modify esp32/Makefile for BT

@MrSurly : You shouldn't need to modify it that much. For the following line you added:

$(BUILD)/$(ESPCOMP)/bt/%.o: CFLAGS = $(CFLAGS_BT)

you can just change that to append the include dirs to CFLAGS, eg:

$(BUILD)/$(ESPCOMP)/bt/%.o: CFLAGS +=
        -I$(ESPCOMP)/bt/bluedroid/include \
        -I$(ESPCOMP)/bt/bluedroid/stack/include \
        -I$(ESPCOMP)/bt/bluedroid/osi/include \

etc. That should work and be more self-contained.

Looks like there's something about the µPy stack / threads that causes this crash when accessed from the callback.

Ok, I think I know what's going on (and thanks to @projectgus for the hint that "callback runs in the context of the Bluedroid task"): in the BT task the stack is different and so when uPy does a stack-depth check it fails because the top of the stack has moved. There are two ways to fix this:

  1. Do minimal processing in the BT callback, in particular don't call any uPy code that does a stack check. Calling mp_obj_get_array should be ok. But if anything raises an exception it'll crash because the exception handler lives in the uPy task and you can't do a long jump from one stack to another. You can use mp_sched_schedule() to schedule a callback in the main uPy task.

  2. Save the old stack extents in the BT callback, change them, then restore them at the end. This will allow the stack checker to work and you can call any code (but be careful you don't overflow the BT task's stack). To do this you'd need to do something like:

    char *old_stack_top = MP_STATE_THREAD(stack_top);
size_t old_stack_limit = MP_STATE_THREAD(stack_limit);
MP_STATE_THREAD(stack_top) = &old_stack_top; // a but of a hack but should work ok
MP_STATE_THREAD(stack_limit) = 2048; // how much stack does the BT task have?
// do things
MP_STATE_THREAD(stack_top) = old_stack_top;
MP_STATE_THREAD(stack_limit) = old_stack_limit;
return;
mahanet commented 7 years ago

@MrSurly Thank you for the great progress!

My usecase is mainly a server with services and chars, being able to advertise, connect and read\write the chars. Client is bonus for me.

I played a bit with fa8b776 on bluetooth-dev, noticed that every service created after the first one, is falsely detected as an existing service. I believe it has to do with "network_bluetooth_find_item" under "network_bluetooth_service_make_new", maybe because arg "handle" is always zero? (is "handle" member of newely created service object initiated by the event? ) Maybe I'm just doing something wrong?

MrSurly commented 7 years ago

@MrSurly : You shouldn't need to modify it that much. For the following line you added: ... $(BUILD)/$(ESPCOMP)/bt/%.o: CFLAGS += etc. That should work and be more self-contained.

If CFLAGS already contains all the paths needed to compile µPy, then BT won't compile. Problem is that there are include paths needed by the "main" µPy code that contain a config.h file, and there are different paths used by BT that also contain a config.h file. So, depending on your path ordering, one or the other won't compile, since it will be getting the wrong config.h. Also, there are some includes that are used by both µPy and BT. Thus:

Which leads to this:

CFLAGS_BT := $(CFLAGS) $(INC_COMMON) $(INC_BT)
CFLAGS    +=           $(INC_COMMON) $(INC_MAIN)

Though looking at it now, I could clearly eliminate INC_COMMON, and make it part of CFLAGS.

Do minimal processing in the BT callback, in particular don't call any uPy code that does a stack check. Calling mp_obj_get_array should be ok. But if anything raises an exception it'll crash because the exception handler lives in the uPy task and you can't do a long jump from one stack to another. You can use mp_sched_schedule() to schedule a callback in the main uPy task

I came to the same realization, and this is exactly what I implemented; it's working now, but I need to test further.

MrSurly commented 7 years ago

I played a bit with fa8b776 on bluetooth-dev, noticed that every service created after the first one, is falsely detected as an existing service. I believe it has to do with "network_bluetooth_find_item" under "network_bluetooth_service_make_new", maybe because arg "handle" is always zero? (is "handle" member of newely created service object initiated by the event? ) Maybe I'm just doing something wrong?

You were doing it right, my logic was wrong:

Was: if ((uuid != NULL && uuid_eq(&service->service_id.id.uuid, uuid)) || service->handle == handle)

Now (64fee58): if ((uuid != NULL && uuid_eq(&service->service_id.id.uuid, uuid)) || (uuid == NULL && service->handle == handle))

Here's my current test script:

import network as n
b = n.Bluetooth()
b.ble_settings(adv_man_name = "mangocorp", adv_dev_name="mangoprod" )
b.ble_adv_enable(True)

s1 = b.Service(0xFFFF)
s2 = b.Service(0xEEEE)
c1 = s1.Char(0xAAAA)
c2 = s2.Char(0xAAAA)
s1.start()
s2.start()
def ccb(c, e, d, u): 
 print(c,e,d,u)
 return 'kumquat'
c1.callback(ccb,'callback1')
c2.callback(ccb,'callback2')

I then use gatttool:

$ gatttool -I -b 24:0A:C4:05:7E:E6
[24:0A:C4:05:7E:E6][LE]> connect
Attempting to connect to 24:0A:C4:05:7E:E6
Connection successful
[24:0A:C4:05:7E:E6][LE]> char-desc
handle: 0x0001, uuid: 00002800-0000-1000-8000-00805f9b34fb
handle: 0x0002, uuid: 00002803-0000-1000-8000-00805f9b34fb
handle: 0x0003, uuid: 00002a05-0000-1000-8000-00805f9b34fb
handle: 0x0014, uuid: 00002800-0000-1000-8000-00805f9b34fb
handle: 0x0015, uuid: 00002803-0000-1000-8000-00805f9b34fb
handle: 0x0016, uuid: 00002a00-0000-1000-8000-00805f9b34fb
handle: 0x0017, uuid: 00002803-0000-1000-8000-00805f9b34fb
handle: 0x0018, uuid: 00002a01-0000-1000-8000-00805f9b34fb
handle: 0x0019, uuid: 00002803-0000-1000-8000-00805f9b34fb
handle: 0x001a, uuid: 00002aa6-0000-1000-8000-00805f9b34fb
handle: 0x0028, uuid: 00002800-0000-1000-8000-00805f9b34fb
handle: 0x0029, uuid: 00002803-0000-1000-8000-00805f9b34fb
handle: 0x002a, uuid: 0000aaaa-0000-1000-8000-00805f9b34fb
handle: 0x002b, uuid: 00002800-0000-1000-8000-00805f9b34fb
handle: 0x002c, uuid: 00002803-0000-1000-8000-00805f9b34fb
handle: 0x002d, uuid: 0000aaaa-0000-1000-8000-00805f9b34fb
[24:0A:C4:05:7E:E6][LE]> char-write-cmd 2a 010203
[24:0A:C4:05:7E:E6][LE]> char-write-cmd 2d 040506
[24:0A:C4:05:7E:E6][LE]>

And this is what I see from µPy (lots of debug output, but look at the callback outputs): 

network_bluetooth_gatts_event_handler(event = 0E / CONNECT, if = 04, param = (conn_id = 0000, remote_bda = 5C:F3:70:7D:8A:D6, is_connected = true)
entering network_bluetooth_gap_event_handler()
entering network_bluetooth_gap_event_handler()
network_bluetooth_gatts_event_handler(event = 02 / WRITE, if = 04, param = (conn_id = 0000, trans_id = 00000001, addr = 5C:F3:70:7D:8A:D6, handle = 002A, offset = 0000, need_resp = false, is_prep = false, len = 0003, data = 01 02 03 )
data pushed
In data handler
Found 1 queue items
BTChar(uuid = AAAA, perm = 11, prop = 1A, handle = 002A) 2 bytearray(b'\x01\x02\x03') callback1
network_bluetooth_gatts_event_handler(event = 02 / WRITE, if = 04, param = (conn_id = 0000, trans_id = 00000002, addr = 5C:F3:70:7D:8A:D6, handle = 002D, offset = 0000, need_resp = false, is_prep = false, len = 0003, data = 04 05 06 )
data pushed
In data handler
Found 1 queue items
BTChar(uuid = AAAA, perm = 11, prop = 1A, handle = 002D) 2 bytearray(b'\x04\x05\x06') callback2
mahanet commented 7 years ago

Ok I'm able to create multiple services now!

I'm using "nRF connect" app for android for testing the chars, but It fails connecting. Tried with gatttool but I'm not sure my PC Bluetooth supports BLE. The micropython prompt doesn't print out anything related to the connection attempts.

I'll try getting more information about my fails.

MrSurly commented 7 years ago

I'm using "nRF connect" app for android for testing the chars, but It fails connecting. Tried with gatttool but I'm not sure my PC Bluetooth supports BLE. The micropython prompt doesn't print out anything related to the connection attempts.

Yes, the IDF BLE connection thing seems a bit weird. When I use gatttool to perform a connection, it states "connected." On the ESP32/µPy, there isn't any CONNECT event, just this (printed directly by the IDF): E (6085885) BT: btm_ble_resolve_random_addr_on_conn_cmpl unable to match and resolve random address.

However, when I send a command (like reading a characteristic), that's when the IDF sends a connect event.

I'm not sure if it's a bug in the IDF

MrSurly commented 7 years ago

Okay, new problem; I'm getting crashes whenever I append to a list, but only in the IDF GAP callback handler.

Implementation

Code is here: https://github.com/MrSurly/micropython-esp32/commit/7bb0328827a72485ad0027c39a20197104d552b2

I've taken the approach of having the callback simply append relevant information to a queue, and then calling mp_sched_schedule() to have a different handler read this queue and call any user-define Python callbacks.

Implementation is done via a global µPy list; in the IDF callback handler, I create a list of parameters (mp_obj_new_list), and append to the global list.

In the handler that calls the Python callbacks, it iterates through the entire list, and then zeroes it's length.

Accesses to the global list are guarded using a xSemaphoreTake and xSemaphoreGive.

Here's the gist:
Calls to mp_obj_list_append from the IDF callback will result in the stack trace seen below, right after calling gc.collect()

So, there's something going on with the GC that I don't understand.

Moving forward:

I think I'll just use a ring buffer of a union object that can contain the relevant raw data from the IDF callback, for now; essentially bypass any memory management.

>> gap append
>> cbQ Take
<< cbQ Take
Guru Meditation Error of type StoreProhibited occurred on core  0. Exception was unhandled.
Register dump:
PC      : 0x4000c46c  PS      : 0x00060030  A0      : 0x800e0c04  A1      : 0x3ffe9d70  
A2      : 0x00000004  A3      : 0x00000000  A4      : 0xfffffffc  A5      : 0x00000004  
A6      : 0x3f404b90  A7      : 0x0fffffff  A8      : 0x800d321f  A9      : 0x3ffe9d40  
A10     : 0x00000000  A11     : 0x00000000  A12     : 0x00000001  A13     : 0x00000000  
A14     : 0x00000000  A15     : 0x00000000  SAR     : 0x0000001c  EXCCAUSE: 0x0000001d  
EXCVADDR: 0x00000004  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x0ffffffe  

Backtrace: 
0x4000c46c: ?? ??:0
0x400e0c04: mp_obj_list_append at /home/epoulsen/workspaces/micropython-esp32/esp32/../py/objlist.c:240
0x400f1bc6: network_bluetooth_gap_event_handler at /home/epoulsen/workspaces/micropython-esp32/esp32/network_bluetooth.c:1563
0x40106294: btc_gap_ble_cb_to_app at /home/epoulsen/workspaces/esp-idf/components/bt/bluedroid/btc/profile/std/gap/btc_gap_ble.c:35
0x40106a06: btc_gap_ble_cb_handler at /home/epoulsen/workspaces/esp-idf/components/bt/bluedroid/btc/profile/std/gap/btc_gap_ble.c:707
0x40105503: btc_task at /home/epoulsen/workspaces/esp-idf/components/bt/bluedroid/btc/core/btc_task.c:84

Line 240 in mp_obj_list_append is:


231: mp_obj_t mp_obj_list_append(mp_obj_t self_in, mp_obj_t arg) {
232:    mp_check_self(MP_OBJ_IS_TYPE(self_in, &mp_type_list));
234:    mp_obj_list_t *self = MP_OBJ_TO_PTR(self_in);
235:    if (self->len >= self->alloc) {
236:        self->items = m_renew(mp_obj_t, self->items, self->alloc, self->alloc * 2);
237:        self->alloc *= 2;
238:        mp_seq_clear(self->items, self->len + 1, self->alloc, sizeof(*self->items));
239:    }
240:    self->items[self->len++] = arg;
241:    return mp_const_none; // return None, as per CPython
242:}
dpgeorge commented 7 years ago

Okay, new problem; I'm getting crashes whenever I append to a list, but only in the IDF GAP callback handler.

If the IDF callback is a true IRQ (which it is) then you can't allocate or free memory in this callback. This is because the code that was interrupted may have been in the middle of an allocation and the GC is not reentrant.

I think I'll just use a ring buffer of a union object that can contain the relevant raw data from the IDF callback, for now; essentially bypass any memory management.

Yes this would be the way to do it.

igrr commented 7 years ago

If the IDF callback is a true IRQ (which it is)

I don't think this is the case here. Based on the backtrace above, the callback runs from btc_task. (The rest of the comment about interrupting an allocation in progress is still valid)

projectgus commented 7 years ago

If the IDF callback is a true IRQ (which it is) then you can't allocate or free memory in this callback.

From the IDF side, it should be OK to perform libc-level memory operations here as the BT event callbacks aren't interrupts, they run in the context of the bluedroid task started in btc_task.c (see the backtrace in the dump above.) [^]

I don't know anything about the Micropython side. Are there any restrictions on concurrently allocating/freeing memory, or accessing GC-managed objects from multiple tasks?

Store Prohibited at EXCVADDR 0x4 implies (I think) that "self" may be NULL, although the field at offset 4 appears to be "alloc" so the crash may actually be at line 237 (addr2line seems to get it a bit wrong sometimes due to optimisations, or possibly I'm misreading the code.)

[^]: For the record, it's probably better not to block that bluedroid task for an extended period, as the queue of pending BT notifications will eventually fill up - although priority inheritance should kick in and prevent absolute resource starvation if all the tasks involved are blocking on RTOS primitives. Anyway, I would think that short delays (blocking on RTOS primitives) are fine.

EDIT: Sorry, I missed the very clear "the GC is not reentrant" part of damien's post. So this is comment is mostly irrelevant!

projectgus commented 7 years ago

Oops, my post crossed with Ivan's!

dpgeorge commented 7 years ago

Thanks @igrr and @projectgus for the hints. Memory can only be allocated if control is within the main MicroPython task and an IRQ is not being processed. As said above, this is because the GC is not reentrant.

In the case here for BT, one cannot allocate memory from the btc_task (because the preemptive task switch may have interrupted a GC memory allocation).

@MrSurly a neat way of getting around this problem is to acquire the GIL before you do any memory allocations, because that will ensure you have exclusive access to the GC. Do something like this in the callback:

MP_THREAD_GIL_ENTER();
// code that may allocate memory
MP_THREAD_GIL_EXIT();

That will work quite efficiently, and the btc_task will block (via FreeRTOS's mutex's) if the GIL is held by the main uPy task. The main task will release the GIL without too much delay so btc_task won't block for long.

MrSurly commented 7 years ago

Updated API

Start the BT stack

import network
b = network.Bluetooth()

Services & Characteristics

Service constructor
service = b.Service(uuid, primary = True)

Services with the same UUID are the same object. Constructing a second service with the same UUID will return the same service object. The exception are services that have been closed (via the .close() method) -- this deallocates the service from the BLE stack.

Characteristic constructor
char service.Char(uuid, value = None, permissions = b.PERM_READ | b.PERM_WRITE, properties = b.PROP_READ | b.PROP_WRITE | b.PROP_NOTIFY)

Characteristics are associated with services.

Start or stop service
service.start()

# stuff
service.stop()

# services can be restarted:
service.start()
Closing a service
service.close() # Will call stop()

service.start() # Runtime error; service is no longer usable

Closed service objects are completely "forgotten" by the BLE stack, you needn't keep reference. You will no longer receive callbacks characteristics attached to the service.

Get a list of current services
currentServices = b.services()
Get a list of service characteristics
chars = service.chars()

Callbacks

Bluetooth Callback
def bt_callback(btObj, event, data, userdata):
    print ("Bluetooth object", btObj)
    if event == btObj.CONNECTED:
        print ("connected")
    if event == btObj.DISCONNECTED:
        print ("disconnected")

Bluetooth.callback(bt_callback, "hello")

Bluetooth.callback takes two parameters: The callback function itself, and an optional userdata value to be passed in on callbacks. It returns the current callback function and usedata as tuple.

cb_function can be None, which disables callbacks. cb_function is called with four parameters; the bluetooth object, event, event data, and the userdata.

Event values:

Characteristic Callbacks
def char_callback(charObj, event, value, userdata):
    print ("Char object", charObj)
    if event == btObj.CONNECT:
        print ("connected")
    if event == btObj.DISCONNECT:
        print ("disconnected")

Char.callback(cb_function, userdata)

Char.callback takes two parameters: The callback function itself, and an optional userdata value to be passed in on callbacks. It returns the current callback function and userdata as tuple.

cb_function can be None, which disables callbacks. cb_function is called with four parameters; the characteristic object, the event type, the event value (the sent value for WRITE or the current characteristic value for READ), and userdata.

The event type will be one of Bluetooth.READ or Bluetooth.WRITE.

Characteristics have an internal value, which is not used and not updated when there is a callback, though the callback is free to update or use the value as it sees fit. For READ operations (or WRITE operations that require a return value) the return value of the callback is sent, which must be a bytearray, str, or None.

Value behavior without a callback

If there is no callback defined for a characteristic, it's internal value will be used for READ or WRITE events, and WRITE events that expect a response will simply return the just-written value.

Advertising

BLE settings
b.ble_settings(
    int_min = 1280,
    int_max = 1280,
    type b.ADV_TYPE_IND,
    own_addr_type = b.BLE_ADDR_TYPE_PUBLIC,
    peer_addr = bytearray([0 for x in range(6)]),
    peer_addr_type = b.BLE_ADDR_TYPE_PUBLIC,
    channel_map = b.ADV_CHNL_ALL,
    filter_policy = b.ADV_FILTER_ALLOW_SCAN_ANY_CON_ANY,
    adv_is_scan_rsp = False,
    adv_dev_name = None,
    adv_man_name = None,
    adv_inc_txpower = False,
    adv_int_min = 1280,
    adv_int_max = 1280,
    adv_appearance = 0,
    adv_uuid = None,
    adv_flags = 0)

b.adv_enable(True)

Scanning

Scan start:

Advertising and scanning cannot be run at the same time

Bluetooth.scan_start(timeout)

Scan stop:

Bluetooth.scan_stop()

Scan callback data

The Bluetooth callback will receive:

mahanet commented 7 years ago

Getting better here on c505d56 with an Iphone.

For my Ubuntu - there is no hardware BLE support. The nordic nrfConnect app for android fails on connecting (although it operates good on IDF bluetooth apps). But the same nordic app for Iphone works!

I can connect, view the services, chars, but when I try reading writing to them the whole thing fails - disconnects or throws a processor exception - I guess things are still under dev? Or should reading/writing be working?

Concerning the RFC, minor typos (for those who try to copy-paste from it)-

The script I played with-

import network

def bt_callback(btObj, event, data, userdata):
    print ("Bluetooth object", btObj)
    if event == btObj.CONNECT:
        print ("connected")
    if event == btObj.DISCONNECT:
        print ("disconnected")

def char_callback(charObj, event, value, userdata):
    print ("Char object", charObj)
    if event == b.READ:
        print ("read")
    if event == b.WRITE:
        print ("write")

b = network.Bluetooth()

b.ble_settings(adv_man_name = "mangocorp", adv_dev_name="mangoprod")

b.callback(bt_callback, "hello")

s1 = b.Service(uuid=0xaabb)
s2 = b.Service(uuid=0xa123)

c1 = s1.Char(0xcc, 15, b.PERM_READ | b.PERM_WRITE, b.PROP_READ | b.PROP_WRITE | b.PROP_NOTIFY)
c2 = s2.Char(0xdd, 16, b.PERM_READ | b.PERM_WRITE, b.PROP_READ | b.PROP_WRITE | b.PROP_NOTIFY)

c1.callback(char_callback, "char1 hello")
c2.callback(char_callback, "char2 hello")

s1.start()
s2.start()

b.ble_adv_enable(True)

The faults I was talking about probably had to do with chars values. Proper values should be strings or arraybytes but when I try either one I get an hard exception and a reset (on connection or read\write from chars).

When I use plain numbers I just get this output and no callback is called when reading writing from client to board-

network_bluetooth_gatts_event_handler(event = 0E / CONNECT, if = 04, param = (conn_id = 0000, remote_bda = 4B:A1:D1:9A:EB:18, is_connected = true)
>> gatts append
>> cbQ Take
<< cbQ Take
<< cbQ Give
>> cbQ Give
<< gatts append
>> handler
>> cbQ Take
<< cbQ Take
<< cbQ Give
>> cbQ Give
Bluetooth object Bluetooth(params=())Bluetooth(conn_id = 0000, gatts_connected = True, adv_params = (adv_int_min = 1280, 
adv_int_max = 1280, 
adv_type = 0, 
own_addr_type = 0, 
peer_addr = 00:00:00:00:00:00, 
peer_addr_type = 0, 
channel_map = 7, 
adv_filter_policy = 0
), data = (set_scan_rsp = false, 
include_name = true, 
include_txpower = false, 
min_interval = 2048, 
max_interval = 2048, 
appearance = 0, 
manufacturer_len = 9, 
p_manufacturer_data = mangocorp, 
service_data_len = 0, 
p_service_data = E (100286) BT: MTU request PDU with MTU size 517

nil, 
flag = 0
)

connected
E (100296) BT: Call back not found for application conn_id=3
E (100306) BT: Call back not found for application conn_id=5
network_bluetooth_gatts_event_handler(event = 04 / MTU, if = 04, param = (conn_id = 0000, mtu = 0205)

>>> 
>>> 
>>> E (120506) BT: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0013
E (120506) BT: bta_gattc_conn_cback() - cif=5 connected=0 conn_id=5 reason=0x0013
E (120506) BT: btm_sec_disconnected clearing pending flag handle:0 reason:19
MrSurly commented 7 years ago

For my Ubuntu - there is no hardware BLE support.

I ended up buying one of these: https://www.amazon.com/gp/product/B009ZIILLI/ref=oh_aui_detailpage_o08_s00?ie=UTF8&psc=1 Works fine on Ubuntu

Getting better here on c505d56 with an Iphone

I'm not going to respond to your individual points here. On Monday, I'll work through what you wrote, and try to reproduce/fix the issues.

Thank you for taking the time to try everything. If you're good with how the API doc is written, then I'll make the code conform. Changes/suggestions are welcome.

mahanet commented 7 years ago

I'm not going to respond to your individual points here. On Monday, I'll work through what you wrote, and try to reproduce/fix the issues.

Thank you

If you're good with how the API doc is written, then I'll make the code conform.

These changes are very minor so both the current API and RFC are good for me. I just wanted to have you noticed there are small differences so the next RFC you post will correlate 100% :)

MrSurly commented 7 years ago

@mahanet https://github.com/MrSurly/micropython-esp32/commit/cc14148d26edafad407b680ce2e31c368fc726b2 has two fixes:

I'm working on fixing up the RFC, your comments are all spot-on.

Here's what I'm using for boot.py (below). You can simply type gatts() in the µPy console to start a BLE server.

import network as n
import gc
b = n.Bluetooth()

def bcb(b,e,d,u):
    print('BT callback: ', gc.mem_free(), end = '')
    if e == b.CONNECT:
    print("CONNECT")
    elif e == b.DISCONNECT:
    print("DISCONNECT")
    elif e == b.SCAN_DATA:
        if type(d) is tuple:
            adx, name = d
            print ('Found:' + ':'.join(['%02X' % i for i in adx]), name)
        else:
            print()
    elif e == b.SCAN_CMPL:
    print("Scan Complete")
    else:
        print ('Unknown event', e,d)

def cb (cb, event, value, userdata):
    print('charcb ', cb, userdata,' ', end='')
    if event == b.READ:
        print('Read')
        return 'ABCDEFG'
    elif event == b.WRITE:
        print ('Write', value)

def gatts():
    s = b.Service(0xaabb)
    c = s.Char(0xccdd)
    c.callback(cb)

    s.start() 

    b.ble_settings(adv_man_name = "mangocorp", adv_dev_name="mangoprod")
    b.ble_adv_enable(True)

b.callback(bcb)
MrSurly commented 7 years ago

@dpgeorge

Edit: Looking into FreeRTOS queues. Edit 2: Problem is that there's no RAM left

Would MP_THREAD_GIL_ENTER(); work if what's being called ends up causing garbage collection?

I'm getting this stack trace, from inside an MP_THREAD_GIL_ENTER block (below).

Some background:

Re-implemented the ring buffer for event queue as doubly linked list, thus could be more easily dynamically sized. Turns out that doing BLE scans (in my office) causes a ridiculous amount of found BLE devices. Of course most of the events are simply duplicates of the same advertising packets.

To combat this, I created a global mp_set_t for "seen / not seen", using the bluetooth address as the key, and only enqueueing events for devices not yet seen.

For longer scans, this eventually triggers a GC, since I'm creating new string objects to pass on to mp_set_lookup.

0x400ef190: gc_collect_inner at /home/epoulsen/workspaces/micropython-esp32/esp32/gccollect.c:52
0x400ef178: gc_collect_inner at /home/epoulsen/workspaces/micropython-esp32/esp32/gccollect.c:44
0x400ef178: gc_collect_inner at /home/epoulsen/workspaces/micropython-esp32/esp32/gccollect.c:44
0x400ef178: gc_collect_inner at /home/epoulsen/workspaces/micropython-esp32/esp32/gccollect.c:44
0x400ef178: gc_collect_inner at /home/epoulsen/workspaces/micropython-esp32/esp32/gccollect.c:44
0x400ef178: gc_collect_inner at /home/epoulsen/workspaces/micropython-esp32/esp32/gccollect.c:44
0x400ef178: gc_collect_inner at /home/epoulsen/workspaces/micropython-esp32/esp32/gccollect.c:44
0x400ef178: gc_collect_inner at /home/epoulsen/workspaces/micropython-esp32/esp32/gccollect.c:44
0x400ef178: gc_collect_inner at /home/epoulsen/workspaces/micropython-esp32/esp32/gccollect.c:44
0x400ef1b8: gc_collect at /home/epoulsen/workspaces/micropython-esp32/esp32/gccollect.c:65
0x400d3908: gc_alloc at /home/epoulsen/workspaces/micropython-esp32/esp32/../py/gc.c:446
0x400d31fc: m_malloc at /home/epoulsen/workspaces/micropython-esp32/esp32/../py/malloc.c:76
0x400e27e9: mp_obj_new_str_of_type at /home/epoulsen/workspaces/micropython-esp32/esp32/../py/objstr.c:1976
0x400e30d4: mp_obj_new_str at /home/epoulsen/workspaces/micropython-esp32/esp32/../py/objstr.c:2034
0x400f1132: network_bluetooth_gap_event_handler at /home/epoulsen/workspaces/micropython-esp32/esp32/network_bluetooth.c:1691
0x40106820: btc_gap_ble_cb_to_app at /home/epoulsen/workspaces/esp-idf/components/bt/bluedroid/btc/profile/std/gap/btc_gap_ble.c:35
0x40106f92: btc_gap_ble_cb_handler at /home/epoulsen/workspaces/esp-idf/components/bt/bluedroid/btc/profile/std/gap/btc_gap_ble.c:707
0x40105a8f: btc_task at /home/epoulsen/workspaces/esp-idf/components/bt/bluedroid/btc/core/btc_task.c:84

Code is here: https://github.com/MrSurly/micropython-esp32/commit/eb69763b7e6e6a21f903c87bfbc999a8bdbfff53

MrSurly commented 7 years ago

@mahanet Also implmented indication/notification for characteristics in latest commit.

MrSurly commented 7 years ago

Well,

Things have come to a grinding halt.

Been seeing a lot of weird memory issues lately, and today I finally figured it out. I had already lowered the µPy heap size to 16Kib in order to get µPy to work at all when BT is enabled, and I lowered it again to 12Kib.

The good news is the weird memory problems went away. Bad news is that even _boot.py dies with MemoryError: memory allocation failed, allocating 4212 bytes.