Redesign communication between components on different cores

[danngreen]

• Use a variant rather than a enums.

• To support multiple simultaneous requests on different cores, we could attach a unique ID to each message and the handler would attach the same ID to its response. This would allow two A7 cores to make the same request type at the same time, and are guaranteed to not get the other core's response. Such a situation could come up with modules getting filesystem access since the modules run on both A7 cores.

Something like:

////////////
// A7 client:
// make request:
if (auto msg_id = proxy.request_read_file(buffer, offset, length)) {
    // request went through
} else {
    // communication bus is busy, try again later
}
...
// check for response, using our msg id:
if (auto msg = proxy.get_message(msg_id.value())) {
    if (msg.type == Proxy::Response::OK) 
        //OK, file was read into buffer
    else 
        //handle request failure (file not found, etc..)
}

////////////
// A7 proxy:
std::optional<uint32_t> Proxy::request_read_file(...){
    uint32_t uid = create_unique_id();

    if (comm.send_message({.uid = uid, .type = ReadFile, .buffer = buffer, ...}))
        return uid;
    else
        return std::nullopt;
}

std::optional<InterCoreMessage> Proxy::get_message(uint32_t uid) {
    if (auto msg = comm.get_message()) {
        if (msg.uid == uid) {
           // client request and response match
           return msg;
        } else {
            // message is a response to a different client request, store it for later
            rx_message_queue.push_back(msg);
        }
    }
    if (auto msg = rx_message_queue.find(uid)) {
       rx_message_queue.remove(uid);
       return msg;
    } else {
      // no matching response found yet
        return std::nullopt;
    }
}

////////////
// M4 handler:
if (message.type == ReadFile) {
    if (read_file(message.filename, message.buffer, message.offset))
        return Message{.uid = message.uid, .type = Response::OK, .bytes_read = ...};
    else 
        return Message{.uid = message.uid, .type = Response::Fail};
}

[LnnrtS]

Sounds good. I think the UID can just be a incremental - no need for anything fancy.

When reading the message I would first look in rx_message_queue.

• std::optional could be enhanced by std::expected. Just once we're already touching this
• There could be a timeout functionality integreated so the callers don't need to handle that all themselves. When you send the request the timer is started and at some certain point not having an answer will turn into a timeout error
• To decouple the mostly independent subsystems that use the ICC mechanism, the message could be a hierarchical variant which each subvariant defined by a subsystem. So each subsystem would only know it's own variant and could only receive and send messages inside that variant. At one place all subsystem variants need to be known to create the top level one though. Alternatively, subsystems could register with ICC system and get assigned an id that is prepended to the message (which would be presented to the ICC system as a type erased list of bytes). The corresponding receiver would then cast that back to it's subsystem variant. This could be done at compile time or runtime. When done at compile time it's basically the same as the hierarchical variant with less automatic type safety but the generic ICC part does not need to know anything of the message types used (apart from the size)

[danngreen]

One thing of interest is that the IPCC hardware peripheral used for the inter-core communication, actually has 6 channels. We were just using 1 channel for all communication. I took advantage of this with the File System access implementation in the STS-coreproc-chanmap branch, and am using a new IPCC channel just for that (also using a std::variant).