Iridium Proposal for Two Phase Send

After discussions with MSIPs, it was established that there is potential for an MSIP (and RCC in the future) to lose track of an MSI id with the current one–phase approach to broadcast an MSI. To mitigate those concerns, we are proposing a new API call to allow MSIPs to issue request and receive a new unique MSIID prior to invoking the API call to deposit the actual MSI in. This new proposed call (ex: /msi/generatemsiid) will be called first and the MSIP will receive he MSIID back. The second proposed change to the existing deployed API is to add an optional parameter “msiID” that can be optionally included with the existing API /msi method. By including it (as the parameter ‘msiID’), the MSIP can ensure that they are already in possession of the msiID prior to posting the MSI and can subsequently query the system with that msiID. This mitigates the risk of losing track of the MSI id due to the lack of delivery guarantees of HTTP responses. The new optional msiID parameter is optional and thus ensures that any users of the current API do not experience any impact to their existing deployment based on what has already been deployed by Inmarsat and Iridium. The system will reject any /msi POST calls attempted with the same msiID. Any generated msiID by an MSIP request can be used at any time or simply discarded.

We have added some mockups of our documentation with the new schema for the post /msi call below postMsi getMsiGenerateMsiId

Hi thanks, the current design at https://amsa-code.github.io/egc-api/#section/EGC-API/Design-considerations improves on your suggestion already.

My suggestion is that the message id is returned at the submission of message content (the create call). The send method is supplied a message id and that's all. If you do this then there is no check needed at the backend in terms of what to do if the message content (possibly with differences) is submitted multiple times with the same id.

The new optional msiID parameter is optional and thus ensures that any users of the current API do not experience any impact to their existing deployment based on what has already been deployed by Inmarsat and Iridium.

The purpose of this collaboration point is to get the API right for all parties without fault tolerance problems and bottlenecks that limit scalability (all discussed in the above link). This collaboration point is not about applying fixes to existing implementations by Inmarsat and Iridium that appear to have attained production status before thorough community review. That said, if the discussions here help to improve a product that is in use then that's good. By the way who is using these new APIs already in a production sense? How many organizations? I can certainly continue with that question offline if you prefer.

Hi Dave

The concern with the original API that Iridium and Inmarsat implemented was that with a single step create/submit, there was a change of the MSIP loosing track of the MSI if the ack Containing the assigned MSIID for that MSI gets lost and never gets back to the MSIP. That is, agreed, possible. However, unlike Inmarsat system, we also had an optional parm called ExternalRefid designed for the MSIPs so they can track their MSI either with the Iridium MSIID or their own externalRefID. If they never get that ack with MSISD, all they would have to do is query the system with the externalRefId and go from there. This mitigates this issue as well as makes our API consistent with our Web portal.

In an attempt to address your concern, we have proposed a two phase broadcast as suggested. First step where the MSIP secures an MSIID with new optional API call. A second step create which does not change because we added MSIID as optional parameter. This ensures no change to the existing API and allows All to strengthen the MSI broadcast service in case of communication errors.

In your reply, the suggestion is to make the first call a “Create_dont_Submit” call which would deposit the MSI body and return the MSIID. Then require a second call “Broadcast_MSIID” Both approaches require two calls. Our approach endeavors to not break the current implementation. We don’t know who has already started using it but if they have, we will hear from them surely.

Our approach may result in an unused MSIID created and never used when an ack to MSIP is lost. That is of no consequences. The MSIID are temporal. The other approach can result in MSIs deposited and never broadcast when an ack to MSIP is lost. That is more consequential because we may have to investigate why it was never broadcast and we would have nothing To go off of. There is also a risk of MSIPs using our system for storage for which it is not designed.

Our recommendation is the Iridium optional two phase approach. In one case a severe communication issue would result in X wasted MSIID. Every second that goes by 1000000 ids are wasted because we have time down to the microseconds encoded in our ids. In the other case, we end up with X MSI duplicates stored in our system building up which we cannot delete due to audit requirements.

Regards

However, unlike Inmarsat system, we also had an optional parm called ExternalRefid designed for the MSIPs so they can track their MSI either with the Iridium MSIID or their own externalRefID. If they never get that ack with MSISD, all they would have to do is query the system with the externalRefId and go from there. This mitigates this issue as well as makes our API consistent with our Web portal.

I discuss this also in https://amsa-code.github.io/egc-api/#section/EGC-API/Design-considerations. When we allow an external party to provide their own message id then we are exposed to another fault scenario that requires a bottleneck validation step. That step is to check that another message has not been submitted with the same id. If we follow the design I suggest then that fault scenario disappears.

Our approach endeavors to not break the current implementation

That is not a concern with this collaboration. We should not have backend implementation concerns bleeding into the api being developed here. You can put something in front of your current api if you want that fulfills the API contract being collaborated upon here.

The other approach can result in MSIs deposited and never broadcast when an ack to MSIP is lost. That is more consequential because we may have to investigate why it was never broadcast and we would have nothing To go off of. There is also a risk of MSIPs using our system for storage for which it is not designed.

No, if someone doesn't complete a send it is not your problem and there is nothing to investigate. Yes there are failure scenarios (say a client's internet connection dropped out and they didn't receive the message id from the create action) which could result in messages accumulating on your end that never get sent. Just institute a cleanup after some suitable interval (or if you are using nice scalable storage in the cloud don't bother cleaning it up!). I've already discussed this in the service definition:

https://github.com/amsa-code/egc-api/blob/6994e8115c2fba85cf53fea904c0f65c84c772f1/openapi.yml#L224-L236

In one case a severe communication issue would result in X wasted MSIID. Every second that goes by 1000000 ids are wasted because we have time down to the microseconds encoded in our ids.

This is a great example of an implementation concern bleeding into api design. You are free to use whatever id system (e.g. temporal) internally that you like. I recommend that the create method returns a string id (UUID is good) of some maximum length, the send method is parameterized with that id and from that point you guys are free to establish an association of that id with an internal id. Any future calls to the API with the non-internal id will be associated by you with your internal id and you implement the response.

Of course I'm not mandating use of a UUID, it's just a good method of ensuring scalability and availability of service. You can return a time based id if you like as long as uniqueness is guaranteed and you decide what levels of scalability and availability you provide (hopefully high). However, I do want to start with an API that enables the features of fault tolerance, scalability and availability.

Hi Dave,

Kyle wanted me to reach out and let you know that he is going to have a discussion with Stuart, he feels that we need Inmarsat's view on this as well before we make any breaking changes to the current API Standard.

amsa-code / egc-api

Iridium Proposal for Two Phase Send #2