Asynch functionality for mproc rules

[vetss]

Now mproc rule interface does not allow to perform acynch invocations (like access to databases ot external resources). We need to implement some interface for it.

[faizann]

I have also noticed this limitation. It looks like MprocManagement wants to see the return of mproc rule right in sync manner. This means that MProc has to act like an SBB in itself where it can return result as an EVENT. Moreover, we need a pooling interface so that MProc rules can do their part in async like querying a DB and then return result.

I am interested in solving it with your guidance.

[vetss]

Hello @faizann

thanks for your comments. This issue is unfortunately not a simple one and demands some refactoring at core level and changing of mproc api interfaces too. The target that I want to achieve is that mproc api interfaces remain simple. To start the solution we need to suggest / discuss some architecture of this firstly. Do you have ideas for it ?

Sorry, what do you mean "pooling interface" when you described the topic ?

[faizann]

Hi Sergey

Have you had more thoughts on this one? When I said pooling interface, I meant a sort of cache which marks the number of mprocrules running and then some other process reading the end results once pushed. Just like SmsCache is there to check inprocess Sms. But nevermind that idea.

It is indeed a very difficult problem and I am still not sure how best to implement it. But I would like to keep discussing.

Lets take example of making Async HTTP Calls from Mproc Rules.

We can do is to create an MProc SBB that can be invoked as a child SBB of other SBBs like MtSbb. It can dispatch an MPROC_FINISHED event to parent (MtSbb) on Mproc rules finish. This will let MtSbb continue processing using MprocResult. MProcSBB will also need to listen/register for HTTP Client RA events as well as others like JDBC. This makes it a bit complicated.

The MprocRules somehow need to be able to access HTTPClient RA. (from MProcSbb may be?). The Mproc rule can make the call to HTTP and MProcSbb will receive the EVENT when client is finished. The MprocRule is again given the result of the rule and at this stage MProc rule can raise an event (or return PROCESSING_DONE) and MProcSbb will move on to the next MProcRule. When all are finished it can raise the MPROC_FINISHED event or just call a function using Parent’s interface (just the way SriSbb calls MtSbb) and then MtSbb can continue.

So a flow is like this MtSbb—>CreateMProcSbbObject——>MProcSbb—>applyOnArrivalRules(Sms sms)—>rule1—>applyOnArrivalRule()—>return pending; HTTP ResponseEvent—>SLEE Dispatcher——>MProcSbb—>rule1—>onHttpResponse(EventData)—>return (MProcResult,DONE); MProcSbb—>MtSbb.MprocFinished();

[vetss]

Hello @faizann

Sorry for delayed response. The topic that you are asking is rather difficult as for design and I do not have a clear solution too.

The general solution when a user can add any code and usage of any external resoures (connecions to external services, databases, etc) is rather complicated and demands that a user implement and deploy their own SBB(s) + then update some SMSC GW configs or even code so it can invoke user's code from SMSC GW code and return back for furthe message processing.

May be we can as a first step implement access to two external resources:

• requests to some HTTP server
• requests to some database via jdbc driver

For this we can (this is a draft design, feel free to add your ideas): 1) deploy two extra RAs to SMSC GW (we do not have them now - HTTP Client RA and JDBC RA) 2) each of trigger processor (like PostArrivalProcessor) will have extra methods via which a user may initiate outgoing http request / database access request. PostArrivalProcessor.sendHttpRequest(<request full data>); PostArrivalProcessor.sendJdbcRequest(<request full data>); After invoking of such methods mproc rule will return and SMSC GW will initiate requests to external resources 3) For requests we can add two new SBBs: MProcHttpRequestSbb and MProcJdbcRequestSbb (we have as an example of such sort SBBs we have ChargingSbb, you can check it) 4) After responses we need to invoke MProcRule with a response result for the same mproc rule (and message). MProcRule.onPostArrival() - an initial event when a message has come MProcRule.onPostArrival_HttpResponse() - a HTTP resonse has come MProcRule.onPostArrival_JdbcResponse() - a JDBC resonse has come And then a customer may add code for responses processing and results applying 5) We need to refactor code that invokes methods like MProcRule.onPostArrival() so it interupts (if needed) of processing and after response results are processed by mpro rules code resume of processing from the next rule. 6) We need to implement timers for requests timeouts and add methods of errors/timeouts in processing

So a flow may be: MtSbb (or other SBB) -> Invoke of onPostArrival() methods -> creating MProcHttpRequestSbb / MProcJdbcRequestSbb -> initiates requests to external resources -> push responses back to an original SBB -> Invoke of onPostArrival_HttpResponse()/onPostArrival_JdbcResponse() -> next rule

[faizann]

Hi Sergey

This plan looks good to me.

Step 5 will be quite a bit of work as all SBBs Mt/Mo/Sip/Smpp will need to be refactored. May be they can implement a child MProcSBB just like ChargingSBB but instead of calling reject/accept it would fire an event when all MProcrules have finished and Sms object is updated.

The scenario of usage of multiple RAs by Mproc can also occur and will be flexible. For example: 1-- Mproc.onPostArrival is called. 2-- Mproc rule calls HTTP RA for a request 3-- MProc rule gets HTTP Response 4-- MProc then calls JDBC RA for querying 5-- MProc then gets JDBC response 6-- Mproc finishes up processing and SBB can move to next Mproc rule.

In this case the Original SBB has to know when MProc rule has really finished processing and can move to the next one.

[vetss]

Hello @faizann

your last suggestion makes sense but we need to check what is easier. Generally we need to learn what is easier (The code needs some refactoring).

We need also to make code in the way when other SBB will be involed by firing of events not by direct invoking via SbbLocalObject.

[faizann]

Hi

Cool. Events for MProc will be good instead of LocalObjects. So as a start only one type of RA can be used by MProc. Either HTTP or JDBC.

[ovoo-unif]

Hi,

what is the main reason for introducing the async mproc interface?

Regards, adam

[vetss]

Hello @adamgorak

what is the main reason for introducing the async mproc interface?

The main reason is productivity when we are making of requests to databases or external servers. Async approach will not stop threads inside SLEE SBBs.

[ovoo-unif]

Ok,

I see, what is the max number of microseconds we can afford?

If we go for async solution, I understand we ends the onSubmitSm handling and wait for event from say DB RA. In this case we spent some CPU time on the SBB state store and load (CMP field).

Perhaps the async approach improves the capacity of the SMSC GW, but it does not decrease an overall time of the message request processing. In general, we don't want the SMSC GW to wait for the DB.

It is important to note, the SMSC GW accesses the DB in the RO mode. So we should go for caching as much as possible. It is not a problem (to cache) when certain entity has say 100k objects. If we have more records to cache (> 1M) - it needs to be analyzed.

Regards, adam

[faizann]

Hi adamorak

External I/O always wastes processing time if the thread is blocked until I/O completion. In case of MProc it is easy to add 50-100ms of extra processing time using DB queries. HTTP API calls can take even a lot longer. SLEE by default has 8 threads and it takes between 10-40ms to process an SMS in SBB. With MProc adding 100ms on top will limit SMSC to process between 50-80 SMS a second.

In my case I had HTTP API calls that were taking 200ms as they had some heavy DB stuff going on and that really killed the performance of SMSC.

Async I/O is always a good thing and the cost of CMP field(store/load) etc should be far less than any I/O blocking requests in MProc rules.

[vetss]

I see, what is the max number of microseconds we can afford?

Hmm, the best answer is 0...

If we go for async solution, I understand we ends the onSubmitSm handling and wait for event from say DB RA. In this case we spent some CPU time on the SBB state store and load (CMP field).

We MAY "spent some CPU time on the SBB state store and load", but usually this process do not occure (CMP fields are not serialized / deserialized between SBB calls). Serializations occur if there are too much time between SBB invoking and so not typical for high loaded systems. And if a thread is waiting for synch DB operation then it can easily care for serializing.

Perhaps the async approach improves the capacity of the SMSC GW, but it does not decrease an overall time of the message request processing.

Yes, concrete message processing time will not reduced, only SMSC GW overload is reduced.

It is important to note, the SMSC GW accesses the DB in the RO mode. So we should go for caching as much as possible. It is not a problem (to cache) when certain entity has say 100k objects. If we have more records to cache (> 1M) - it needs to be analyzed.

For caching - agree, we need to analize it. If we have > 1 000 000 subscribers (or > 10 000 000), cache may occupy too much memory space (and time for access). But for some purposes it may helps much.

[ovoo-unif]

Guys,

thanks for explanations.

If SBB manages with SMS submission within 10-40 ms - the average is 25ms. That is about 40 SMS/s in one thread and 320 SMS/s in 8 threads. I think you are able to handle more right now.

Anyway, 25ms is quite a lot. What is it spent for? I/O towards Cassandra?

Regards, adam

[vetss]

Spent time much depends on which interface (SS7, SMPP, SIP, HTTP) is used. As for my experiense SS7 interface takes much more time as compared with SMPP for example (SS7 is much more complicated). And if we use cassandra then it also reduce productivity much. We still need to optimize cassandra side (see https://github.com/RestComm/smscgateway/issues/108).

[vetss]

We have another idea to be implemented. We can add "dummy RA" - Mproc RA that does nothing by default but can be implemented by a user for usage with customized mproc rules.

MprocInterface RA may implement interface like

public interface MprocRaSbbInterface {
    public String invokeTaskSynch(String task) throws Exception;
    public Object invokeTaskSynch(Object task) throws Exception;
    public void invokeTaskAsynch(String task) throws Exception;
    public void invokeTaskAsynch(Object task) throws Exception;
}

that accepts synch and asynch invokations. invokeTaskSynch will return a response after a task is done. invokeTaskAsynch will return nothing. Then Mproc RA must make some asynch request and then fire a special new event for which SMSC GW's SBBs will listern. After getting of a response SMSC GW will invoke the custom mproc rule again with processing results.

[faizann]

That seems more flexible indeed. This means any type of async tasks can be implemented in the custom RA by users and SBBs will not have to implement only listeners for events from there only.

[ovoo-unif]

Yes, let me just indicate how we could proceed, below.

Also, regarding the above methods and the ones currently in MProcRule interface.

What do you think about keeping the onPost... methods in the MProcRule RA Type (at least at the begining) - so that the migration/refactoring would be easier and safer?

Regards, adam

The steps:

1. Migrate current mproc interface/implementation into MProc RA (define sync/async capable RA Type and move current base functionality - it'll be Default MProc RA). In this step the MProc RA would be used synchronously.
2. Changes to review.
3. Verification of the migration/refactoring done in step 1 by regression tests (or at least some happy tests).
4. Quick check and agreement on how to switch the service MProc RA usage, from sync to async. 5a. SMSC GW Core SBBs adaptation to async. 5b. Default MProc RA adaptation to async.

[vetss]

Hello @adamgorak

please go ahead. Please provide draft version for checking.

[vetss]

Implementing a draft version of mproc RA (without asynch functionality) :

https://github.com/RestComm/smscgateway/commit/dbf539e98ede8a9ecba52d1f6b6e9082a2b3cb87 https://github.com/RestComm/smscgateway/commit/b202603fa2bfa486acd9199996ea59e8fed113b5 https://github.com/RestComm/smscgateway/commit/be2dbf188ca61c730cab71168491f13932c09294 https://github.com/RestComm/smscgateway/commit/17c383e9e23873029b80d25342197dfcd54210e0 https://github.com/RestComm/smscgateway/commit/ae5455019f3c4bcad16268b36d444058774e7ebf https://github.com/RestComm/smscgateway/commit/1642ea65faeba82c5b6041ec58fa7731d771cb69 https://github.com/RestComm/smscgateway/commit/20166106c462077eb28eef8988f7653c2b471f6b https://github.com/RestComm/smscgateway/commit/8c414b7cde0dc14f0c4c7f3d8a66bbc45c9c0ba6 https://github.com/RestComm/smscgateway/commit/d765f7941da365702cd98fc4b6637fc5ffeeb4ac

[faizann]

Hi I am reading code for Mproc RA. Is there any design document that shows how we can move towards making it Async? I am working on MProc for the other ticket #188 and would like to get this out of the way too.

Thanks

[faizann]

I am having quite a bit of difficult time to figure out how to do all this Async. We have mproc rules being called from synchronized blocks which means making it async is even harder. Is there a good way to do async, specifically for obtainNextMessage/obtainNextMessagesSendingPool that are part of DeliveryCommonSbb and multiple mproc rules are called while looping on the SmsSet in a lock.

[vetss]

Hello @faizann

we do not have a design document how to make mproc rules asynch, only info in this issue.

How to make of asych rules processing: we need to redesign code, I do not see another way. We discussed here above possible interfaces. Do you want us to discuss possible solution again to move forward to the design ?

[faizann]

Hi @vetss

I have been staring at the code for past few days and making notes. I can create a design document and put my thoughts on it. We can discuss that. Does that sound like a way to move forward.

Otherway is that I can commit code for one of the mproc rules (like OnDelivery) and we can discuss it if the approach is correct.

[vetss]

Hello @faizann

if you can prepare and provide a design document it will be the best, I will check it. You can provide some PR for partial code changing too.

[faizann]

Hi Sergey

Here is an initial document detailing first few Mproc rules. I didn't have your email address. I can share the editable link there. https://docs.google.com/document/d/1x2KfSUsE96SxP-ST2rs_QgL9FUFclWv1uzsAvFDKz1w/edit?usp=sharing

[vetss]

@faizann

sorry for delayed response, I was very busy last week.

Thanks for providing of design document. I am trying to understand which design can we have. The most complicated is a delivery part (applyMProcPreDelivery for example) where we need to apply mproc rules for several messages.

current algo:

DeliveryCommnSbb.java:
for( Sms sms : <smsList>) {
  applyMProcPreDelivery(sms, processingType);
}
private boolean applyMProcPreDelivery(Sms sms, ProcessingType processingType) {
  MProcResult mProcResult = MProcManagement.getInstance().applyMProcPreDelivery(itsMProcRa, sms, processingType);
}

MProcManagement.java:
public MProcResult applyMProcPreDelivery(final MProcRuleRaProvider anMProcRuleRa, Sms sms, ProcessingType processingType) {
  PostPreDeliveryProcessorImpl pap = ...;
  for (FastList.Node<MProcRule> n = cur.head(), end = cur.tail(); (n = n.getNext()) != end;) {
     MProcRule rule = n.getValue();
     rule.onPostPreDelivery(anMProcRuleRa, pap, message);
  }
  <here we process cululative result of processing of all mproc rules for a Sms stored in "pap" object>
}

For each response in rule.onPostPreDelivery(anMProcRuleRa, pap, message); we need to check if a processing is asynch and if yes stop of processing, store the info of which message / which rule is under processing and wait for an event from a MProc RA for finish of processing + setup a timer that will trigger if no response from mproc RA for long time to continue of processing. Another issue - we need to store (for example in CMP fields) a set of local variables that must survive of asynch processing, this is a special topic for differnt SBBs.

All these changing are rather complicated. From another side I am going to change some refactring of how we keep SBB variables (especially SmsSet). I have an idea to store them into a ScheduleActivity object, that may reduce of complicity. The best if we can make both refactoring (SMSC GW delivery data keeping design + asynch mproc rules design) at a single step o reduce of efforts.

My mail is serg.vetyutnev@telestax.com