Fault tolerant microservices (MP17)

[lauracowen]

Introduce different aspects of fault tolerance in microservices (retry policies, bulkheads, fallback, circuit breakers) with brief example code. Make clear why a developer should care about each - what value does each type of fault tolerance have? - give concrete examples. See https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_microprofile_fault_tolerance.html for some examples. Also see the guides in case they have some better examples you can grab snippets from.

Maybe also mention the interceptor priority (inc change between versions from here https://github.com/OpenLiberty/docs/issues/136#issuecomment-471654257) - may be a footnote - not sure?

Relevant guides:

• https://www.openliberty.io/guides/retry-timeout.html
• https://www.openliberty.io/guides/bulkhead.html
• https://www.openliberty.io/guides/microprofile-fallback.html
• https://www.openliberty.io/guides/circuit-breaker.html

Spec (may contain some clear descriptions of the different fault tolerance concepts you can use): https://github.com/eclipse/microprofile-fault-tolerance/blob/master/spec/src/main/asciidoc/microprofile-fault-tolerance-spec.asciidoc

[dmuelle]

@lauracowen First draft of new topic is now up at https://draft-openlibertyio.mybluemix.net/docs/ref/general/#building-resilient.html

Can you recommend someone for tech review? Most of the examples are from the spec but it still needs a look.

Also- I included all the information about breaking changes between MPFT 1 & 2 directly in this doc because I didn't think I should link to the commercial liberty page for it (maybe that page should point here once we publish?). Not sure if we need all of that, can revise as necessary.

[dmuelle]

comments from @Emily-Jiang:

Can you add a section on how to use these annotations to solve a real time case such as a time critical situation. In this case, you will use Asynchronous Timeout Fallback etc. Please add some usage on using CompletionStage

Once you have got a CompletionStage, what do you do with it? Obviously you will chain it up and get the result.

You then explain what happens if the completionstage completed with an error or cancelled etc.

[lauracowen]

Feedback:

• I would collapse the first paragraphs (down to the end of the bulleted list) into a single introductory paragraph. No reason to separate it really. Or at least the first three paragraphs into one if you prefer to keep the list of policies separate. Actually, having read further down, I think this list of links would be better in the MPFT section before the sub-sections start (rather than in this intro para).
• Is 'policies' the term used in the spec? I hadn't seen it like that before but if that's accepted term, np.
• Is it not more relevant to the developer reading it to phrase as "The MicroProfile Fault Tolerance feature provides a standard API to implement…" (rather than talking in terms of what the spec defines)? Then "The policies that you implement in your code then guide (control?) how long requests run, when they retry, …" Can you apply this throughout (ie don't refer to the spec so much as the capability provided to the developer using the feature in Open Liberty)?
• I think your editor will pick up on "executions" :) - can you talk about requests or running or something instead of "an execution" or "executions". "Request" is actually clearer to read in this context, I think.
• I'm not sure you need to say that the annotations abide by the rules of another spec. From a developer perspective, they just need to know how to use it - not the background spec info (which they can go find if they really care). Don't even need "which are implemented as…". Annotations are annotations - if they want to understand how annotations are implemented, they can go find out.
• The sentence saying "supports the following annotations" implies there's a list coming but it's really sections. Maybe make it more definitive? eg "MicroProfile Fault Tolerance supports the following annotations: ". Then go into the sub-sections (Timeout etc as sub-sections of MicroProfile Fault Tolerance). Are those headings tagged as different levels in the source? If so, we should raise that against the website team because they look the same level but I think Timeout should be smaller than MP FT.
• Don't include a full reference list of the parameters in this topic - the focus here is conceptual introduction. Also, the reference info you've listed is already available elsewhere in the Javadoc. Can you instead link to the Javadoc for each annotation? eg for Retry: https://openliberty.io/docs/ref/microprofile/3.0/#package=org/eclipse/microprofile/faulttolerance/package-frame.html&class=org/eclipse/microprofile/faulttolerance/Retry.html
• There are missing descriptions in that Javadoc. Can you create a list of the missing info across all the annotations' Javadoc you link to from this topic and raise an issue on the https://github.com/eclipse/microprofile-fault-tolerance/issues backlog. Emily says she'll fix it in time for the next release of the feature.
• Don't include all the examples either - pick one and use it as an example to help understand the basic concept of the annotation. I think maybe ask Emily to include the examples in the relevant parts of the javadoc - javadoc needs more examples.
• "Circuit Breaker" shouldn't be capitalised in general use (eg "the circuit break opens and fails the operation"). I'm not even sure it's necessary to capitalise as the 'name' of the policy - it's not really a thing. See what you think throughout.
• Again, for the other annotations don't give a list of all the options. Provide a single example if that's useful to help understand the concept, but not a comprehensive list here. Link to the relevant Javadoc instead (see if you can link to the javadoc without specifying the version number - I'm not sure how possible that is for the javadoc.
• I don't think the version differences should be listed in this topic (as you mentioned above). I think this topic should just assume the latest version. Not sure the best approach but with Metrics we've broken it out into a separate topic (https://draft-openlibertyio.mybluemix.net/docs/ref/general/#mp_metrics_migration.html) which can then be dropped when we get passed the point that people will still be using FT1.1. We need to be encouraging developers forward so we assume they're using the latest (not an unfair approach if they're using Maven and developing new - only becomes an issue if they're workin with existing production microservices, which they'll know the version of). So maybe create a separate short topic and then provide a link at the end of this article pointing to the FT version differences article?
• I like your 'what to do next' section. Don't think you should highlight the @Fallback annotation in the guide name (just looks weird and not really necessary there).
• I saw Emily has requested some more stuff - see what you can do with that, esp the CompletionStage stuff (hopefully it's in the spec? If not, try talking to Andrew Rouse). For the real-time case with multiple annotations, if it's simple that's fine. If it's not, this should maybe be developed as a guide…see what you find out.

[dmuelle]

Hi @lauracowen - I think I've addressed all the issues from your comment above. I've also added a brief usage example and Emily ok'd the code snippet I put together for it. There is an issue with the draft site currently but my updated doc is in github at https://github.com/OpenLiberty/docs/blob/draft/ref/general/building-resilient.adoc

[dmuelle]

I also opened an issue for the missing descriptions in the javadoc which is currently under discussion

https://github.com/eclipse/microprofile-fault-tolerance/issues/485

[dmuelle]

draft site finally updated- https://draft-openlibertyio.mybluemix.net/docs/ref/general/#building-resilient.html

[dmuelle]

Doc is published at https://www.openliberty.io/docs/ref/general/#building-resilient.html