NHSDigital / integration-adaptor-lab-results

Integration Adaptor to simplify processing of Pathology and Screening results
Apache License 2.0
0 stars 6 forks source link

Mesh issue: If mesh inbox queue have huge number of unprocessable message (e.g ack, or may be other cause), the adapter queue stop picking up the new valid message that get into the inbox. #149

Open Kusnaditjung opened 3 years ago

Kusnaditjung commented 3 years ago

Bugs This is highly likely to happen in live environment, so this is quite an important issue to be handled. If there is a high number of unprocessable message (e.g 60 or something like that), and new valid message coming in, the adapter will not pick the new message.

**Step to reproduce ***

  1. Put a number of non-pathology and non-screening messages into the mailbox (e.g 70)...
  2. Start putting a valid pathology and screening, check that the adapter does not pick the new message into the active mq queue.

Importance Quite high

Expected behavior The adapter should have a cache of the messages that do not qualify to be processed by pathology and screening adapter, so a new message should always be processed (the number of messages processed in each cycle should discount the already known non-procesable messages).

bartek-sarul commented 3 years ago

This is a known limitation caused by how MESH can be used. Adaptor first gets a list of MESH message ids and then iterates through all of them downloading each and checking it's workflow id. If it doesn't match pathology or screening, message is ignored and left in the mailbox.

You sugestion to have a cache seems like a good improvement. We'll take that into consideration.

Kusnaditjung commented 3 years ago

@bartek-sarul we have been implemented a mesh client for our own project, and that is the approach we have taken (cache unprocessable message ids) to avoid this problem (and also avoid downloading the messages on every cycle). These messages are likely to stay in the inbox for seven days before they are cleared by the MESH server as uncollected, once they are cleared, the cache will be updated.