UI: Add search filter option to incoming messages and connection logs

[thfries]

In order to analyze and navigate in long lists of incoming message table and also in connection logs, it would be good to have a search filter in top of the table

example filters for connection logs:

• direction:inbound type:consumed level:success
• timestamp<1.1.2900
• correlationId:4711
• thingId:234235

example filters for incoming messages:

• feature path that was updated if you want to watch a specific feature
• ?

The filter should be easily filled by e.g clicking on the connection metrics or somehow on the correlationId/thingId of one log entry. In case of incoming messages the filter should be set somehow by selecting a feature.

[thfries]

@thjaeckle, pls assign this issue to me. Thanks

[thjaeckle]

@thfries good idea, I also was thinking about that. For the incoming messages, you could even provide an RQL expression and do server side filtering (when opening the SSE / EventSource).

Eg in order to receive all updates to a certain feature only:

exists(feature/important)

Problematic is that you will miss updates to all other features and cannot simply change the filter to show eg already received updates of other features.

Benefit is that this is super efficient and also makes use of the same RQL as for the search.

I assume you want to do client side filtering?

[thfries]

Yes, I just thought about client side filtering - for the same reason as you mentioned. I see the purpose more like a log for debugging. So you don't know before what you are searching for. I thought already that it is a drawback that you need to select the thing before that you want to "watch".

Since the fix for #1712, the SSE has become verbose. I remember now that before we merged the "delta" into the known state of the thing on the client-side.

Can you nowadays still see what part of the Thing was modified? We have a lot of different components on the edge that in parallel update different features of a thing, so it was nice to see the deltas only. So you can better debug what the edge components are doing (if they update full features or modify only the relevant part of a feature.

So I would argue that also for that reason it would make sense to reduce the SSE events as much as possible to the modification that happened to the Thing. Don't know if #1712 can still be fixed with that condition.

Btw. I am evaluating different options for the filtering. Next to plain javascript, Lodash would provide some support (which is no longer used) and also JSONata looks interesting. Currently I tend to plain javascript. Any recommendation or opinion from your side?

[thjaeckle]

Can you nowadays still see what part of the Thing was modified? We have a lot of different components on the edge that in parallel update different features of a thing, so it was nice to see the deltas only. So you can better debug what the edge components are doing (if they update full features or modify only the relevant part of a feature.

So I would argue that also for that reason it would make sense to reduce the SSE events as much as possible to the modification that happened to the Thing. Don't know if #1712 can still be fixed with that condition.

We would still be able to do that by not selecting all fields of the thing as "extraFields". The benefits of selecting all the fields is that Ditto can correctly merge in the backend. Which is just not possible in the frontend one, because of deletions via PATCH, which don't show up in the delta. So #1712 can't be fixed if reverting to the old behavior.

We could either make the behavior configurable (per environment for example or directly on the things page) or we could even subscribe for 2 event streams, one with complete thing for the json and features views and editors. And one for the incoming messages at the bottom.

By the way: with Ditto 3.4, selecting the new "Context only" option at the incoming messages select will show only the delta, available in "_context/value". That's why I added the select in the first place, because I could also no longer quickly see the delta.

[thfries]

Hi @thjaeckle, understood the _context now. Works good. Let s keep it as is for now. Did you notice that the feature list is changing the order in case of incoming SSE? May be we should sort the feature list by name to avoid this?

Btw. JSONPath seems to do the job 😀

[thjaeckle]

Did you notice that the feature list is changing the order in case of incoming SSE? May be we should sort the feature list by name to avoid this?

Yes, I did notice and I am still wondering if this is useful or annoying 🤔 You will get the most recently updated feature sorted at the top of the list, which is sometimes really useful 😉

[thfries]

Hi @thjaeckle,

after observing it for a while: It is definitely annoying imho. 😀 If you like that info for "latest updated", we may add a new marker next to the feature name. We could even count the number of updates per feature with a badge and highlight the latest one in a different color.

wdyt?

[thjaeckle]

@thfries good morning :) Yes, I have to agree - it is annoying and confusing to always search an expected feature at a different place.

Maybe highlighting with a color could work well, yes .. E.g. for the table cell background do a background color with opacity depending on the most recent updated - e.g. color the 5 most recently updated features - the most recent one having the strongest opacity and the least recently updated of the 5 having the weakest opacity.

On the other hand, a badge might probably be less "intrusive".