Live vs. static feeds for events

[caesay]

Issue

There should be a differentiation between a live feed and a static feed. A static feed would be anchored at a specific element id so new elements would not shift the list.

For instance, currently the only mode is live, lets say you're on page 5 of an events view, events are still piling in, but you might want to be looking at that specific page. You shouldn't need to follow the events constantly by going to the next, next, next page.

Suggestion

There might be a (Live|Static) selector at the top of an events feed (defaults to live), clicking to the next page will automatically transition to Static mode so that the page events do not change. You now should be able to navigate through the pages in static mode. Clicking Live mode should take you back to the first page with live events. Also navigating back to the first page might also take you into Live mode automatically.

Implementation

A very simple execution of this would be to replace the current paging mechanism (in the first instance without a selector at all). When you click "next" instead of using a page id it will use the last element in the current view as the anchor to start the next view and send that to the server. The server calculates the next X elements from that ID and returns it. Clicking next again does the same thing but with the last element from the current page. Clicking X (ie, 5th page) would need an additional step of requesting for the page by number and then switching to using an id anchor for the next request.

[niemyjski]

I think maybe we should just do this without a selector. If you are not on the first page we don't update the list at all. If you go back to the first page or the filters / route changed then we go back to updating (already logic there to switch pages). What are your thoughts on this? There is a method on the event and stack lists called CanRefresh... The question is what do we do for things like deletes or bulk updates where events were marked as fixed (I guess keep it static)?

Any chance you'd be willing to submit a pr for this?

[caesay]

bulk changes are a non-issue really as long as we use an anchor, I could be on page 5, and all the events from pages 2-4 could get deleted and it won't matter because I'm anchored at a specific element. I think we should still do the refreshing, if an element gets deleted on the page i'm looking at then I'd still want to see that.

The actual page "number" part of it can be a fake concept and the current 'page' could be calculated when we refresh by the server if you really wanted to show something. For example: If i'm looking at page 5 and a few pages get deleted (2-4) then the page number i'm looking at in the UI might get updated to show page 2 but I should still be looking at the same events, and going to the "next" page should always show me the events following what i'm looking at regardless of what the actual number of any page is.

What are your thoughts on this?

[niemyjski]

I'm not sure how it would be possible to even do this. Events can be filled in based on the submitted events and events can be updated at anytime with hidden fixed or deleted. We don't keep track of events that you've seen. Except for not refreshing at all I don't know how this would be possible. @ejsmith thoughts?

[caesay]

I'm not entirely sure what the confusion is - maybe I'm missing something? I'd imagine an endpoint that takes an anchorId (an id of an event or stack) as a parameter instead of a page. The server then returns the next limit items starting from the anchor. The implementation of the above is trickier said than done, in SQL you could use ROW_NUMBER() to find the row number of the anchorId and then a skip. It doesn't matter what happens to anything before anchorId in the result set now.

Simply not updating is definitely not a solution - because when i click "next page", i could end up staring at the same events I was just looking at if 10 events have come in since then. Also if i share the URL i'm looking at like in exceptionless/Exceptionless#1281 then when they open the page they probably won't be seeing the same events as me anyways because events are continually rolling in.

[caesay]

ElasticSearch supports this natively, so we just need to expose this instead of a page number: https://www.elastic.co/guide/en/elasticsearch/reference/master/search-request-search-after.html

[niemyjski]

Yes, that would work for the most recent lists, but I doubt it would work for the most frequent or ones sorted by user counts. It's worth looking into but the data could be always changing. Also our experience with scrolling api which it seems like this uses something familiar doesn't always work the greatest. Like you will sometimes get different results if you reuse the same scroll id (it says it uses scrollid, but then also says it's stateless), they are also short lived and there isn't a guaranteed next page after a scrollid, even if there is data there. So if this is a live cursor and not scrolling, how long does it live for?

search_after is not a solution to jump freely to a random page but rather to scroll many queries in parallel. It is very similar to the scroll API but unlike it, the search_after parameter is stateless, it is always resolved against the latest version of the searcher. For this reason the sort order may change during a walk depending on the updates and deletes of your index.

[niemyjski]

Also, thanks for pointing out that api, I don't think I've come across that before :) @ejsmith do we even support the search after with our parser/repos?

[ejsmith]

Yeah I see how this could be frustrating when you have a lot of data coming in.

We could probably change paging to be based on the id of the last document in the list. Give me the next page of docs where the id is less than last id on the current page of the grid. It's going to be hard to show page numbers with that approach though. I think we would just be able to show first page, last page and next page buttons.

[rstarkov]

If implemented, please consider making the static feed show actual timestamps instead of "x hours ago". The motivation is that if I'm looking at a static feed, I am probably correlating the events with some other log or data source, and the exact time is more important than a very approximate whole-number-of-hours ago (and hovering on each timestamp is really slow).

[niemyjski]

I agree, would be good to show the static times, we could probably even make this a setting in the options pretty easily.

[caesay]

fwiw, a scrolling API is not what you'd wan't either.search_after is very different and is exactly what you want. You want to retrieve all results that come after a specific result id. That result id would be in the url param and would allow you to share any page of data to anyone by sending the link.

[niemyjski]

I think we could definitely do this and may want to consider to moving only to paging with search after but we'd have to ditch our paging control completely (you'd only have a previous and next). Also we'd have to think of a url suffix for this. I kinda feel like we should just use query string parameters for this or do something like /before/id. The problem with this approach is on dashboards with multiple lists your routes break down a bit at having a friendly sharable name :.