Identical revision/commit for multiple events

[askasp]

This might be something I don't understand about eventstore and not a Spear problem at all.. However, I'm trying to set the :from parameter on a subscription to a $by-category stream.

On other libraries, it's common to make a position from the commit_position and prepare_position values. In spear this is wrapped by just providing the last event you want to start the catch up subscription from (I really like that feature!)

Unfortunately, I seem to get all events regardless on the from: event. Further, by inspection I see that the commit_position and prepare_position is equal for many of my events. Here are three events in different streams that all have the same value for commit and prepare. Is this intentional from eventstore? Is there any way to specify the from: parameter on a $by-category stream?

%Spear.Event{
  body: %{"hadet" => "e983a588-a6da-11eb-9840-f02f742e9919"},
  id: "852b72a9-05a8-4315-93b7-7fd73be6e316",
  metadata: %{
    commit_position: 18446744073709551615,
    content_type: "application/json",
    created: ~U[2021-04-26 22:01:11.574827Z],
    custom_metadata: "",
    prepare_position: 18446744073709551615,
    stream_name: "jaa_api_live-stream5",
    stream_revision: 0,
    subscription: #Reference<0.3430913167.73400327.38263>
  },
  type: "second"
}
%Spear.Event{
  body: %{"hadet" => "ea609394-a6da-11eb-83d4-f02f742e9919"},
  id: "ddd15ce0-213f-43c4-83cc-ab8063978e9c",
  metadata: %{
    commit_position: 18446744073709551615,
    content_type: "application/json",
    created: ~U[2021-04-26 22:01:13.022904Z],
    custom_metadata: "",
    prepare_position: 18446744073709551615,
    stream_name: "jaa_api_live-stream6",
    stream_revision: 0,
    subscription: #Reference<0.3430913167.73400327.38263>
  },
  type: "second"
}
%Spear.Event{
  body: %{"hadet" => "eb86aa10-a6da-11eb-a356-f02f742e9919"},
  id: "0fa0c07f-33d5-4ab6-b5a4-b6a535a8f0c5",
  metadata: %{
    commit_position: 18446744073709551615,
    content_type: "application/json",
    created: ~U[2021-04-26 22:01:14.949941Z],
    custom_metadata: "",
    prepare_position: 18446744073709551615,
    stream_name: "jaa_api_live-stream8",
    stream_revision: 0,
    subscription: #Reference<0.3430913167.73400327.38263>
  },
  type: "second"
}

[the-mikedavis]

ah I think this is a bug in spear, nice find! :slightly_smiling_face:

For projected streams you need to provide the link revision/positions and the translation to t:Spear.Event.t/0 is currently discarding the link information.

Projected streams (like $by-category streams) don't copy the full event to the projected stream, just a very thin pointer event: just enough information to go find the full event in its origin stream. When reading projected streams, there's an option in the protobufs (see here) to resolve those links to the origin events which Spear sets to true by default. When that's set to true, the :event field in the Spear.Records.Streams.read_resp/0 gets set to the real event (not the link) and the link event gets set in a :link field.

Currently spear discards the link information by default since it rarely has any important information in it and links are kind of annoying to deal with in general, but I totally forgot that you need the link to track position in a projected stream! Links are a headache :facepalm:

Luckily I think the fix for this is pretty small: put the necessary link information into the Spear.Event.metadata packet and use that information for stream position when reading with the :from option. I'll get a fix going in just a min :+1:

[the-mikedavis]

v0.8.0 should have fix for this, looks like it got published right now :+1:

fwiw I think it's normal to have many events with the same commit/prepare positions. I'm not totally clear on where those come from but I think it has to do with how EventStoreDB indexes events. If I'm not mistaken, the reason why the subscription kept starting from scratch with your streams is that the Spear.Event.metadata.stream_revision field is 0 for all of those streams in the category

thanks for reporting! :slightly_smiling_face:

[askasp]

Thanks for the quick response! Great package, and great work! :rocket: