Prepare for integration with `vere`

[matthew-levan]

@midlyx-hatrys will work on preparing seguro for integration with vere. We discussed the following items offline:

• [ ] Clear the db automatically before/after write benchmark
• [ ] Run seguro as a long-lived, listening process that accepts read/write requests and returns responses
• [ ] Use the FDBTenant abstraction of FDB to segregate ships' event logs from one another
• [ ] Review urbit/urbit#5701's saga interface
• [ ] Consider where both ship metadata and ship snapshots should live

[matthew-levan]

@midlyx-hatrys how's it going with this? Any questions?

[midlyx-hatrys]

Oh good, I was just wondering where to do questions, let's do them right here. :)

All I got to do up to now with Seguro was getting familiar with the sources and some experimentation; most of the following are observations and not questions as such. Would appreciate if you could confirm those and/or clarify the historical motivations.

• Fragment encoding
  • I've dumped my planet's lmdb to get some idea about typical event sizes. It seems that the overwhelming majority of events are downright tiny, as in sub-100 bytes (which would make sense for your usual Ames pokes, timer firings etc.). Which raises the question: why does Seguro fragment every event, especially in light of its seeming effort to minimize storage overhead (like the clever bit-banging to compress fragment ids, which also happens to blithely assume little-endian host, which probably only bothers me because I'm old, but). Might as well just write small (sub-10K) events as-is, with keys sans fragment number so they can be detected right away by the key length?
  • Why prepend a header to the first fragment's data? Could append the total number of fragments to its key instead, which would arguably be simpler and also save the extra temp buffer building in add_event_set_transactions().
• The key & fragment header formats as described in the grant text and PROPOSAL.md differ from what the code actually does (those ASCII colon characters, for instance). Should I update the code or the docs?
• Re: tenants: tenant support is still considered experimental in FDB. The only benefit of tenants for Seguro that I can see is not having to include ship patp in event keys, i.e. some space savings (which is not nothing in itself, of course). There's no added security (FDB does not have any auth or the like anyway), and the transaction isolation feature does not seem very relevant (except for timeouts, maybe?). What am I missing here?

I've not digested the saga interface properly yet; does it allow shared memory access? Having to stream event data between (local) Vere and Seguro would be unfortunate.

Also, how would you feel about a Rust rewrite, while Seguro is still small? I'm not actually sure I'll be able to pull that off in a timely fashion anyway (it's sort of a speculative background task now), but assuming me and dear ChatGPT do pull that off? Feels sort of bad to inflict a new piece of systems software on the world in 2023 that is not written in a memory-safe language (but no biggie as far as I'm concerned, to be clear); also writing an async server in Rust would be nicer than in C.

[matthew-levan]

Great questions.

• Fragment encoding?
  • Little-endian won, see well-regarded "blogger" posts on this.
  • I'm open to further discussion on the fragmentation scheme, but they could be borderline bikeshed territory. cc @ashelkovnykov @joemfb
• Bad docs?
  • Yes, thank you for noting this again; I was aware of the mismatch. Please update the docs.
• Tenants?
  • Ah ok, good to know. I hadn't looked into FDBTenant in detail, but if the feature isn't worth the hassle, we don't need to use it.
  • I do wonder what they mean by "experimental" though. Is it planned for stable release?
  • If FDBTenant is similar to a SQL schema (or database, as some implementations call it), it seemed like a reasonable feature to isolate ship logs from one another-- that's all.
• saga?
  • I saw you asked about who's implementing epoc and saga. I'm currently working epoc, but not saga. Perhaps saga will be next, but I'm not sure. Don't spend too much time reading it.
• Rust rewrite?
  • This would be awesome. I think there are a ton of advantages. Do give it a whirl!

[ashelkovnykov]

@midlyx-hatrys

• "fragments every event" I don't know what you mean by "fragments every event". Do you mean that it uses the FragmentedEvent structure? Any event that's under 10,000 bytes in size has its payload directly stored. Yes, it will have a header prepended, but I can't think of a way around this. Even if you had some sort of union going here, you'd still need some way to determine that this is a single-fragment event (e.g. a header to say "there is no header").
• "key sans fragment number" We use the key without fragment number as a guaranteed first entry. For a sub-10,000 byte event, it's exactly what you're describing. You query the key, get the header and payload, and from the header determine that the payload is the entire event.
• "store the # fragments in the key" How would Urbit know the number of fragments into which the event was split?
• docs don't match code That was my decision to apply some optimizations to the initial design. I presented Wexpert Systems with both the design as they originally outlined to me, and another version I thought that they would like, leaving the choice up to them. The code is the ultimate truth, and the docs would ideally reflect that.
• FDB Tenant system I'm not familiar with this feature of FDB, so I have no preference. Seems like it'd be convenient, but it makes perfect sense to me that a business (referring to C1) might not want to make use of an experimental feature.
• saga Not sure what this is - if it was in the original design, I've since forgotten it.
• Seguro in Rust Go for it, if there's a FDB Rust interface and you think there's efficiency + ease of use benefits down the line. The C version isn't going anywhere, always there if it's needed or wanted.

[matthew-levan]

Ah right, there is no Rust API for FDB. Pretty much a showstopper there.

On Thu, Apr 6, 2023 at 7:58 PM Alex Shelkovnykov @.***> wrote:

@midlyx-hatrys https://github.com/midlyx-hatrys

• "fragments every event" I don't know what you mean by "fragments every event". Do you mean that it uses the FragmentedEvent structure? Any event that's under 10,000 bytes in size has its payload directly stored. Yes, it will have a header prepended, but I can't think of a way around this. Even if you had some sort of union going here, you'd still need some way to determine that this is a single-fragment event (e.g. a header to say "there is no header").
• "key sans fragment number" We use the key without fragment number as a guaranteed first entry. For a sub-10,000 byte event, it's exactly what you're describing. You query the key, get the header and payload, and from the header determine that the payload is the entire event.
• "store the # fragments in the key" How would Urbit know the number of fragments into which the event was split?
• docs don't match code That was my decision to apply some optimizations to the initial design. I presented Wexpert Systems with both the design as they originally outlined to me, and another version I thought that they would like, leaving the choice up to them. The code is the ultimate truth, and the docs would ideally reflect that.
• FDB Tenant system I'm not familiar with this feature of FDB, so I have no preference. Seems like it'd be convenient, but it makes perfect sense to me that a business (referring to C1) might not want to make use of an experimental feature.
• saga Not sure what this is - if it was in the original design, I've since forgotten it.
• Seguro in Rust Go for it, if there's a FDB Rust interface and you think there's efficiency + ease of use benefits down the line. The C version isn't going anywhere, always there if it's needed or wanted.

— Reply to this email directly, view it on GitHub https://github.com/wexpertsystems/seguro/issues/3#issuecomment-1499770720, or unsubscribe https://github.com/notifications/unsubscribe-auth/AV2DQREWCKBRFFVIHBCITOLW75KAVANCNFSM6AAAAAAWNJCND4 . You are receiving this because you authored the thread.Message ID: @.***>

[midlyx-hatrys]

https://github.com/Clikengo/foundationdb-rs (does not seem official, not tested it yet, but looks serious, packaged as a crate and all)

oh wait, this one does look official (more up-to-date anyway): https://docs.rs/foundationdb/latest/foundationdb/ (which seems to come from the same origin, but the latest available crate is 0.7.0 (and has fdb 7.1 support) whereas the latest tag in the seeming source repo is 0.5.0 from 2+ years ago. weird)

[matthew-levan]

Seems risky. Not sure the benefit outweighs the downsides. We should move ahead with our existing codebase, in my opinion.

On Fri, Apr 7, 2023 at 8:57 AM midlyx-hatrys @.***> wrote:

https://github.com/Clikengo/foundationdb-rs (does not seem official, not tested it yet, but looks serious, packaged as a crate and all)

— Reply to this email directly, view it on GitHub https://github.com/wexpertsystems/seguro/issues/3#issuecomment-1500265421, or unsubscribe https://github.com/notifications/unsubscribe-auth/AV2DQRAXBQFMR3AOZWJLW4DXAAFKLANCNFSM6AAAAAAWNJCND4 . You are receiving this because you authored the thread.Message ID: @.***>

[midlyx-hatrys]

So let's see what I need to do (not sure yet how exactly this maps to a proper phase 3 proposal, but it probably should at least inform it):

Innards

• The existing "generic" FDB layer code is sync, but should be async to properly utilize FDB. Seguro already has a working async FDB layer (on the write path), but only for benchmarking purposes
• Events must be per-ship, whether by way of prefixing keys with patp or using per-patp tenants. What about fake or testnet ships?
• FDB transactions are overkill for the ship event logging use-case -- events are totally separated by definition, which means that it does not matter at which point a transaction (which is effectively a DB snapshot) is created. Currently transactions are created too early, which has 2 problems:
  • Increased risk of transactions timing out by the time they are committed, especially under load
  • Unfriendliness to debugging (with gdb, that is), for the same reason of transactions timing out 5s after they are created -- if you put a breakpoint between creating a transaction and committing it, oops
  • Basically every transaction should be created, filled-in, and committed in one logical operation, i.e. totally abstracted out

    Seguro <-> Vere interface

• Decide on the desired deployment use-case: does the Seguro service always run on the same machine as the ships that use it, or should it permit remote usage? This has implications on what the interface ends up being like, i.e. is event data streamed to/from Seguro or we can use shared memory
• Spec a protocol. It obviously needs versioning, but beyond that I have no idea: HTTP-based or just plain sockets (would have to use plain sockets to pass shared memory descriptors in any case)? Auth or no auth? Etc., etc.

[matthew-levan]

Nice, good thoughts here. I think streaming makes sense, and auth can be a feature left for the future.

On Mon, Apr 17, 2023 at 7:48 AM midlyx-hatrys @.***> wrote:

So let's see what I need to do (not sure yet how exactly this maps to a proper phase 3 proposal, but it probably should at least inform it): Innards

• The existing "generic" FDB layer code is sync, but should be async to properly utilize FDB. Seguro already has a working async FDB layer (on the write path), but only for benchmarking purposes
• Events must be per-ship, whether by way of prefixing keys with patp or using per-patp tenants. What about fake or testnet ships?
• FDB transactions are overkill for the ship event logging use-case -- events are totally separated by definition, which means that it does not matter at which point a transaction (which are effectively DB snapshots) is created. Currently transactions are created too early, which has 2 problems:
  • Increased risk of transactions timing out by the time they are committed, especially under load
  • Unfriendliness to debugging (with gdb, that is), for the same reason of transactions timing out 5s after they are created -- if you put a breakpoint between creating a transaction and committing it, oops
  • Basically every transaction should be created, filled-in, and committed in one logical operation, i.e. totally abstracted out

Seguro <-> Vere interface

• Decide on the desired deployment use-case: does the Seguro service always run on the same machine as the ships that use it, or should it permit remote usage? This has implications on what the interface ends up being like, i.e. is event data streamed to/from Seguro or we can use shared memory
• Spec a protocol. It obviously needs versioning, but beyond that I have no idea: HTTP-based or just plain sockets (would have to use plain sockets to pass shared memory descriptors in any case)? Auth or no auth? Etc., etc.

— Reply to this email directly, view it on GitHub https://github.com/wexpertsystems/seguro/issues/3#issuecomment-1511191917, or unsubscribe https://github.com/notifications/unsubscribe-auth/AV2DQRFR5IDLF474EXTP4ZTXBUUXXANCNFSM6AAAAAAWNJCND4 . You are receiving this because you authored the thread.Message ID: @.***>