Add `updated-at` and `created-at` to bazaar catalog

drunkplato commented 1 year ago

We should do this as a quick fix and provide new scry endpoints to pull the catalog based on the last timestamp we have on the UI.

We could do a more fancy refactor, but this will allow optimized loading now.

See chat-db for scry examples.

%bazaar
%friends
%spaces

Edit: also need delete-log scries to properly keep state in sync with the frontend SQLite tables.

ajlamarc commented 1 year ago

Commenting on my recent architecture thoughts: Eventually everything should be SSS or remote scry.

For small actions (1→1 or peer → ~10 peers) pushing updates (poke) rather than pulling (sub) makes sense. Depending on frequency of bugs with poke-only messing, we should also support scry paths for catching things that fell through.

For large actions (1→100+ peers), subscriptions are practically required. See #988 and #1214. Current culprits are spaces, bazaar, and soon chat-db. (see below example)

Like @drunkplato has mentioned with Rooms, chat-db should have an upper limit on members in a "fully peer-to-peer" chat. This version of chat should be resilient to the host ship being offline, and more buggy behavior is tolerable.

For large chats, messages should go to the chat host first, who maintains SSS connections with the chat members for providing updates. (And eventually we can offer hosting the chat with a load-balanced Holium fleet).

Example

Every time someone in a space sets contact, the ship host has to poke that information to everyone in the space (potentially thousands of pokes).

::  share contact with all members
%+  murn  ~(tap in ~(key by members))
|=  =ship
^-  (unit card)
?~  =(our.bowl ship)  ~
=/  cage  friends-action+!>([%share-contact ship])
`[%pass / %agent [our.bowl %friends] %poke cage]

Versioned pokes and subscriptions

Reading https://roadmap.urbit.org/project/versioned-pokes-subscriptions, this isn't coming natively for a while. They mention Tlon's "pull hook" and "push hook" libraries, but:

causes bloat of 3 agents per 1 real agent
they don't make any sense. It's not clear who is being migrated where.
Knowing Urbit, their kernelspace solution might be impossible or more complicated to implement than the below.

Pull inspiration from agent state: versions + migrations. We need to maintain a solid changelog to provide context for the agents. Some changes will break some actions, i.e. asking for a state value that doesn't exist anymore. Stub those actions with a crash, and the requesting agent should be able to handle that somewhat gracefully.

Pokes

Keep different versions of pokes in different marks like so. JSON decoders happen in mar/tome/action-0/hoon. Then in on-poke check the mark, and then handle the pokes like normal.

+$  tome-action-0
  $%  [%init-tome ship=@t =space =app]
      [%init-kv ship=@t =space =app =bucket =perm]
      [%init-feed ship=@t =space =app =bucket =log =perm]
  ==
::
+$  tome-action-1
  $%  [%init-tome ship=@t =space =app]
      [%init-kv ship=@t =space =app =bucket =perm]
      [%init-feed ship=@t =space =app =bucket =log =perm]
      [%init-sql ship=@t =space =app =bucket =perm]
  ==

The downside (or upside!) of this is that upgrades must specifically be chosen. If you don't update pokes like shown below, you'll keep talking to the old agent. [This is better than forcing a shim. Better to work with the "last known well" version.]

Also, there's a lot more "duplicate" code - matters less since that code is deprecated. Resolve agents becoming hard to read by hiding legacy version code in a library file for that version. We should also permanently remove old versions after a grace period, or it no longer makes sense supporting them.

(poke [ship agent] [%kv-action-0 !>(act)])
::  to
(poke [ship agent] [%kv-action-1 !>(act)])

Scries

Add a ver to the end of all scry paths. This version is used on a per-path basis. A "different" scry path, providing similar functionality, will start over at %0. These need to be documented.

++  peek
  |=  pol=(pole knot)
  ^-  (unit (unit cage))
  ?+    pol  ~|(bad-peek-path/pol !!)
      [%x %kv ship=@ space=@ app=@ bucket=@ %data %all ver=@ ~]
      ...

Subs

Still fuzzy pending SSS work, but they will also be along a path, so they can have a version number.

TODOs

[ ] Enable SSS for %bazaar
[ ] Enable SSS for %spaces
[ ] Enable created-at and updated-at for all agents
[ ] As part of refactor, force agent code into consistent style. Will modify https://www.ajlamarc.com/blog/2023-02-26-urbit-style/ as necessary.
[ ] Enable "versioned pokes and subscriptions" behavior as described above.
[ ] Link Gitbook to Realm repo, create template changelogs / agent documentation. Configure to sync from GitHub to Gitbook
[ ] Think about spaces sur file breaking when being used via other agents. Need more context

won't do

[ ] Apply the above to %notif-db and %chat-db pending more hoon experience

ajlamarc commented 1 year ago

These rules for versioning must also extend to updates (reactions) and views. Scrying for a %3 should receive back a friends-view-3 mark. Poking with a friends-action-2 mark should lead to subscription updates with marks of friends-update-2. So I think versions between pokes / scries / subs will have to be more tied together.

Going to have to run with something like this and tweak the strategy over time. When the agents are consistent + well documented, they will be easier to iterate on.

drunkplato commented 1 year ago

Also the version must map to the state as well, i assume that is just a given.

ajlamarc commented 1 year ago

Also the version must map to the state as well, i assume that is just a given.

I think we'll always want to map to the most recent state instead of managing multiple concurrent states. Ex. if a side effect of an old poke is to set is-public to True but is-public no longer exists, it does nothing instead.

holium / realm