Datascript DevTools

[tangjeff0]

We currently have re-frame-10x setup, but it doesn't work with re-posh, which is one reason we are relying less on re-frame and re-posh (#95).

Similar tools in this space are fulcro-inspect (demo), redux-devtools, and Elm debugger (demo).

We do not need every dev tool feature like performance statistics.

What we do want is:

• watchers: datascript entity/q/pull queries should update visibly and reactively if the db is updated
• transactions: all the things we can expect from Datomic — datoms (assertions or retractions), before and after diffs, timestamps, etc.
• time-travel: one of the best features of immutability :)

I'm not exactly sure the best implementation strategy. Options I can think of off the top of my head:

• make our own developer sidebar — easiest IMO
• fork and instrument re-frame-10x ourselves — not a good idea IMO, don't want to look at their code, and we are divesting from re-frame
• make our own Chrome Extension — most powerful, but we don't need to leverage custom tabs in Chrome yet

[jelmerderonde]

Some initial thoughts on implementing this:

• posh seems to provide the watchers we want. Its functions return a Reagent Reaction that can be observed by components that will reactively update when the db changes.
• We can watch datascript transactions using datascript's listen!. It should be easy enough to display the transaction reports in some ui.
• Time travel also seems reasonably easy, at least in the context of the datascript database: we can reset the database using reset-conn! with an earlier database value :db-before from the transaction reports we collect using listen!.

Questions

• I'm unsure how much I can reuse from the Datascript explorer (#67). Maybe it is as simple as replacing the datascript q, entity and pull functions with their posh equivalents. But it seems unlikely that the returned Reaction will survive the transition from sci. Will have to experiment with this.
• Another open question is how to visualize all these elements. UI design is not my strongest point 😉.
• The whole thing becomes more complex if we also want to monitor other parts of the state (those not in the datascript db). Is that something we want?
• For now I'm proceeding with the idea of turning it into a developer sidebar. But if there are good arguments for a Chrome Extension I can look at that too (although I prefer Firefox myself).

Discovered

• [x] Monitor the datascript conn using listen!, collect transaction reports and display them.
• [x] Implement time travel using reset-conn!.
• [x] Experiment with sci and posh.

I will proceed with some small experiments to check if my assumptions are correct and how sci and posh function together.

[tangjeff0]

• I'm unsure how much I can reuse from the Datascript explorer

  • Yeah, I'm not sure if we can reuse too much code there, or how well sci and posh play together.

• how to visualize all these elements

  • Just show the data™. All the dev tools above, REBL, Datomic Console, etc. are just simple layouts, inputs, and tables. Once we have the data, I don't think it will take @athensresearch/designers long at all to do UX.

• The whole thing becomes more complex if we also want to monitor other parts of the state (those not in the datascript db). Is that something we want?

  • We can still use re-frame-10x! Not mutually exclusive.

• For now I'm proceeding with the idea of turning it into a developer sidebar.

  • Apparently, it's becoming easier to develop cross-browser extensions. But what if we use tauri 😉

@jelmerderonde

[jelmerderonde]

Alright, more updates!

• sci and posh seem to work fine together. If you return a Reaction from an eval done by sci it still triggers updates when new transactions occur.
• I've got basic time travel working, although I do have some reservations about the way it is implemented now. Datascript's reset-conn! function basically creates a transaction that (1) removes all existing datoms and (2) creates the datoms from the db-value you pass it. A cleaner solution probably exists.
• Because reuse of code from the Datascript Exlorer (#67) was possible, I've refactored the browser component into a seperate namespace (athens.devcards.data-browser).

Discussion

• To some extent the data browser obfuscates the actual data. It renders data into rows and columns, but hides the actual shape of the data. For the Datascript explorer this might have been ok, but for a development tool this could be problematic. REBL and other data browsers have some solutions that we could use, like showing what function is used to render the data (for example coll-of-maps or tuples).
• Refactoring the data browser exposed some things that could be better. It should be simple for the caller to provide rendering functions for rows and cells.
• Posh has different query semantics, as @tangjeff0 noted earlier: https://github.com/athensresearch/athens/issues/95#issuecomment-642683723. This could be confusing if you're expecting datascript semantics.
• Using sci allows users to do other transformations. This could be a valuable feature, but complicates watches, because user defined transformation do not necessarily return Reactions.

There might be a simple (and possibly naive) way to deal with the last two points: don't use posh for querying, just supply the datascript functions in the sci bindings. Instead, we just re-evaluate the query every time a datascript transaction occurs. This is less sophisticated, because re-evaluating will occur even if the will query return the same results.

Discovered

• [x] Improve the data browser: show the renderer that's used
• [x] Research how other devtools implement data browsing
• [ ] Create a list/detail view for the transaction panel
• [ ] Research other ways of implementing time-travel
• [x] Try out the naive way to implement watchable queries?

[jelmerderonde]

There might be a simple (and possibly naive) way to deal with the last two points: don't use posh for querying, just supply the datascript functions in the sci bindings. Instead, we just re-evaluate the query every time a datascript transaction occurs. This is less sophisticated, because re-evaluating will occur even if the will query return the same results.

@tangjeff0 what do you think about this solution?

[tangjeff0]

@jelmerderonde in general I have a bias towards quick, naive first passes.

In this case, definitely go for it. Posh doesn’t offer full datascript features anyways. I mentioned lack of pull in query. But it also doesn’t bridge the entity API.

[tangjeff0]

Something else I am trying to figure out — it seems I’m not getting reactive updates for posh pull.

[jelmerderonde]

Just a small update, didn't have much time last week to work on this unfortunately. I reworked the data browser, it is now much more extensible and uses datafy and nav.

Still to do:

• [x] Fix some bugs with the Reagent reactions
• [x] Reimplement tuples view
• [ ] Research other ways of implementing time-travel
• [ ] Create a list/detail view for the transaction panel
• [x] Implement Datafiable for transaction reports
• [x] Make it into an actual panel you can trigger with a key combo

Latest version (not fully usable yet) can be found at: https://jelmerderonde.github.io/athens/cards.html#!/athens.devcards.devtool. Load some real data, but you first have to click "Transactions" and then "Query" to get it to work.

[jelmerderonde]

Just added a draft PR #164 to show progress so far. Some things still to be done, but the major functionality seems to be there.

[jelmerderonde]

Another update! Just set PR #164 as "ready-to-merge". It still needs some work, but I think it is at a stage where it would benefit from some help from others as well.

The devtool-panel is implemented in the actual app and you can summon it with ctrl-g.

Still todo:

• [ ] Research other ways of implementing time-travel
• [ ] Create a better view for the transaction panel
• [ ] Fix the styling (help wanted, because my css-fu is not so good)
• [ ] Squash a bug with navving that seems to have returned lately

[tangjeff0]

@jelmerderonde I think this issue is amazing. We have time travel with ctrl-z and ctrl-shift-z. You are free to continue working on this, but I'm really happy with it right now. I think further improvements would make this its own library/open-source package 😄