Databases as the window to life, the universe, and everything

[SuperSonicHub1]

One hypothesis for why cross-app collaboration is so hard is because data is siloed. This can be fixed by having a universal query substrait which can then be easily connected to lots and lots of "database-shaped" sources.

[SuperSonicHub1]

Previous work:

• musquery, an SQLite-based music metadata management system
• RequestQL, a GraphQL-based DSL for specifying web scraping operations
• trustfall-rustc-adapter: a nicer query front-end to the rustc database
• osquery
• TabFS
• SQLite virtual tables
  • Zipfile
  • Bytecode
  • CSV
  • JSON
• Databases as interfaces
• Uniform interfaces

[SuperSonicHub1]

Query substraits:

• Ibis
• Steampipe
• Shillelagh
• Madata
• osquery
• Virtual tables/foreign data wrappers
  • SQLite
  • Postgres
  • Postgres
  • SQL/MED

[SuperSonicHub1]

Writings:

• Alexander Obenauer and itemized computation
  • The Potential Merits of an Itemized OS
  • The Graph OS
  • Gestural view construction
  • OLLOS: an itemized personal computing timeline
• Bring Your Own Client
• The program is the database is the interface

[SuperSonicHub1]

Challenges and potential solutions:

Assembling adapters is hard

• Normalizing and reshaping real world data is either very hard or super-tedious/anti-DRY.
  • Discussion of DSLs for solving this issue is deserving of its own issue.
• Very easy to overscope; only consider personal use-cases first!
  • Something Predrag drilled into me during development of trustfall-rustc-adapter.
• Caching expensive data sources is hard because cache invalidation is hard. :)
• The sync vs async war.

Here lie:

• Trunkless
• Hacker News to Schema.org
• BrowniePoints
• comment-dl

Actually querying

Going from having an idea of a query to a working piece of code is not trivial. Going from exploration to writing a query needs to be entirely transparent. Discussion of query DSLs and query visualization is deserving of its own issue.