DepViz offers a range of useful features, but its current configuration can be difficult to customize and modify. To make the most of its existing capabilities and to facilitate future development, we should strive to make it more user-friendly and open to customization.
Suggested changes:
Split the fetcher/processor so that people can easily use DepViz to fill and maintain a database that they can access manually.
We should consider switching to a database that is easier to use. Currently, we are using a graph database which is excellent for graph queries but is challenging to manage, limited in some ways, and has few clients. It may be overkill for our needs. We should switch to an RDBS like Sqlite by default and use recursive queries for the graph. A good caching feature can address most of the worst cases, and then we can add arbitrary limits.
Alternative: I suggest using SQLite and a graph database rather than switching. SQLite should be used as the primary database for fetching, filling, checking, and updating, which should be enough for many usages. Then, a dedicated script/helper should generate a graph database from the SQLite database, allowing a web interface or other applications to access the graph engine when needed.
In general, I suggest revamping the CLI to have specific actions by default, i.e.,
depviz fetch DRIVER -> subcommand with all fetchers, each fetch has its options; easy to extend.
[x] depviz fetch github
[ ] depviz fetch trello
depviz db SUBCOMMAND -> subcommand to manage the database
[ ] depviz db to-graphdb
[ ] depviz db query
[ ] depviz db shell
[ ] depviz db cleanup
[ ] depviz db sync-with-airtable
depviz gen SUBCOMMAND -> subcommand to use the db to generate outputs, without requiring any fetch
[ ] depviz gen graphviz
[x] depviz gen json
[x] depviz gen csv
later we'll try to make compositions at the top level for the most famous flows.
Adding a download status on entities, so that, instead of having to fetch something missing, we can create an empty object with status "TO_DOWNLOAD"; and then wait for a fetcher to process it. it will solve the diamond dependency we had in a previous PR.
DepViz offers a range of useful features, but its current configuration can be difficult to customize and modify. To make the most of its existing capabilities and to facilitate future development, we should strive to make it more user-friendly and open to customization.
Suggested changes:
depviz fetch DRIVER
-> subcommand with all fetchers, each fetch has its options; easy to extend.depviz fetch github
depviz fetch trello
depviz db SUBCOMMAND
-> subcommand to manage the databasedepviz db to-graphdb
depviz db query
depviz db shell
depviz db cleanup
depviz db sync-with-airtable
depviz gen SUBCOMMAND
-> subcommand to use the db to generate outputs, without requiring any fetchdepviz gen graphviz
depviz gen json
depviz gen csv
┆Issue is synchronized with this Trello card