Extract graph extraction to its own package and CLI

[GuillaumeDesforges]

Other use cases than exploration consume the data about packages and their dependencies from Nix files defining derivations. Sometimes, for other flakes than nixpkgs. This is the case for instance for NLNet founded project SBoM generation (cc @rapenne-s). However, they are not always interested by the graph database, API and UI we provide. Even more so, their requirements forbid from using them.

In order for this project to be helpful to more people, we want to make our work usable to other parties that need the same data as we do.

In a meeting with everyone from the project (see minute), we agreed we should extract the part of the ETL that evaluates Nix code and outputs the raw graph data to its own reusable piece.

This piece should be usable

• in Python (API)
• in the command line (CLI)

This leads to the following tasks:

• [x] extract the Nix extraction logic to its own package #55
• [ ] build a thin wrapper as a CLI #57

[zz1874]

Since we would like to extract the graph extraction to multiple uses, we can follow an approach from https://medium.com/opendoor-labs/our-python-monorepo-d34028f2b6fa to build a monorepo-like structure.

We might create a new subdirectory called libs and move the nixpkgs-graph.nix from etl to it. A possible structure can be like this

.
├── api
├── config
├── docker
├── docker-compose.dev.yaml
├── docker-compose.yaml
├── etl
│   ├── Dockerfile
│   ├── etl.py
│   ├── example_query.py
│   ├── README.md
│   ├── requirements.txt
│   ├── test-requirements.txt
│   └── test_spec.py
├── flake.lock
├── flake.nix
├── js-client
├── libs
│   └── extract-nix-data
│       ├── main
│       │   ├── __init__.py
│       │   ├── main.py
│       │   └── nixpkgs-graph.nix
│       ├── pyproject.toml
│       ├── requirements.txt
│       └── tests
├── LICENSE
├── README.md
└── scripts

, and we can create CLI based on it.

What do you think?

[GuillaumeDesforges]

I'm all for eating our own food :) cc @smelc

[GuillaumeDesforges]

I can suggest the following

• nix-graph-core: a shared library (Python package) that provides a Python API to extract data from a flake to different output formats
• nix-graph-cli: a CLI that uses the shared nix-graph-core

Then we'll have nixpkgs-graph-api, what @dorranh did, but I don't think it'll need to use the core library.

Thoughts?

[dorranh]

@GuillaumeDesforges I could potentially be up for that. To me it feels like the size of the package does not quite warrant the fission into separate core and cli packages, but that is a matter of personal preference, and I'm happy proceeding either way.

[dorranh]

PS: In general I agree that borrowing from the structure proposed in our recent blog post is a good idea, @zz1874.

[zz1874]

Yes that makes good sense!