Closed GuillaumeDesforges closed 1 year ago
Since we would like to extract the graph extraction to multiple uses, we can follow an approach from https://medium.com/opendoor-labs/our-python-monorepo-d34028f2b6fa to build a monorepo-like structure.
We might create a new subdirectory called libs
and move the nixpkgs-graph.nix
from etl
to it. A possible structure can be like this
.
├── api
├── config
├── docker
├── docker-compose.dev.yaml
├── docker-compose.yaml
├── etl
│ ├── Dockerfile
│ ├── etl.py
│ ├── example_query.py
│ ├── README.md
│ ├── requirements.txt
│ ├── test-requirements.txt
│ └── test_spec.py
├── flake.lock
├── flake.nix
├── js-client
├── libs
│ └── extract-nix-data
│ ├── main
│ │ ├── __init__.py
│ │ ├── main.py
│ │ └── nixpkgs-graph.nix
│ ├── pyproject.toml
│ ├── requirements.txt
│ └── tests
├── LICENSE
├── README.md
└── scripts
, and we can create CLI based on it.
What do you think?
I'm all for eating our own food :) cc @smelc
I can suggest the following
nix-graph-core
: a shared library (Python package) that provides a Python API to extract data from a flake to different output formatsnix-graph-cli
: a CLI that uses the shared nix-graph-core
Then we'll have nixpkgs-graph-api
, what @dorranh did, but I don't think it'll need to use the core library.
Thoughts?
@GuillaumeDesforges I could potentially be up for that. To me it feels like the size of the package does not quite warrant the fission into separate core
and cli
packages, but that is a matter of personal preference, and I'm happy proceeding either way.
PS: In general I agree that borrowing from the structure proposed in our recent blog post is a good idea, @zz1874.
Yes that makes good sense!
Other use cases than exploration consume the data about packages and their dependencies from Nix files defining derivations. Sometimes, for other flakes than nixpkgs. This is the case for instance for NLNet founded project SBoM generation (cc @rapenne-s). However, they are not always interested by the graph database, API and UI we provide. Even more so, their requirements forbid from using them.
In order for this project to be helpful to more people, we want to make our work usable to other parties that need the same data as we do.
In a meeting with everyone from the project (see minute), we agreed we should extract the part of the ETL that evaluates Nix code and outputs the raw graph data to its own reusable piece.
This piece should be usable
This leads to the following tasks: