search

DavHau / pypi-deps-db

Dependency DB for python packages on pypi
MIT License
66 stars 40 forks source link

Host full database as a static file #3

Open izzues opened 2 years ago

izzues commented 2 years ago

Hey, thanks for the great work!

Have you seen this?

https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/

The basic idea is that it's possible to host an entire SQLite database on GitHub Pages and query it using HTTP Range requests (so there's no need to download the entire database).

This might be a great way to make the data more accessible and especially relevant to mach-nix since querying the database can be made significantly faster for slower connections. Also relevant to #1.

What do you think?

DavHau commented 2 years ago

The problem I see with this is that we cannot do arbitrary web requests during a nix build without providing the sha256 for the response upfront. We would basically need to provide a sha256 for every possible answer of this sql database. I don't see a way how this could be implemented as a nix build.

We could do what many nixpkgs fetchers do: Write a nix expression with an empty hash -> execute the build -> let the build fail with hash mismatch -> enter the correct hash -> rebuild. Whenever the requirements or database version is updated, the user would have to go through this process again. Alternatively we could automate that via tooling, but then maintaining a mach-nix environment requires an external tool.