Introduction of a personal intelligence workload

[sdake]

This demonstration inferences a llama2 model on our hardware. The personal intelligence is (not particularly) intelligent, just yet.

A note to reviewers. Please take a look at the packaging model. I really do want to stay near and dear to the Python community vibe. I am not opposed to alternate packaging approaches. Any software within the runtime really does need to be packaged using whatever approach the ecosystem we integrate with typically uses. For this example, the code is Python, so we need to use "Pythonic" packaging. Brew, Nix, Portage, etc, these are all alternatives. We need to pick our Pythonic packaging and ensure any workloads use that packaging model.

My proposal for Pythonic packaging is realized in this PR. I also propose we do not hedge.

Cage match?

Alternative packaging may be a distraction.

Alternative packaging may also generate demand...

Therefore, I propose Nix (or Brew) should be documented, if the overhead vs demand generation is in our favor.

[sdake]

(And also implemented). I don't want to go overboard on the distribution channels. I have built brew packages before, it takes a few minutes. @MostAwesomeDude would you mind adding a PR after this merges, that updates the docs with Nix instructions?

You are free to use your time as you wish, although if you could time one of these, that would help me understand our overall distribution costs.

Thank you! -steve

[sdake]

To clarify, implementing brew packaging for this would take me about 30 minutes. I don't have the time to spend, and the dependencies would be difficult to deal with. I am seeking a signal on the initial cost of packaging workloads using Nix. I am super interested in the dependency cost. Flatpak seems hot too, although I don't have time to eval.

Thank you< -steve

[MostAwesomeDude]

I'm giving this a swing before lunch. I'm not using a speedrunning timer, just a wall clock.

• First I went to poetry2nix. Gotta figure out how to use it.
• Ehhh... nix flake init --template github:nix-community/poetry2nix
• Hmm, maybe... git add flake.nix to enable the flake, and then nix flake show
• error: getting status of '/nix/store/gapzxdimkhmgvwkyjblcf6c46qndsjps-source/poetry.lock': No such file or directory
• I should set projectDir to point to this interior directory? In flake.nix, I try projectDir = ./.;
• Now the flake shows a package: default: package 'python3.10-llama2pi-0.1.0'
• nix build and maybe it's now taco time?
• Yeah, taco time: [0/374 built, 4/38/341 copied (145.7/3239.9 MiB), 25.6/586.3 MiB DL]

So far I've been at the keyboard for just about 10min. And I'll return later.

[MostAwesomeDude]

Okay, more Nix time. This is the hard part.

• ModuleNotFoundError: No module named 'hatchling' while building einops
• Probably means that we need to override some stuff
• :taco:
• Okay, gonna try cribbing this example to inject a dependency on hatchling for einops
• nix build, :taco:, :phone:
• The override works! But now I need another one?
• Oh, hatchling is a special case; see einops upstream config. Reading another few override examples...
• :taco: :taco: Okay, enough of this. I know that einops is packaged; in fact, torch is packaged. So let's just slap together a nixpkgs environment for now.
• This gets me all of pykoi dependencies, but not at the versions that pykoi demands

Okay, so I've been here for an hour. This is a sort of good sign, I guess; it means that nobody else has tried to package pykoi yet. I need to take a heat break, and then I'll figure out the rest of the details.