Ideas for what to do next - Githubissues

gpu-mode / discord-cluster-manager

Hook up GPUs to your Discord channel and start running jobs via DMs!

4 stars 2 forks source link

Ideas for what to do next #6

Open msaroufim opened 2 weeks ago

msaroufim commented 2 weeks ago

Discord based leaderboard

UX

[ ] How does the leaderboard get rendered @AndreSlavescu
[x] Slash commands that autofill that a script is expected and maybe more information like the kernel name or gpu type will be needed @msaroufim
[x] Do not create an extra message, just thread user reply @msaroufim
[x] Don't render rich links
[ ] Give clearer feedback on time job took and wait time so far

Leaderboard infra

[ ] Where do we store all the run data, do we use some DB like postgres, where do we host it
[ ] Do we run X times and warm up the machines so that the benchmarks are good
[ ] Let people run code from the leaderboard

GPU infra

[x] How do we plug in new gpu - doc only change
[x] How do we assign GPU in a round robin way so that a single job won't occupy 8 GPUs @msaroufim - not needed
[x] How do we detect malicious jobs and kill them, baseline is probably how long the job takes to run

Startup times

The faster the startup time the more interactive the bot becomes and more popular

[x] NVIDIA container setup @msaroufim @liveaverage
[x] AMD containers setup @msaroufim @saienduri
[x] pytorch takes too long to install - wont fix just use modal instead @msaroufim
[x] Killing long running jobs so they don't bork up the queue
[x] Setup modal scheduler @msaroufim - main gap right now is passing in configs to remotee machine

Testing infra @S1ro1

Right now this is omega jank

[x] We have a staging env but its slow to test - improve the local development experience @b9r5
[x] Merging PRs feels very yolo still, I'm never sure something works until I test it so some CI sanity tests would be nice @b9r5
[x] Maybe we don't "test" but we fix fast because we work in different timezones
[x] Modularize code with Discord cogs so its easier to maintain and isolate breakages @S1ro1

What do people upload

[x] numpy script
[x] torch script
[x] triton script @alexzhang13
[ ] cuda script - this is the trickiest but we need to on our end have most of the boilerplate including things like launch params. For advanced users we can get launch params from slash @alexzhang13
[x] Basic cuda support @msaroufim - still jank and it breaks with things like \n

Profiling/Ranking

[ ] Run ncu on user scripts - blocked on getting permissions to run ncu @msaroufim - this is what the PR would look like if we did have permissions https://github.com/gpu-mode/discord-cluster-manager/commit/4da129b064bc80a5fde1f2f422e863f85311f066
[ ] Run some simple profiling of scripts