assistants API file upload

[phact]

Added a new cli command process-repo-files which clones the repo, switches to the right commit hash, and iterates [sequentially for now] over all the files, uploading them to assistants-api. This gets us embeddings and RAG for free:

python -m swe_bench_util process-repo-files

I clone to /tmp for now, not sure if that's what we want.

In the end it dumps a list of file_ids that you can give your assistant for searching.

If this is interesting I'm happy to help optimize how assistants chunks and embeds for our purposes.

[raymyers]

Thanks @phact! Looks like a minor rebase needed. At first glance I think this is pretty good start, just except a couple notes:

• It looks like this would clone a repo once per SHA, since we're going to loop through them that will take time and space. We only need one clone per repo and we can checkout (or do worktrees if we want to get fancy later) the different SHAs from that clone
• I would prefer to checkout into a gitignored subdir of the repo like "checkouts" instead of tmp.

[raymyers]

I'm getting this for most files, maybe will need to do something to make OpenAI understand these are text files

Dockerfile: Error code: 501 - {'message': 'Unsupported file type'}

[phact]

I was thinking about what file types we should add. Maybe the play is to use a block list like sweep does:

https://github.com/sweepai/sweep/blob/bdcd1195bc8c2a90aa277a9169dcb13eed702868/docs/pages/blogs/generating-50k-embeddings-with-gte.mdx#L22-L26

[raymyers]

Merging, going to structure a bit and make my notes a new issue