Open neubig opened 4 months ago
@neubig I'm interested in this, can take a look at this, this weekend.
@neubig Ok, so I think the easiest approach would be to implement this with huggingface's assistant model feature.
I found some references to this with whisper, and another hf blog here.
So would a valid implementation be to use a hf model, with an assistant model, and then update the do_completion
call here, so that it can use the assistant. And then benchmark against using without an assistant?
One downside of using hf is it does not look there's a way to make it compatible with non-hf models like gpt-4, etc. I looked to see if there was a way to implement a hf model wrapper, but didn't see a solution.
Hmm, I'm a bit confused. I think the main difference that we would need to have is that we would need to generate edits in the diff format like that cited on the blog:
@@ ... @@
function binarySearch(arr, x) {
- let low = 0, high = arr.length - 1;
- while (low <= high) {
- let mid = Math.floor((low + high) / 2);
- if (arr[mid] === x) {
- return mid;
- }
- low += 1;
- }
- return -1;
+ let low = 0, high = arr.length - 1;
+ while (low <= high) {
+ let mid = Math.floor((low + high) / 2);
+ if (arr[mid] === x) {
+ return mid;
+ } else if (arr[mid] < x) {
+ low = mid + 1;
+ } else {
+ high = mid - 1;
+ }
+ }
+ return -1;
}
I'm not sure how this has something to do with the assistant feature on hugging face. Could you elaborate more?
Ok, that helps. I was looking at the implementation portion, where they talk about speculative decoding in the blog post.
"We achieve speeds of >1000 tokens/s (just under 4000 char/s) on our 70b model using a speculative-decoding variant tailored for code-edits, called speculative edits."
"Our biggest win comes from our custom speculative decoding algorithm called “speculative edits”. It is equivalent to a full-file-rewrite, while being up to 9x faster."
And the assistant model is how speculative decoding works with huggingface.
Ok, so now I get that you are talking about generating edits in a diff format. I'll look into this.
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
Can I take a look at this if no one else is? I do not have much of an experience with agents, but would like to learn. Also happy to collaborate with others on this.
Hi @anushkumarv , you're welcome to take a look at this! A good place to start would be to look at our "edit file" functionality.
Got some unexpected work for a couple of weeks. I should be able to come back to this after that. Happy if someone else wants to look at this in the meantime.
Anyone have advice for ramping up on current file editing capabilities in open devin?
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
What problem or use case are you trying to solve?
File editing is not perfect with our current method using SWE-Agent style actions.
Do you have thoughts on the technical implementation?
Cursor has recently published work on speculative editing, that we could possibly incorporate into our agent workflows: https://cursor.sh/blog/instant-apply
We would want to implement a prototype and benchmark it against SWE-Bench.
This would be a good issue for new contributors who have significant experience with LLMs and agents!