🚀 Here's the PR! #16

See Sweep's progress at the progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: 680b5a19cf)

Actions (click)

[ ] ↻ Restart Sweep

Sandbox Execution ✓

Here are the sandbox execution logs prior to making any changes:

Sandbox logs for d40f4f8

Checking dspy/teleprompt/random_search.py for syntax errors... ✅ dspy/teleprompt/random_search.py has no syntax errors! 1/1 ✓
Checking dspy/teleprompt/random_search.py for syntax errors...
✅ dspy/teleprompt/random_search.py has no syntax errors!

Sandbox passed on the latest main, so sandbox checks will be enabled for this issue.

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/darinkishore/dspy/blob/6a14181ef651cc6b6282e902e2d45bbfe52545cf/dspy/primitives/box.py#L1-L155 https://github.com/darinkishore/dspy/blob/6a14181ef651cc6b6282e902e2d45bbfe52545cf/dspy/teleprompt/random_search.py#L28-L146 https://github.com/darinkishore/dspy/blob/6a14181ef651cc6b6282e902e2d45bbfe52545cf/dspy/teleprompt/ensemble.py#L6-L39 https://github.com/darinkishore/dspy/blob/6a14181ef651cc6b6282e902e2d45bbfe52545cf/docs/teleprompters.md#L1-L281

Step 2: ⌨️ Coding

[X] Modify dspy/teleprompt/random_search.py ✓ https://github.com/darinkishore/dspy/commit/930f948164b645f835119d20dd31d87957544c59 Edit
Modify dspy/teleprompt/random_search.py with contents:
• Convert the `BootstrapFewShotWithRandomSearch` class and its methods to asynchronous. This involves adding the `async` keyword to the method definitions and using `await` where necessary.
• For example, the `compile` method should be changed to `async def compile(self, student, *, teacher=None, trainset, valset=None, restrict=None):` and any calls to other methods or functions within it that support asynchronous operations should be awaited.
• Repeat this process for all other methods in the class that can be made asynchronous.

--- 
+++ 
@@ -48,7 +48,7 @@
         # print("Going to sample", self.max_num_traces, "traces in total.")
         print("Will attempt to train", self.num_candidate_sets, "candidate sets.")

-    def compile(self, student, *, teacher=None, trainset, valset=None, restrict=None):
+    async def compile(self, student, *, teacher=None, trainset, valset=None, restrict=None):
         self.trainset = trainset
         self.valset = valset or trainset  # TODO: FIXME: Note this choice.

@@ -70,14 +70,14 @@
             elif seed == -2:
                 # labels only
                 teleprompter = LabeledFewShot(k=self.max_labeled_demos)
-                program2 = teleprompter.compile(student, trainset=trainset2)
+                program2 = await teleprompter.compile(student, trainset=trainset2)

             elif seed == -1:
                 # unshuffled few-shot
                 program = BootstrapFewShot(metric=self.metric, max_bootstrapped_demos=self.max_num_samples,
                                            max_labeled_demos=self.max_labeled_demos,
                                            teacher_settings=self.teacher_settings, max_rounds=self.max_rounds)
-                program2 = program.compile(student, teacher=teacher, trainset=trainset2)
+                program2 = await program.compile(student, teacher=teacher, trainset=trainset2)

             else:
                 assert seed >= 0, seed
@@ -95,7 +95,7 @@
             evaluate = Evaluate(devset=self.valset, metric=self.metric, num_threads=self.num_threads,
                                 display_table=False, display_progress=True)

-            score, subscores = evaluate(program2, return_all_scores=True)
+            score, subscores = await evaluate(program2, return_all_scores=True)

             all_subscores.append(subscores)

[X] Running GitHub Actions for dspy/teleprompt/random_search.py ✓ Edit
Check dspy/teleprompt/random_search.py with contents:

Ran GitHub Actions for 930f948164b645f835119d20dd31d87957544c59:

[X] Modify dspy/teleprompt/ensemble.py ✓ https://github.com/darinkishore/dspy/commit/09314f6b96c42d4e7bb468927b797681a3376f0e Edit
Modify dspy/teleprompt/ensemble.py with contents:
• Convert the `Ensemble` class and its methods to asynchronous. This involves adding the `async` keyword to the method definitions and using `await` where necessary.
• For example, the `compile` method should be changed to `async def compile(self, programs):` and any calls to other methods or functions within it that support asynchronous operations should be awaited.
• Repeat this process for all other methods in the class that can be made asynchronous.

--- 
+++ 
@@ -18,11 +18,12 @@
         self.size = size
         self.deterministic = deterministic

-    def compile(self, programs):
+    async def compile(self, programs):
         size = self.size
         reduce_fn = self.reduce_fn

         import dspy
+        import asyncio
         class EnsembledProgram(dspy.Module):
             def __init__(self):
                 super().__init__()
@@ -30,7 +31,10 @@

             def forward(self, *args, **kwargs):
                 programs = random.sample(self.programs, size) if size else self.programs
-                outputs = [prog(*args, **kwargs) for prog in programs]
+                async def gather_outputs():
+                    return [await prog(*args, **kwargs) for prog in programs]
+
+                outputs = asyncio.run(gather_outputs())

                 if reduce_fn:
                     return reduce_fn(outputs)

[X] Running GitHub Actions for dspy/teleprompt/ensemble.py ✓ Edit
Check dspy/teleprompt/ensemble.py with contents:

Ran GitHub Actions for 09314f6b96c42d4e7bb468927b797681a3376f0e:

[X] Modify dspy/teleprompt/finetune.py ✓ https://github.com/darinkishore/dspy/commit/215893de9abe55fa6b5a441e81869be6ebdde238 Edit
Modify dspy/teleprompt/finetune.py with contents:
• Convert the `BootstrapFinetune` class and its methods to asynchronous. This involves adding the `async` keyword to the method definitions and using `await` where necessary.
• For example, the `compile` method should be changed to `async def compile(self, student, *, teacher=None, trainset, valset=None, target='t5-large', bsize=12, accumsteps=1, lr=5e-5, epochs=1, bf16=False):` and any calls to other methods or functions within it that support asynchronous operations should be awaited.
• Repeat this process for all other methods in the class that can be made asynchronous.

--- 
+++ 
@@ -58,7 +58,7 @@
                                              teacher_settings=teacher_settings)

-    def compile(self, student, *, teacher=None, trainset, valset=None,
+    async def compile(self, student, *, teacher=None, trainset, valset=None,
                 target='t5-large', bsize=12, accumsteps=1, lr=5e-5, epochs=1, bf16=False, int8=False, peft=False, path_prefix=None):

         # It's usually better to supply a few-shot teacher, rather than uncompiled module (the student).
@@ -72,7 +72,7 @@

         for teacher in teachers:
             # Dummy compilation to get bootstraps.
-            compiled = self.teleprompter.compile(student, teacher=teacher, trainset=trainset)
+            compiled = await self.teleprompter.compile(student, teacher=teacher, trainset=trainset)
             multitask = self.multitask

             # Prepare finetune  pairs.
@@ -141,7 +141,7 @@
             training_data_path = finetune_paths[name]
             compiler_config_ = dict(compiler_config)
             compiler_config_['save'] = compiler_config['save'] + '.' + name
-            best_ckpt_path = finetune_hf(training_data_path, target, compiler_config_)
+            best_ckpt_path = await finetune_hf(training_data_path, target, compiler_config_)

             print(f"#> Best checkpoint path: {best_ckpt_path} for {name}")
             finetune_models[name] = dsp.HFModel(model=target, checkpoint=best_ckpt_path) # best_ckpt_path

[X] Running GitHub Actions for dspy/teleprompt/finetune.py ✓ Edit
Check dspy/teleprompt/finetune.py with contents:

Ran GitHub Actions for 215893de9abe55fa6b5a441e81869be6ebdde238:

[X] Modify dspy/primitives/box.py ✓ https://github.com/darinkishore/dspy/commit/1068536d62eb0a2fd4d4789e8b60a6151e436d71 Edit
Modify dspy/primitives/box.py with contents:
• Ensure that the `Box` class and its methods are compatible with asynchronous operations. This may involve adding the `async` keyword to the method definitions and using `await` where necessary.
• For example, if the `__getitem__` method makes a call to another method or function that supports asynchronous operations, it should be changed to `async def __getitem__(self, index):` and the call should be awaited.
• Repeat this process for all other methods in the class that can be made asynchronous.

--- 
+++ 
@@ -99,7 +99,7 @@
         'radd', 'rsub', 'rmul', 'rtruediv', 'rfloordiv', 'rmod', 
         'rpow', 'rlshift', 'rrshift', 'rand', 'ror', 'rxor',
         # Sequence operations
-        'getitem', 'setitem', 'delitem', 'contains',
+        # 'getitem', 'setitem', 'delitem', 'contains', # Handled separately to enable async operations
         # Unary and other operations
         'neg', 'pos', 'abs', 'invert', 'round', 'len', 
         'getitem', 'setitem', 'delitem', 'contains', 'iter',
@@ -111,13 +111,14 @@

     def __init__(cls, name, bases, attrs):
         def create_method(op):
-            def method(self, other=None):
+            async def method(self, other=None):
                 if op in ['len', 'keys', 'values', 'items']:
-                    return getattr(self._value, op)()
+                    return await getattr(self._value, op)()
+                # 'getitem' and 'setitem' are special cases and must be handled outside of this loop
                 elif isinstance(other, Box):
-                    return Box(getattr(self._value, f'__{op}__')(other._value))
+                    return Box(await getattr(self._value, f'__{op}__')(other._value))
                 elif other is not None:
-                    return Box(getattr(self._value, f'__{op}__')(other))
+                    return Box(await getattr(self._value, f'__{op}__')(other))
                 else:
                     return NotImplemented
             return method
@@ -126,6 +127,13 @@
             setattr(cls, f'__{op}__', create_method(op))

         super().__init__(name, bases, attrs)
+
+    async def __getitem__(self, index):
+        return await self._value.__getitem__(index)
+
+    async def __setitem__(self, index, value):
+        await self._value.__setitem__(index, value)
+

 class Box(metaclass=BoxType):
     def __init__(self, value, source=False):
@@ -142,8 +150,8 @@
         return bool(self._value)

     # if method is missing just call it on the _value
-    def __getattr__(self, name):
-        return Box(getattr(self._value, name))
+    async def __getattr__(self, name):
+        return Box(await getattr(self._value, name))

     # # Unlike the others, this one collapses to a bool directly
     # def __eq__(self, other):

[X] Running GitHub Actions for dspy/primitives/box.py ✓ Edit
Check dspy/primitives/box.py with contents:

Ran GitHub Actions for 1068536d62eb0a2fd4d4789e8b60a6151e436d71:

[X] Modify docs/teleprompters.md ✓ https://github.com/darinkishore/dspy/commit/0fcdd68ee4d6123d2f3a39c0edd68cd53868b212 Edit
Modify docs/teleprompters.md with contents:
• Update the documentation to reflect the changes made to the `BootstrapFewShotWithRandomSearch`, `Ensemble`, and `BootstrapFinetune` classes and the `Box` class.
• This involves adding information about the asynchronous nature of these classes and their methods, and providing examples of how to use them in an asynchronous context.

--- 
+++ 
@@ -31,9 +31,9 @@

 ### Method

-#### `compile(self, student, *, trainset)`
-
-This method compiles the `LabeledFewShot` instance by configuring the `student` predictor. It assigns subsets of the `trainset` in each student's predictor's `demos` attribute. If the `trainset` is empty, the method returns the original `student`.
+#### `async compile(self, student, *, trainset)`
+
+This method asynchronously compiles the `LabeledFewShot` instance by configuring the `student` predictor. It assigns subsets of the `trainset` in each student's predictor's `demos` attribute. If the `trainset` is empty, the method returns the original `student`.

 **Parameters:**
 - `student` (_Teleprompter_): Student predictor to be compiled.
@@ -46,6 +46,7 @@

 ```python
 import dspy
+import asyncio

 #Assume defined trainset
 class RAG(dspy.Module):
@@ -64,6 +65,10 @@

 #Define teleprompter
 teleprompter = LabeledFewShot()
+
+# Compile!
+compiled_rag = asyncio.run(teleprompter.compile(student=RAG(), trainset=trainset))
+```

 # Compile!
 compiled_rag = teleprompter.compile(student=RAG(), trainset=trainset)

[X] Running GitHub Actions for docs/teleprompters.md ✓ Edit
Check docs/teleprompters.md with contents:

Ran GitHub Actions for 0fcdd68ee4d6123d2f3a39c0edd68cd53868b212:

Step 3: 🔁 Code Review

I have finished reviewing the code for completeness. I did not find errors for sweep/asynchronous_compiled_programs.

🎉 Latest improvements to Sweep:

We just released a dashboard to track Sweep's progress on your issue in real-time, showing every stage of the process – from search to planning and coding.
Sweep uses OpenAI's latest Assistant API to plan code changes and modify code! This is 3x faster and significantly more reliable as it allows Sweep to edit code and validate the changes in tight iterations, the same way as a human would.
Try using the GitHub issues extension to create Sweep issues directly from your editor! GitHub Issues and Pull Requests.

💡 To recreate the pull request edit the issue title or description. To tweak the pull request, leave a comment on the pull request. ^{Join Our Discord}

darinkishore / dspy

Sweep: Asynchronous Compiled Programs #7

Details