Open darinkishore opened 8 months ago
680b5a19cf
)Here are the sandbox execution logs prior to making any changes:
d40f4f8
Checking dspy/teleprompt/random_search.py for syntax errors... ✅ dspy/teleprompt/random_search.py has no syntax errors!
1/1 ✓Checking dspy/teleprompt/random_search.py for syntax errors... ✅ dspy/teleprompt/random_search.py has no syntax errors!
Sandbox passed on the latest main
, so sandbox checks will be enabled for this issue.
I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.
dspy/teleprompt/random_search.py
✓ https://github.com/darinkishore/dspy/commit/930f948164b645f835119d20dd31d87957544c59 Edit
Modify dspy/teleprompt/random_search.py with contents:
• Convert the `BootstrapFewShotWithRandomSearch` class and its methods to asynchronous. This involves adding the `async` keyword to the method definitions and using `await` where necessary.
• For example, the `compile` method should be changed to `async def compile(self, student, *, teacher=None, trainset, valset=None, restrict=None):` and any calls to other methods or functions within it that support asynchronous operations should be awaited.
• Repeat this process for all other methods in the class that can be made asynchronous.
--- +++ @@ -48,7 +48,7 @@ # print("Going to sample", self.max_num_traces, "traces in total.") print("Will attempt to train", self.num_candidate_sets, "candidate sets.") - def compile(self, student, *, teacher=None, trainset, valset=None, restrict=None): + async def compile(self, student, *, teacher=None, trainset, valset=None, restrict=None): self.trainset = trainset self.valset = valset or trainset # TODO: FIXME: Note this choice. @@ -70,14 +70,14 @@ elif seed == -2: # labels only teleprompter = LabeledFewShot(k=self.max_labeled_demos) - program2 = teleprompter.compile(student, trainset=trainset2) + program2 = await teleprompter.compile(student, trainset=trainset2) elif seed == -1: # unshuffled few-shot program = BootstrapFewShot(metric=self.metric, max_bootstrapped_demos=self.max_num_samples, max_labeled_demos=self.max_labeled_demos, teacher_settings=self.teacher_settings, max_rounds=self.max_rounds) - program2 = program.compile(student, teacher=teacher, trainset=trainset2) + program2 = await program.compile(student, teacher=teacher, trainset=trainset2) else: assert seed >= 0, seed @@ -95,7 +95,7 @@ evaluate = Evaluate(devset=self.valset, metric=self.metric, num_threads=self.num_threads, display_table=False, display_progress=True) - score, subscores = evaluate(program2, return_all_scores=True) + score, subscores = await evaluate(program2, return_all_scores=True) all_subscores.append(subscores)
dspy/teleprompt/random_search.py
✓ Edit
Check dspy/teleprompt/random_search.py with contents:
Ran GitHub Actions for 930f948164b645f835119d20dd31d87957544c59:
dspy/teleprompt/ensemble.py
✓ https://github.com/darinkishore/dspy/commit/09314f6b96c42d4e7bb468927b797681a3376f0e Edit
Modify dspy/teleprompt/ensemble.py with contents:
• Convert the `Ensemble` class and its methods to asynchronous. This involves adding the `async` keyword to the method definitions and using `await` where necessary.
• For example, the `compile` method should be changed to `async def compile(self, programs):` and any calls to other methods or functions within it that support asynchronous operations should be awaited.
• Repeat this process for all other methods in the class that can be made asynchronous.
--- +++ @@ -18,11 +18,12 @@ self.size = size self.deterministic = deterministic - def compile(self, programs): + async def compile(self, programs): size = self.size reduce_fn = self.reduce_fn import dspy + import asyncio class EnsembledProgram(dspy.Module): def __init__(self): super().__init__() @@ -30,7 +31,10 @@ def forward(self, *args, **kwargs): programs = random.sample(self.programs, size) if size else self.programs - outputs = [prog(*args, **kwargs) for prog in programs] + async def gather_outputs(): + return [await prog(*args, **kwargs) for prog in programs] + + outputs = asyncio.run(gather_outputs()) if reduce_fn: return reduce_fn(outputs)
dspy/teleprompt/ensemble.py
✓ Edit
Check dspy/teleprompt/ensemble.py with contents:
Ran GitHub Actions for 09314f6b96c42d4e7bb468927b797681a3376f0e:
dspy/teleprompt/finetune.py
✓ https://github.com/darinkishore/dspy/commit/215893de9abe55fa6b5a441e81869be6ebdde238 Edit
Modify dspy/teleprompt/finetune.py with contents:
• Convert the `BootstrapFinetune` class and its methods to asynchronous. This involves adding the `async` keyword to the method definitions and using `await` where necessary.
• For example, the `compile` method should be changed to `async def compile(self, student, *, teacher=None, trainset, valset=None, target='t5-large', bsize=12, accumsteps=1, lr=5e-5, epochs=1, bf16=False):` and any calls to other methods or functions within it that support asynchronous operations should be awaited.
• Repeat this process for all other methods in the class that can be made asynchronous.
--- +++ @@ -58,7 +58,7 @@ teacher_settings=teacher_settings) - def compile(self, student, *, teacher=None, trainset, valset=None, + async def compile(self, student, *, teacher=None, trainset, valset=None, target='t5-large', bsize=12, accumsteps=1, lr=5e-5, epochs=1, bf16=False, int8=False, peft=False, path_prefix=None): # It's usually better to supply a few-shot teacher, rather than uncompiled module (the student). @@ -72,7 +72,7 @@ for teacher in teachers: # Dummy compilation to get bootstraps. - compiled = self.teleprompter.compile(student, teacher=teacher, trainset=trainset) + compiled = await self.teleprompter.compile(student, teacher=teacher, trainset=trainset) multitask = self.multitask # Prepare finetunepairs. @@ -141,7 +141,7 @@ training_data_path = finetune_paths[name] compiler_config_ = dict(compiler_config) compiler_config_['save'] = compiler_config['save'] + '.' + name - best_ckpt_path = finetune_hf(training_data_path, target, compiler_config_) + best_ckpt_path = await finetune_hf(training_data_path, target, compiler_config_) print(f"#> Best checkpoint path: {best_ckpt_path} for {name}") finetune_models[name] = dsp.HFModel(model=target, checkpoint=best_ckpt_path) # best_ckpt_path
dspy/teleprompt/finetune.py
✓ Edit
Check dspy/teleprompt/finetune.py with contents:
Ran GitHub Actions for 215893de9abe55fa6b5a441e81869be6ebdde238:
dspy/primitives/box.py
✓ https://github.com/darinkishore/dspy/commit/1068536d62eb0a2fd4d4789e8b60a6151e436d71 Edit
Modify dspy/primitives/box.py with contents:
• Ensure that the `Box` class and its methods are compatible with asynchronous operations. This may involve adding the `async` keyword to the method definitions and using `await` where necessary.
• For example, if the `__getitem__` method makes a call to another method or function that supports asynchronous operations, it should be changed to `async def __getitem__(self, index):` and the call should be awaited.
• Repeat this process for all other methods in the class that can be made asynchronous.
--- +++ @@ -99,7 +99,7 @@ 'radd', 'rsub', 'rmul', 'rtruediv', 'rfloordiv', 'rmod', 'rpow', 'rlshift', 'rrshift', 'rand', 'ror', 'rxor', # Sequence operations - 'getitem', 'setitem', 'delitem', 'contains', + # 'getitem', 'setitem', 'delitem', 'contains', # Handled separately to enable async operations # Unary and other operations 'neg', 'pos', 'abs', 'invert', 'round', 'len', 'getitem', 'setitem', 'delitem', 'contains', 'iter', @@ -111,13 +111,14 @@ def __init__(cls, name, bases, attrs): def create_method(op): - def method(self, other=None): + async def method(self, other=None): if op in ['len', 'keys', 'values', 'items']: - return getattr(self._value, op)() + return await getattr(self._value, op)() + # 'getitem' and 'setitem' are special cases and must be handled outside of this loop elif isinstance(other, Box): - return Box(getattr(self._value, f'__{op}__')(other._value)) + return Box(await getattr(self._value, f'__{op}__')(other._value)) elif other is not None: - return Box(getattr(self._value, f'__{op}__')(other)) + return Box(await getattr(self._value, f'__{op}__')(other)) else: return NotImplemented return method @@ -126,6 +127,13 @@ setattr(cls, f'__{op}__', create_method(op)) super().__init__(name, bases, attrs) + + async def __getitem__(self, index): + return await self._value.__getitem__(index) + + async def __setitem__(self, index, value): + await self._value.__setitem__(index, value) + class Box(metaclass=BoxType): def __init__(self, value, source=False): @@ -142,8 +150,8 @@ return bool(self._value) # if method is missing just call it on the _value - def __getattr__(self, name): - return Box(getattr(self._value, name)) + async def __getattr__(self, name): + return Box(await getattr(self._value, name)) # # Unlike the others, this one collapses to a bool directly # def __eq__(self, other):
dspy/primitives/box.py
✓ Edit
Check dspy/primitives/box.py with contents:
Ran GitHub Actions for 1068536d62eb0a2fd4d4789e8b60a6151e436d71:
docs/teleprompters.md
✓ https://github.com/darinkishore/dspy/commit/0fcdd68ee4d6123d2f3a39c0edd68cd53868b212 Edit
Modify docs/teleprompters.md with contents:
• Update the documentation to reflect the changes made to the `BootstrapFewShotWithRandomSearch`, `Ensemble`, and `BootstrapFinetune` classes and the `Box` class.
• This involves adding information about the asynchronous nature of these classes and their methods, and providing examples of how to use them in an asynchronous context.
--- +++ @@ -31,9 +31,9 @@ ### Method -#### `compile(self, student, *, trainset)` - -This method compiles the `LabeledFewShot` instance by configuring the `student` predictor. It assigns subsets of the `trainset` in each student's predictor's `demos` attribute. If the `trainset` is empty, the method returns the original `student`. +#### `async compile(self, student, *, trainset)` + +This method asynchronously compiles the `LabeledFewShot` instance by configuring the `student` predictor. It assigns subsets of the `trainset` in each student's predictor's `demos` attribute. If the `trainset` is empty, the method returns the original `student`. **Parameters:** - `student` (_Teleprompter_): Student predictor to be compiled. @@ -46,6 +46,7 @@ ```python import dspy +import asyncio #Assume defined trainset class RAG(dspy.Module): @@ -64,6 +65,10 @@ #Define teleprompter teleprompter = LabeledFewShot() + +# Compile! +compiled_rag = asyncio.run(teleprompter.compile(student=RAG(), trainset=trainset)) +``` # Compile! compiled_rag = teleprompter.compile(student=RAG(), trainset=trainset)
docs/teleprompters.md
✓ Edit
Check docs/teleprompters.md with contents:
Ran GitHub Actions for 0fcdd68ee4d6123d2f3a39c0edd68cd53868b212:
I have finished reviewing the code for completeness. I did not find errors for sweep/asynchronous_compiled_programs
.
💡 To recreate the pull request edit the issue title or description. To tweak the pull request, leave a comment on the pull request. Join Our Discord
Details
Currently the compiled programs are not async and hence are not efficient to serve using a python server. It would be useful to merge the PRs aiming to add async across the dspy library.
This could also involve adding nurseries in order to await an ensemble of requests simultaneously.
Checklist
- [X] Modify `dspy/teleprompt/random_search.py` ✓ https://github.com/darinkishore/dspy/commit/930f948164b645f835119d20dd31d87957544c59 [Edit](https://github.com/darinkishore/dspy/edit/sweep/asynchronous_compiled_programs/dspy/teleprompt/random_search.py#L30-L146) - [X] Running GitHub Actions for `dspy/teleprompt/random_search.py` ✓ [Edit](https://github.com/darinkishore/dspy/edit/sweep/asynchronous_compiled_programs/dspy/teleprompt/random_search.py#L30-L146) - [X] Modify `dspy/teleprompt/ensemble.py` ✓ https://github.com/darinkishore/dspy/commit/09314f6b96c42d4e7bb468927b797681a3376f0e [Edit](https://github.com/darinkishore/dspy/edit/sweep/asynchronous_compiled_programs/dspy/teleprompt/ensemble.py#L11-L39) - [X] Running GitHub Actions for `dspy/teleprompt/ensemble.py` ✓ [Edit](https://github.com/darinkishore/dspy/edit/sweep/asynchronous_compiled_programs/dspy/teleprompt/ensemble.py#L11-L39) - [X] Modify `dspy/teleprompt/finetune.py` ✓ https://github.com/darinkishore/dspy/commit/215893de9abe55fa6b5a441e81869be6ebdde238 [Edit](https://github.com/darinkishore/dspy/edit/sweep/asynchronous_compiled_programs/dspy/teleprompt/finetune.py#L233-L269) - [X] Running GitHub Actions for `dspy/teleprompt/finetune.py` ✓ [Edit](https://github.com/darinkishore/dspy/edit/sweep/asynchronous_compiled_programs/dspy/teleprompt/finetune.py#L233-L269) - [X] Modify `dspy/primitives/box.py` ✓ https://github.com/darinkishore/dspy/commit/1068536d62eb0a2fd4d4789e8b60a6151e436d71 [Edit](https://github.com/darinkishore/dspy/edit/sweep/asynchronous_compiled_programs/dspy/primitives/box.py#L91-L154) - [X] Running GitHub Actions for `dspy/primitives/box.py` ✓ [Edit](https://github.com/darinkishore/dspy/edit/sweep/asynchronous_compiled_programs/dspy/primitives/box.py#L91-L154) - [X] Modify `docs/teleprompters.md` ✓ https://github.com/darinkishore/dspy/commit/0fcdd68ee4d6123d2f3a39c0edd68cd53868b212 [Edit](https://github.com/darinkishore/dspy/edit/sweep/asynchronous_compiled_programs/docs/teleprompters.md) - [X] Running GitHub Actions for `docs/teleprompters.md` ✓ [Edit](https://github.com/darinkishore/dspy/edit/sweep/asynchronous_compiled_programs/docs/teleprompters.md)