search

bespokelabsai / curator

Synthetic Data curation for post-training and structured data extraction
https://docs.bespokelabs.ai/bespoke-curator
Apache License 2.0
23 stars 2 forks source link

Failing to recover when requests.jsonl exists but responses.jsonl and arrow file don't #158

Open vutrung96 opened 3 days ago

vutrung96 commented 3 days ago

Ran into this error on my DCFT run

At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: [36mray::completions__airoboros_stage_3_general_instructions_gpt-4o-mini::complete_none_responses()[39m (pid=608242, ip=10.120.0.3, actor_id=1645bd492f747e09a32a322f82000000, repr=<engine.operators.completions_operator._Completions object at 0x79bf897e5ed0>)
File "/tmp/ray/session_2024-11-14_19-11-03_110436_1/runtime_resources/working_dir_files/_ray_pkg_e529f74676b90b6f/engine/operators/completions_operator.py", line 288, in completions
 dataset = completion(dataset, working_dir=curator_cache_dir)
File "/tmp/ray/session_2024-11-14_19-11-03_110436_1/runtime_resources/pip/71dcb4dc7a71bcfe09f323529818e20eee2981bc/virtualenv/lib/python3.10/site-packages/bespokelabs/curator/prompter/prompter.py", line 128, in __call__
 return self._completions(self._request_processor, dataset, working_dir)
File "/tmp/ray/session_2024-11-14_19-11-03_110436_1/runtime_resources/pip/71dcb4dc7a71bcfe09f323529818e20eee2981bc/virtualenv/lib/python3.10/site-packages/bespokelabs/curator/prompter/prompter.py", line 221, in _completions
 dataset = request_processor.run(
File "/tmp/ray/session_2024-11-14_19-11-03_110436_1/runtime_resources/pip/71dcb4dc7a71bcfe09f323529818e20eee2981bc/virtualenv/lib/python3.10/site-packages/bespokelabs/curator/request_processor/openai_online_request_processor.py", line 187, in run
 dataset = self.create_dataset_files(
File "/tmp/ray/session_2024-11-14_19-11-03_110436_1/runtime_resources/pip/71dcb4dc7a71bcfe09f323529818e20eee2981bc/virtualenv/lib/python3.10/site-packages/bespokelabs/curator/request_processor/base_request_processor.py", line 220, in create_dataset_files
 raise ValueError(f"No responses files found in {working_dir}")
ValueError: No responses files found in /home/ray/.cache/curator/airoboros_stage_3_general_instructions_gpt-4o-mini__complete_none_responses/e302c4e0d278dc3e

The directory actually consists of a single requests_0.jsonl file:

Screenshot 2024-11-20 at 1 57 00 PM

It seems like we should be able to recover from this by just starting with requests_0.jsonl?

vutrung96 commented 3 days ago

Hmm I did not notice this but seems like requests_0.jsonl is empty and that's probably the reason why.