Open arunasank opened 6 days ago
cc @Rocketknight1
UPDATE: I got the code to work with batching. It's just very hard to gauge progress because tqdm does not report progress until the whole pipeline has finished the task. Even so, times are not reported, making it impossible to to sub-sample and determine time either.
TL;DR - 1, 3, and 4 are resolved and 2 still remains as an issue.
UPDATE: It doesn't work! It works for small datasets and smaller batch sizes, but fails silently with no errors on a large dataset with a reasonable batch size. For example, the script uses BATCH_SIZE =2 and a dataset with 4 inputs. This took 4 minutes to run with Llama-2-chat on an A100. When I considered the full dataset, which is ~7200 inputs with a batch size of 4, it was hanging on the for d in tqdm(pipe(KeyDataset(dataset, 'prompt'), batch_size=BATCH_SIZE)):
line silently, with no errors.
@arunasank have you tried using the map method ?
def fn(batch) :
batch["new_col"] = pipe(batch["prompt"])
return batch
dataset = dataset.map(fn, batched = True , batch_size = BATCH_SIZE )
responses = dataset["new_col"]
Yes, I did that with/without setting batch-size
on the pipeline method and it didn't work!
Hi @arunasank, have you tried other models and other methods of dataset creation to see if the issue recurs? It'd be helpful if you could figure out some minimal specific reproducer code for this issue, so that we could run it here and figure out what's going on!
System Info
transformers
version: 4.41.2Who can help?
@ArthurZucker @Narsil @stevhliu
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
for d in tqdm(pipe(KeyDataset(dataset, 'prompt'), batch_size=BATCH_SIZE)):
and doesn't progress. No errors are thrown.batch_size
is not provided in the above line, but tqdm only shows progress on completion.batch_size
with this linepipe = pipeline("text-generation", model="meta-llama/Llama-2-70b-chat-hf", device_map='auto', max_new_tokens=1024, batch_size=BATCH_SIZE, model_kwargs={"cache_dir": os.environ['TRANSFORMERS_CACHE'], "quantization_config": nf4_config})
but it doesn't work.Expected behavior
I would expect the code to work with batching. Clear documentation on batching when using pipelines would be appreciated.