Open ibayer opened 1 month ago
Thank for this great library!
I'm currently exploring if peft allows me to use lora for semantic segmentation.
The example nb states:
The results are definitely not as expected and as mentioned above, this example is not meant to provide a state-of-the-art model. It exists to familiarize you with the end-to-end workflow.
and indeed the provided settings provide visual and metric results that are so far off from expected benchmark results that's hard to judge if the model is doing some meaningful learning.
As a first step to improve the results I'm following the suggestions from the nb
Here are some things that you can try to get better results:
- Increase the number of training samples.
- Try a larger SegFormer model variant (know about the available model variants here).
- Try different values for the arguments available in
LoraConfig
.- Tune the learning rate and batch size.
My previous comment describes the first problem I ran into when increasing the number of training examples.
Any suggestion on how to make the helper function compute_metrics()
more memory efficient or reduce the memory consumption in some other way is very much appreciate.
Hi Immanuel, thanks for reporting the issue.
So I think the issue is not strictly with PEFT, but instead that the custom compute_metris
function defined within the notebook does not perform any batching, as you correctly identified. Therefore, the eval dataset will be run at once, which can easily result in an OOM error.
I don't know much about the evaluate
package, but I think this updated version should work without OOM:
def compute_metrics(eval_pred, batch_size=16):
logits, labels = eval_pred
total_metrics = {}
with torch.no_grad():
for i in range(0, len(logits), batch_size):
logits_tensor = torch.from_numpy(logits[i:i+batch_size])
# scale the logits to the size of the label
logits_tensor = nn.functional.interpolate(
logits_tensor,
size=labels.shape[-2:],
mode="bilinear",
align_corners=False,
).argmax(dim=1)
pred_labels = logits_tensor.detach().cpu().numpy()
# currently using _compute instead of compute
# see this issue for more info:
metrics = metric._compute(
predictions=pred_labels,
references=labels,
num_labels=len(id2label),
ignore_index=0,
reduce_labels=image_processor.do_reduce_labels,
)
# add per category metrics as individual key-value pairs
per_category_accuracy = metrics.pop("per_category_accuracy").tolist()
per_category_iou = metrics.pop("per_category_iou").tolist()
metrics.update({f"accuracy_{id2label[i]}": v for i, v in enumerate(per_category_accuracy)})
metrics.update({f"iou_{id2label[i]}": v for i, v in enumerate(per_category_iou)})
for k, v in metrics.items():
total_metrics[k] = total_metrics.get(k, 0) + v
# calculation is not quite correct, as the last batch could be smaller, np.average would be better
total_metrics = {k: v / len(logits) for k, v in total_metrics.items()}
return total_metrics
Note, I also had to disable jitter
What problem did you encounter?
and restrict to single GPU for get the example nb to work with the default configuration.
What was the error, did it happen to be
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR
What was the error, did it happen to be
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR
No, it was a bit more obscure (see below), but I have only 2 GPUs so I'm losing less then 2x. :)
File ~/micromamba/envs/arcd2-libcheck-peft/lib/python3.11/site-packages/accelerate/state.py:292, in PartialState.__init__(self, cpu, **kwargs)
290 if self.device.type == "cuda" and not check_cuda_p2p_ib_support():
291 if "NCCL_P2P_DISABLE" not in os.environ or "NCCL_IB_DISABLE" not in os.environ:
--> 292 raise NotImplementedError(
293 "Using RTX 4000 series doesn't support faster communication broadband via P2P or IB. "
294 'Please set `NCCL_P2P_DISABLE="1"` and `NCCL_IB_DISABLE="1" or use `accelerate launch` which '
295 "will do this automatically."
296 )
297 # Important: This should be the *only* code outside of `self.initialized!`
298 self.fork_launched = parse_flag_from_env("FORK_LAUNCHED", 0)
NotImplementedError: Using RTX 4000 series doesn't support faster communication broadband via P2P or IB. Please set `NCCL_P2P_DISABLE="1"` and `NCCL_IB_DISABLE="1" or use `accelerate launch` which will do this automatically.
edit: fix error log for last case flags + 2 GPUs (flags need to be set before first import to take effect...)
I don't know much about the evaluate package, but I think this updated version should work without OOM:
Confirmed, this fixed OOM. Thanks!
Note, I also had to disable jitter
What problem did you encounter?
Can't reproduce it anymore, I might have been unlucky with my previous torchvision version. I suspected a missing range clipping, anyway it's gone and hopefully it won't come back. :)
for full dataset
from datasets import load_dataset
# ds = load_dataset("scene_parse_150", split="train[:150]")
ds = load_dataset("scene_parse_150", split="train")
eval still seems to eat to much memory, but GPU this time
OutOfMemoryError: CUDA out of memory. Tried to allocate 8.08 GiB. GPU
NotImplementedError: Using RTX 4000 series doesn't support faster communication broadband via P2P or IB. Please set `NCCL_P2P_DISABLE="1"` and `NCCL_IB_DISABLE="1" or use `accelerate launch` which will do this automatically.
Ah yes, this is an unfortunate annoyance with the 4000 series, see here: https://github.com/huggingface/accelerate/pull/2195.
eval still seems to eat to much memory, but GPU this time
Okay, so it seems that Trainer
still collects all predictions in a giant tensor, even after changing compute_metrics
to use batching. There is a "correct" way of applying batching though. For this, you need to pass batch_eval_metrics=True
to the TrainingArguments
. We also need to rewrite compute_metrics
:
class MyMetrics:
def __init__(self):
self.total_metrics = {}
def __call__(self, eval_pred, compute_result=False):
logits_tensor, labels = eval_pred
with torch.no_grad():
# scale the logits to the size of the label
logits_tensor = nn.functional.interpolate(
logits_tensor,
size=labels.shape[-2:],
mode="bilinear",
align_corners=False,
).argmax(dim=1)
pred_labels = logits_tensor.detach().cpu().numpy()
# currently using _compute instead of compute
# see this issue for more info:
metrics = metric._compute(
predictions=pred_labels,
references=labels.detach().cpu().numpy(),
num_labels=len(id2label),
ignore_index=0,
reduce_labels=image_processor.do_reduce_labels,
)
# add per category metrics as individual key-value pairs
per_category_accuracy = metrics.pop("per_category_accuracy").tolist()
per_category_iou = metrics.pop("per_category_iou").tolist()
metrics.update({f"accuracy_{id2label[i]}": v for i, v in enumerate(per_category_accuracy)})
metrics.update({f"iou_{id2label[i]}": v for i, v in enumerate(per_category_iou)})
for k, v in metrics.items():
self.total_metrics[k] = self.total_metrics.get(k, 0) + v
result = -1
if compute_result:
result = {k: v / len(logits_tensor) for k, v in self.total_metrics.items()}
self.total_metrics.clear()
return result
which we then pass to Trainer
as compute_metrics=MyMetrics()
. For me, this resolves the memory issue. (IMHO, skorch handles this in a more user friendly way)
Thanks, I can confirm that the OOM error is gone. Do you get all nan values as well? Also, I guess mean Iou is way to high and should increase not decrease.
Ah yes, this is an unfortunate annoyance with the 4000 series, see here: https://github.com/huggingface/accelerate/pull/2195.
On RTX 3090+, infinity-band and peer-to-peer communication was removed (so the entirety of the 4000 series and beyond).
I'm not sure what the expected behavior is. Does this only mean 4000 series can't use optimal communication or that they can't communicate efficient at all and therefore parallel processing is completely disabled?
I figured out why the scores are weird. The issue is that we currently calculate the scores on a super small batch size, i.e. most labels are not present for each batch, resulting in nans. The correct way to do the evaluation without hogging memory is probably to pull the data to CPU and store it there, then at the end concatenate it and calculate the loss on all eval samples. I updated the code to do this and new the scores are much more reasonable:
class MyMetrics:
def __init__(self):
#self.total_metrics = {}
self.preds = []
self.labels = []
def __call__(self, eval_pred, compute_result=False):
logits_tensor, labels = eval_pred
logits_tensor = nn.functional.interpolate(
logits_tensor,
size=labels.shape[-2:],
mode="bilinear",
align_corners=False,
).argmax(dim=1)
self.preds.append(logits_tensor.cpu().detach().numpy())
self.labels.append(labels.cpu().detach().numpy())
if not compute_result:
return
preds = np.concatenate(self.preds)
labels = np.concatenate(self.labels)
metrics = metric._compute(
predictions=preds,
references=labels,
num_labels=len(id2label),
ignore_index=0,
reduce_labels=image_processor.do_reduce_labels,
)
# add per category metrics as individual key-value pairs
per_category_accuracy = metrics.pop("per_category_accuracy").tolist()
per_category_iou = metrics.pop("per_category_iou").tolist()
metrics.update({f"accuracy_{id2label[i]}": v for i, v in enumerate(per_category_accuracy)})
metrics.update({f"iou_{id2label[i]}": v for i, v in enumerate(per_category_iou)})
return metrics
Overall, the scores still look quite weak, but I'm no expert in this task at all. Probably some hyper-parameter tuning could yield better results. I also tested full fine-tuning and it didn't do any better, so it's probably not a PEFT error.
One thing to note is that the chosen model is very small, so LoRA might not be the right approach here. For instance, for r=32
, 13% of parameters are trainable. Normally, we see <1%. Perhaps you could try a more powerful model.
Does this only mean 4000 series can't use optimal communication or that they can't communicate efficient at all and therefore parallel processing is completely disabled?
I don't really know how big of a difference this makes. You could try out this patched driver which enables P2P for 4090s:
I did some further testing with this notebook. One mistake I found is that the ignore_index
should have been 255 and not 0, according to this doc. But even after changing that, the calculated scores were still super small.
Next I did something very basic, namely calculating accuracy the brute force way: raw_accuracy = (preds[labels!=ignore_index]==labels[labels!=ignore_index]).mean()
. When using fully pre-trained models, this gives values of 0.8+, so much closer to expectations. I didn't dig deeper into why the evaluate
metrics are so low, I suspect that it does something where the accuracy/IOU for each class is calculated separately and then averaged, but the class level scores are totally off because the "true negatives" (which is the vast majority) are all counted as incorrect. But that's just a guess.
I also found the origin of this notebook, which appears to be: https://huggingface.co/docs/transformers/v4.33.0/en/tasks/semantic_segmentation. Maybe this other notebook actually works better, but I haven't tested it: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SegFormer/Fine_tune_SegFormer_on_custom_dataset.ipynb. In that notebook, they use metric.add_batch
, which is probably the correct way to avoid memory explosion.
Anyway, I attached the notebook that I used for these experiments (still only uses a subset of data), just rename the file ending to .ipynb
. There, we can see that nvidia/mit-b0
can actually be trained with LoRA to improve its accuracy to be better than random (though the loss is gigantic!). I also tested some fully pre-trained models, where there is no improvement really. But that's not surprising, as these models are already pre-trained on the same dataset.
I didn't dig deeper into why the evaluate metrics are so low, I suspect that it does something where the accuracy/IOU for each class is calculated separately and then averaged, [...]
Another guess would be that the score is adjusted for the total number of classes without considering that some don't show up in the validation set. Anyway, I need to take a closer look at how the metric should be calculated.
There, we can see that nvidia/mit-b0 can actually be trained with LoRA to improve its accuracy to be better than random (though the loss is gigantic!).
For the fine-tuned model, it looks like mean IOU ~ 100 * vali_loss
, but the difference for the LoRA model is much larger. Sure, we can't expect the relationship to be linear, but this does look suspicious.
I took a quick peek so far and will take a closer look in the coming days.
Thanks, for all the detailed investigations!
I spent some more time with the tutorial but currently have the impression that I need a setup where I get clearer metric feedback before I can start to optimize hyper-parameters for better feedback.
I think this issue could be closed since we only uncovered issues with the example and not with the PEFT library itself.
Thanks for all the help!
I agree, this requires either digging deeper into how evaluate
calculates the metrics or just rolling a custom metric (or maybe use sklearn?).
We can keep this issue open until stale bot closes it, as this gets more visibility and maybe someone comes along and gives some good insights.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
System Info
Who can help?
@BenjaminBossan
Information
Tasks
examples
folderReproduction
Modestly increase the number of sample in the official semantic_segmentation_peft_lora.ipynb example.
full log:
Note, I also had to disable
jitter
and restrict to single GPU for get the example nb to work with the default configuration.
Expected behavior
Eval memory requirement should be somewhat independent of the training set size and not require excessive amount of ram for small subset of the dataset.