Oneformer model exception on forward pass

[nickponline]

System Info

• transformers version: 4.35.0.dev0
• Platform: macOS-13.4-arm64-arm-64bit
• Python version: 3.10.6
• Huggingface_hub version: 0.16.4
• Safetensors version: 0.3.2
• Accelerate version: not installed
• Accelerate config: not found
• PyTorch version (GPU?): 2.1.0 (False)
• Tensorflow version (GPU?): 2.13.0 (False)
• Flax version (CPU?/GPU?/TPU?): 0.7.2 (cpu)
• Jax version: 0.4.14
• JaxLib version: 0.4.14
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

@amyeroberts @NielsRogge @praeclarumjj3

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

image = Image.open('image.jpg').convert('RGB')
mask = Image.open('mask.png').convert('L')
processor = AutoProcessor.from_pretrained("shi-labs/oneformer_coco_swin_large")
semantic_inputs = processor(images=image, segmentation_maps=mask, task_inputs=["semantic"], return_tensors="pt")
processor.tokenizer.batch_decode(semantic_inputs.task_inputs)
model = AutoModelForUniversalSegmentation.from_pretrained("shi-labs/oneformer_coco_swin_large")

with torch.no_grad():
  outputs = model(**semantic_inputs)

semantic_segmentation = processor.post_process_semantic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]

I get the exception:

texts = ["a semantic photo"] * self.num_text
TypeError: can't multiply sequence by non-int of type 'NoneType'

Expected behavior

It should succeed and return the loss.

Also when I modify to add num_text=1 it still fails:

image = Image.open('image.jpg').convert('RGB')
mask = Image.open('mask.png').convert('L')
processor = AutoProcessor.from_pretrained("shi-labs/oneformer_coco_swin_large", num_text=1)
semantic_inputs = processor(images=image, segmentation_maps=mask, task_inputs=["semantic"], return_tensors="pt")
processor.tokenizer.batch_decode(semantic_inputs.task_inputs)
model = AutoModelForUniversalSegmentation.from_pretrained("shi-labs/oneformer_coco_swin_large")

with torch.no_grad():
  outputs = model(**semantic_inputs)

semantic_segmentation = processor.post_process_semantic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]

I still get an exception but different one:

text_queries = nn.functional.normalize(text_queries.flatten(1), dim=-1)
AttributeError: 'NoneType' object has no attribute 'flatten'

[nickponline]

@amyeroberts do you know if I'm missing something. Thank for the help

[NielsRogge]

Hi @nickponline I finally had the time to look into OneFormer.

Uploaded a demo notebook regarding fine-tuning here: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/OneFormer/Fine_tune_OneFormer_for_semantic_segmentation.ipynb. Hope that helps!

[nickponline]

Thank you that is v. helpful!

On Sun, Nov 12, 2023 at 1:01 PM NielsRogge @.***> wrote:

Hi @nickponline https://github.com/nickponline I finally had the time to look into OneFormer.

Uploaded a demo notebook regarding fine-tuning here: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/OneFormer/Fine_tune_OneFormer_for_semantic_segmentation.ipynb. Hope that helps!

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/27356#issuecomment-1807238164, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEQCR24IYPIVUI4PMXJG7DYEE2KXAVCNFSM6AAAAAA7CDZEJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBXGIZTQMJWGQ . You are receiving this because you were mentioned.Message ID: @.***>

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.