Implementing clip - Githubissues

ToluClassics / mlx-transformers

MLX Transformers is a library that provides model implementation in MLX. It uses a similar model interface as HuggingFace Transformers and provides a way to load and run models in Apple Silicon devices.

Apache License 2.0

47 stars 4 forks source link

Implementing clip #11

Open Seun-Ajayi opened 4 months ago

Seun-Ajayi commented 4 months ago

Proposed changes

Implementing clip. See CLIPTextTransformer, line 449 and 455 both require _create_4d_causal_attention_mask and _prepare_4d_attention_mask respectively . You can find the implementation of this functions and the class the depend on here on transformers - https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_attn_mask_utils.py

Types of changes

What types of changes does your code introduce? Put an x in the boxes that apply

[x] New Model Architecture
[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)