huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
135.32k stars 27.08k forks source link

Fix support for image processors modifications in modular #34866

Open yonigozlan opened 20 hours ago

yonigozlan commented 20 hours ago

What does this PR do?

I need to add a method to an existing image processor in the GotOcr2 PR https://github.com/huggingface/transformers/pull/34721 Right now there seems to be a bug where the model name is not correctly deduced from the file name when we inherit from a module in an image_processing* file. With the example added, I get this error:

Traceback (most recent call last):
  File "/home/ubuntu/models_implem/transformers/utils/modular_model_converter.py", line 1547, in <module>
    converted_files = convert_modular_file(file_name, args.old_model_name, args.new_model_name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/models_implem/transformers/utils/modular_model_converter.py", line 1483, in convert_modular_file
    wrapper.visit(cst_transformers)
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/metadata/wrapper.py", line 204, in visit
    return self.module.visit(visitor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/_nodes/module.py", line 89, in visit
    result = super(Module, self).visit(visitor)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/_nodes/base.py", line 233, in visit
    visitor.on_leave(self)
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/_visitors.py", line 137, in on_leave
    leave_func(original_node)
  File "/home/ubuntu/models_implem/transformers/utils/modular_model_converter.py", line 1199, in leave_Module
    renamed_module = module.visit(renamer)
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/_nodes/module.py", line 89, in visit
    result = super(Module, self).visit(visitor)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/_nodes/base.py", line 227, in visit
    _CSTNodeSelfT, self._visit_and_replace_children(visitor)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/_nodes/module.py", line 74, in _visit_and_replace_children
    body=visit_body_sequence(self, "body", self.body, visitor),
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/_nodes/internal.py", line 227, in visit_body_sequence
    return tuple(visit_body_iterable(parent, fieldname, children, visitor))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/_nodes/internal.py", line 193, in visit_body_iterable
    new_child = child.visit(visitor)
                ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/_nodes/base.py", line 236, in visit
    leave_result = visitor.on_leave(self, with_updated_children)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/matchers/_visitors.py", line 515, in on_leave
    retval = CSTTransformer.on_leave(self, original_node, updated_node)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hf_311_121/lib/python3.11/site-packages/libcst/_visitors.py", line 71, in on_leave
    updated_node = leave_func(original_node, updated_node)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/models_implem/transformers/utils/modular_model_converter.py", line 122, in leave_ClassDef
    new_name = convert_to_camelcase(updated_node.name.value, self.old_name, self.default_old_name)
                                                                            ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ReplaceNameTransformer' object has no attribute 'default_old_name'

Things work as expected with the following change: https://github.com/huggingface/transformers/blob/4e90b99ed916300b80bac9db793f2a96b2a87122/utils/modular_model_converter.py#L1195

to

file_model_name = file.split(".")[-2]

As unless I'm missing something, the model name can be deduced from the folder name in which modeling/processing/config files are located?

Before submitting

Who can review?

@Cyrilvallez @ArthurZucker

HuggingFaceDocBuilderDev commented 19 hours ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.