ValueError: The checkpoint you are trying to load has model type `florence2` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[Romanio1997]

System Info

Windows 10x64 pytorch version: 2.4.0+cu124 Python 3.11.8 transformers-4.46.0. dev0

Who can help?

No response

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

ComfyUI Error Report

Error Details

• Node Type: DownloadAndLoadFlorence2Model
• Exception Type: ValueError

• Exception Message: The checkpoint you are trying to load has model type florence2 but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

  Stack Trace

  
  File "C:\pinokio\ComfyUI_windows_portable\ComfyUI\execution.py", line 323, in execute
  output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  
  File "C:\pinokio\ComfyUI_windows_portable\ComfyUI\execution.py", line 198, in get_output_data
  return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  
  File "C:\pinokio\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in _map_node_over_list
  process_inputs(input_dict, i)
  
  File "C:\pinokio\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs
  results.append(getattr(obj, func)(**inputs))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  
  File "C:\pinokio\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Florence2\nodes.py", line 97, in loadmodel
  model = AutoModelForCausalLM.from_pretrained(model_path, attn_implementation=attention, device_map=device, torch_dtype=dtype,trust_remote_code=True)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  
  File "C:\pinokio\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 526, in from_pretrained
  config, kwargs = AutoConfig.from_pretrained(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  
  File "C:\pinokio\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 1027, in from_pretrained
  raise ValueError(

Expected behavior

DownloadAndLoadFlorence2Model The checkpoint you are trying to load has model type florence2 but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[cct1018]

I meet the same issue on linux with transformers v4.45.1

[LysandreJik]

Hmmm florence2 should be using remote code, and I see it's set to True in there.

@Rocketknight1, could you take a look at this please?

[Rocketknight1]

Hi @Romanio1997 @cct1018, can you tell me which checkpoint you're trying to load so I can investigate?

[Rocketknight1]

I checked microsoft/Florence-2-base and microsoft/Florence-2-large and could not reproduce the issue.

[andrewstoliarov]

@Rocketknight1 try MiaoshouAI/Florence-2-base-PromptGen-v1.5

[Rocketknight1]

@andrewstoliarov I can reproduce the issue there - the cause is the model is misconfigured. Specifically, config.json is missing the auto_map key that is needed to load remote code correctly.

If you open a PR to that model repo to copy the auto_map key from a working florence2 model like microsoft/Florence-2-large, that should fix the issue!

[Romanio1997]

Hi @Romanio1997 @cct1018, can you tell me which checkpoint you're trying to load so I can investigate?

I used this workflow:

https://openart.ai/workflows/myaiforce/svOEmbK8C4zyJsCYmqnx

[Rocketknight1]

@Romanio1997 I'm afraid we can't really debug third-party tools like that if we don't know which checkpoints they're using! My suspicion is that it's loading a florence2 checkpoint on the Hub somewhere that is misconfigured, and this is not a bug in transformers.

[andrewstoliarov]

@Romanio1997 Here the question is about the checkpoint, not the workflow. In fact, you can use any other checkpoint from the list, as it won't significantly affect the result in this workflow.

[Romanio1997]

@Romanio1997 I'm afraid we can't really debug third-party tools like that if we don't know which checkpoints they're using! My suspicion is that it's loading a florence2 checkpoint on the Hub somewhere that is misconfigured, and this is not a bug in transformers.

The model florence2 was downloaded automatically from huggingface.

Possible causes of the error: Unsupported architecture: The checkpoint refers to a model architecture that is not supported by the version of the Transformers library you have installed. In this case, it is the Florence2 architecture, which may not yet be integrated into the library.

Older version of Transformers: The error may occur because your version of the Transformers library is out of date and does not support newer architectures such as Florence2.

[Romanio1997]

@Romanio1997 Here the question is about the checkpoint, not the workflow. In fact, you can use any other checkpoint from the list, as it won't significantly affect the result in this workflow.

I know. I just wanted to give you all the data related to the error.