importing `xgboost` before `transformers.pipeline` breaks pipeline loading for `image-feature-extraction` pipelines

dan-jacobson commented 2 days ago

System Info

transformers version: 4.44.2
Platform: macOS-14.6.1-arm64-arm-64bit
Python version: 3.12.2
Huggingface_hub version: 0.25.0
Safetensors version: 0.4.5
Accelerate version: not installed
Accelerate config: not found
PyTorch version (GPU?): 2.4.1 (False)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using distributed or parallel set-up in script?: No

Who can help?

@Rocketknight1, @amyeroberts, @qubvel

Information

[X] The official example scripts
[X] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction

mkdir pipeline_test && cd pipeline_test
uv init
uv add transformers torch pillow xgboost
uv run python

import xgboost
from transformers import pipeline
pipe = pipeline(task="image-feature-extraction")

This fails silently in my terminal. For some reason I can't get a good error. In VSCode I can sometimes get: [1] 88800 segmentation fault python.

Interestingly, it's the order of xgboost/transformers import that matters. This works on my machine:

from transformers import pipeline
import xgboost

pipe = pipeline(task="image-feature-extraction")

A couple other things I noticed:

Does not appear to be model specific. I was able to get failures using multiple dinov2 models, as well as the base model, google/vit-base-patch16-224.
Does not appear to be a problem for all pipelines. I tried task="feature-extraction" and that worked. I haven't exhaustively tried other pipelines.

Expected behavior

I expected my pipeline to load.

LysandreJik commented 4 hours ago

Hey @dan-jacobson, I'm not sure it's transformers, I have reason to believe it's torch :grin:

Can you let me know if doing import xgboost followed by import torch fails?

LysandreJik commented 4 hours ago

Or, another potential thing to look into: uninstall torch and install tensorflow as a backend for pipelines; now do the same: import xgboost and pipeline. Does that still fail?

huggingface / transformers