importing `xgboost` before `transformers.pipeline` breaks pipeline loading for `image-feature-extraction` pipelines

dan-jacobson commented 2 months ago

System Info

transformers version: 4.44.2
Platform: macOS-14.6.1-arm64-arm-64bit
Python version: 3.12.2
Huggingface_hub version: 0.25.0
Safetensors version: 0.4.5
Accelerate version: not installed
Accelerate config: not found
PyTorch version (GPU?): 2.4.1 (False)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using distributed or parallel set-up in script?: No

Who can help?

@Rocketknight1, @amyeroberts, @qubvel

Information

[X] The official example scripts
[X] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction

mkdir pipeline_test && cd pipeline_test
uv init
uv add transformers torch pillow xgboost
uv run python

import xgboost
from transformers import pipeline
pipe = pipeline(task="image-feature-extraction")

This fails silently in my terminal. For some reason I can't get a good error. In VSCode I can sometimes get: [1] 88800 segmentation fault python.

Interestingly, it's the order of xgboost/transformers import that matters. This works on my machine:

from transformers import pipeline
import xgboost

pipe = pipeline(task="image-feature-extraction")

A couple other things I noticed:

Does not appear to be model specific. I was able to get failures using multiple dinov2 models, as well as the base model, google/vit-base-patch16-224.
Does not appear to be a problem for all pipelines. I tried task="feature-extraction" and that worked. I haven't exhaustively tried other pipelines.

Expected behavior

I expected my pipeline to load.

LysandreJik commented 1 month ago

Hey @dan-jacobson, I'm not sure it's transformers, I have reason to believe it's torch :grin:

Can you let me know if doing import xgboost followed by import torch fails?

LysandreJik commented 1 month ago

Or, another potential thing to look into: uninstall torch and install tensorflow as a backend for pipelines; now do the same: import xgboost and pipeline. Does that still fail?

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

huggingface / transformers