Need preprocessor config error

[mtkgeek]

Hi, I'm trying to run the phi3-vision model using local files. I followed the instructions but ran into this error

panicked at mistralrs-core/src/pipeline/vision.rs:192:22: Need preprocessor config

Even though the instructions do not say a preprocessor config file is needed, I added that as well but still ran into the same error. I'm on a Mac btw

MY CODE

from mistralrs import Runner, Which, ChatCompletionRequest, VisionArchitecture

runner = Runner(
    which=Which.VisionPlain(
        model_id="/Users/path/to/my/phi3v_model",
        tokenizer_json=None,
        repeat_last_n=64,
        arch=VisionArchitecture.Phi3V,
    ),
)

res = runner.send_chat_completion_request(
    ChatCompletionRequest(
        model="phi3v",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "https://www.nhmagazine.com/content/uploads/2019/05/mtwashingtonFranconia-2-19-18-108-Edit-Edit.jpg"
                        },
                    },
                    {
                        "type": "text",
                        "text": "<|image_1|>\nWhat is shown in this image? Write a detailed response analyzing the scene.",
                    },
                ],
            }
        ],
        max_tokens=256,
        presence_penalty=1.0,
        top_p=0.1,
        temperature=0.1,
    )
)
print(res.choices[0].message.content)
print(res.usage)

[EricLBuehler]

Hi @mtkgeek, I just merged #489 which should fix this. As you are using the Python API, can you please follow the instructions to install from source, and try it again?

[mtkgeek]

Oh nice. I'm using the pip version. I tried installing from source but keep running into errors when I run pip install maturin[patchelf].

Building wheels for collected packages: patchelf
  Building wheel for patchelf (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for patchelf (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Traceback (most recent call last):
        File "/private/var/folders/7m/2srhc2n972gfvzq9jtj9n7rr0000gn/T/pip-build-env-x0rlsvm9/overlay/lib/python3.10/site-packages/skbuild/setuptools_wrap.py", line 643, in setup
          cmkr = cmaker.CMaker(cmake_executable)
        File "/private/var/folders/7m/2srhc2n972gfvzq9jtj9n7rr0000gn/T/pip-build-env-x0rlsvm9/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 145, in __init__
          self.cmake_version = get_cmake_version(self.cmake_executable)
        File "/private/var/folders/7m/2srhc2n972gfvzq9jtj9n7rr0000gn/T/pip-build-env-x0rlsvm9/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 102, in get_cmake_version
          raise SKBuildError(msg) from err

      Problem with the CMake installation, aborting build. CMake executable is cmake
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for patchelf
Failed to build patchelf
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (patchelf

I guess I'll just wait for these changes to make it to pypi.

[EricLBuehler]

@mtkgeek I just released v0.1.23 which includes this fix. Can you please try it again after a pip reinstall?

[mtkgeek]

Thanks for the update @EricLBuehler. Now pip install doesn't work though

ERROR

Compiling png v0.17.13
     Compiling candle-core v0.6.0 (https://github.com/EricLBuehler/candle.git?rev=b438cba3f8742519d6f2154209e096c123fc3eb9#b438cba3)
Compiling hyper v1.3.1
  error[E0308]: mismatched types
     --> /Users/me/.cargo/git/checkouts/candle-c6a149c3b35a488f/b438cba/candle-core/src/quantized/metal.rs:132:39
      |
  132 |         let src = crate::Storage::Cpu(src);
      |                   ------------------- ^^^ expected `CpuStorage`, found `&CpuStorage`
      |                   |
      |                   arguments to this enum variant are incorrect
      |
  note: tuple variant defined here
     --> /Users/me/.cargo/git/checkouts/candle-c6a149c3b35a488f/b438cba/candle-core/src/storage.rs:10:5
      |
  10  |     Cpu(CpuStorage),
      |     ^^^
  help: consider using clone here
      |
  132 |         let src = crate::Storage::Cpu(src.clone());
      |                                          ++++++++

  error[E0599]: no method named `as_slice` found for enum `Storage` in the current scope
      --> /Users/me/.cargo/git/checkouts/candle-c6a149c3b35a488f/b438cba/candle-core/src/quantized/metal.rs:136:36
       |
  136  |             storage.from_float(src.as_slice::<f32>()?)?;
       |                                    ^^^^^^^^ method not found in `Storage`
       |
      ::: /Users/me/.cargo/git/checkouts/candle-c6a149c3b35a488f/b438cba/candle-core/src/storage.rs:9:1
       |
  9    | pub enum Storage {
       | ---------------- method `as_slice` not found for this enum
       |
  note: the method `as_slice` exists on the type `CpuStorage`
      --> /Users/me/.cargo/git/checkouts/candle-c6a149c3b35a488f/b438cba/candle-core/src/cpu_backend/mod.rs:1541:5
       |
  1541 |     pub fn as_slice<D: WithDType>(&self) -> Result<&[D]> {
       |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

     Compiling signal-hook-mio v0.2.3
     Compiling derive_builder_macro v0.20.0
     Compiling darling_macro v0.11.0
  Some errors have detailed explanations: E0308, E0599.
  For more information about an error, try `rustc --explain E0308`.
     Compiling itertools v0.11.0
  error: could not compile `candle-core` (lib) due to 2 previous errors
  warning: build failed, waiting for other jobs to finish...
  💥 maturin failed
    Caused by: Failed to build a native library through cargo
    Caused by: Cargo build finished with "exit status: 101"

[EricLBuehler]

@mtkgeek thank you for reporting this, can you please try it again after you update via pip - it should be fixed?

[mtkgeek]

@EricLBuehler the new version installs but the original problem still persists. PanicException: Need preprocessor config when I run the code.

[EricLBuehler]

@EricLBuehler the new version installs but the original problem still persists. PanicException: Need preprocessor config when I run the code.

Can you please paste the output of ls /Users/path/to/my/phi3v_model?

[mtkgeek]

Sure @EricLBuehler . It's

config.json                             model.safetensors.index.json
model-00001-of-00002.safetensors        tokenizer.json
model-00002-of-00002.safetensors        tokenizer_config.json

[EricLBuehler]

Ah ok! I think I must have forgotten to update the docs. As documented here now, if you add preprocessor_config.json this should work.

[mtkgeek]

Thanks, it works now. I also wanted to point out that I can see Loading model/Users/path/to/my/phi3v_modelon cpu. is this normal, I was expecting "on metal" or "on mps" ?

[EricLBuehler]

Ok, that is good that it works. Regarding the device, can you close this issue and open another for that?