Model "Salesforce - Blip large" fails

[ndrean]

Model: "Salesforce/blip-image-captioning-large" fails to build (but "base" is ok). Bumblebee v 0.4.2

[error] Kino.listen with #Function<42.105768164/1 in :erl_eval.expr/6> failed with reason:

** (ArgumentError) invalid dimension in axis 2 found in shape. Each dimension must be a positive integer, got {384, 384} in shape {1, {384, 384}, {384, 384}, 3}
    (nx 0.6.2) lib/nx/shape.ex:60: Nx.Shape.validate!/3
    (nx 0.6.2) lib/nx.ex:3785: anonymous fn/4 in Nx.broadcast/3
...

[jonatanklosko]

Good catch, fixed in a7f4896.

[ndrean]

@jonatanklosko Did you pushed your fix?

v0.4.2

(EXIT) shutdown: failed to start child: Nx.Serving
        ** (EXIT) an exception was raised:
            ** (ArgumentError) invalid dimension in axis 2 found in shape. Each dimension must be a positive integer, got {384, 384} in shape {2, {384, 384}, {384, 384}, 3}
                (nx 0.6.2) lib/nx/shape.ex:60: Nx.Shape.validate!/3
                (nx 0.6.2) lib/nx.ex:3785: anonymous fn/4 in Nx.broadcast/3
                (nx 0.6.2) lib/nx.ex:5431: Nx.apply_vectorized/2
                (nx 0.6.2) lib/nx/defn/expr.ex:1499: Nx.Defn.Expr.ensure_compatible/2
                (nx_image 0.1.1) lib/nx_image.ex:460: NxImage."__defn:to_continuous__"/3
                (bumblebee 0.4.2) lib/bumblebee/vision/blip_featurizer.ex:85: Bumblebee.Vision.BlipFeaturizer.process_batch/2
                (bumblebee 0.4.2) lib/bumblebee/vision/image_to_text.ex:36: anonymous fn/4 in Bumblebee.Vision.ImageToText.image_to_text/5
                (nx 0.6.2) lib/nx/defn/compiler.ex:158: Nx.Defn.Compiler.runtime_fun/3
                (exla 0.6.1) lib/exla/defn.ex:387: anonymous fn/4 in EXLA.Defn.compile/8
                (exla 0.6.1) lib/exla/defn/locked_cache.ex:36: EXLA.Defn.LockedCache.run/2
                (stdlib 5.0.2) timer.erl:270: :timer.tc/2
                (exla 0.6.1) lib/exla/defn.ex:385: EXLA.Defn.compile/8
                (exla 0.6.1) lib/exla/defn.ex:270: EXLA.Defn.__compile__/4
                (nx 0.6.2) lib/nx/defn.ex:305: Nx.Defn.compile/3
                (bumblebee 0.4.2) lib/bumblebee/vision/image_to_text.ex:45: anonymous fn/7 in Bumblebee.Vision.ImageToText.image_to_text/5
                (nx 0.6.2) lib/nx/serving.ex:1759: anonymous fn/3 in Nx.Serving.Default.init/3
                (elixir 1.15.7) lib/enum.ex:4789: Enum.with_index_list/3
                (nx 0.6.2) lib/nx/serving.ex:1755: Nx.Serving.Default.init/3
                (nx 0.6.2) lib/nx/serving.ex:1656: Nx.Serving.handle_init/4
                (nx 0.6.2) lib/nx/serving.ex:1289: Nx.Serving.init/1

[josevalim]

You need to use main I believe.

[ndrean]

ahhh forgot, yes

[LuchoTurtle]

How can I use this?

I'm downloading the dependencies with:

{:bumblebee, git: "https://github.com/elixir-nx/bumblebee", branch: "main", override: true},
{:nx, git: "https://github.com/elixir-nx/nx.git", sparse: "nx", override: true},
{:exla, "~> 0.6.4"},

And I get a weird error with EXLA.

(UndefinedFunctionError) function EXLA.__to_backend__/1 is undefined or private, but the behaviour Nx.Defn.Compiler expects it to be present

[josevalim]

{:bumblebee, git: "https://github.com/elixir-nx/bumblebee", branch: "main", override: true},
{:nx, git: "https://github.com/elixir-nx/nx.git", sparse: "nx", override: true},
{:exla, git: "https://github.com/elixir-nx/nx.git", sparse: "exla", override: true},

[LuchoTurtle]

That was fast, thank you!

Hopefully, these changes will be present in the next version of Bumblebee, NX and EXLA soon! :)