Evision.DNN.DetectionModel can't setInputParams at YOLOv4

[thehaigo]

I tried to use YOLOv4 at Evision.DNN.DetectionModel

dog = "yolov4/dog.jpg"
weights = "yolov4/yolov4.weights"
config = "yolov4/yolov4.cfg"
classes = "yolov4/coco.names"
mat = Evision.imread(dog)

Evision.DNN.DetectionModel.detectionModel(weights, config: config)
|> Evision.DNN.DetectionModel.detect(mat)

return error

{:error,
 "OpenCV(4.6.0) /Users/runner/work/evision/evision/3rd_party/opencv/opencv-4.6.0/modules/dnn/src/model.cpp:98: error: (-201:Incorrect size of input array) Input size not specified in function 'processFrame'\n"}

i guess need such as python code

model.setInputParams(1.0, (416, 416), (0.0, 0.0, 0.0), true, false)

but

Evision.DNN.DetectionModel.detectionModel(weights, config: config)
|> Evision.DNN.Model.setInputParams(scale: 1.0, size: {416, 416}, mean: {0, 0, 0}, swapRB: true, crop: false)

** (UndefinedFunctionError) function Evision.DNN.Model.setInputParams/2 is undefined or private. Did you mean:

Evision.DNN.DetectionModel.detectionModel(weights, config: config)
|> Evision.DNN.Model.setInputParams({416, 416})

{:error, "cannot get `cv::dnn::Model` from `self`: mismatched type or invalid resource?"}

How about the following solution? 1 Convert the loaded model to Evision.DNN.Model 2 Enable setInputParams in DetectionModel as well

Evision.DNN.readFromDarknet load and detect are success but, I wan't to impl post process :-(

Enviroments evision 0.1.16 elixir 1.14.1 elrang 25.0.1 os macOS 12.6

[cocoa-xu]

Hi @thehaigo thanks for reporting! I'll look into this issue today. :)

[cocoa-xu]

setInputParams/2 is fixed in v0.1.17

This code should work as expected now. :) @thehaigo

Evision.DNN.DetectionModel.detectionModel(weights, config: config)
|> Evision.DNN.Model.setInputParams(scale: 1.0, size: {416, 416}, mean: {0, 0, 0}, swapRB: true, crop: false)

[thehaigo]

Ev.DNN.DetectionModel.detectionModel(weights, config: config)
|> Ev.DNN.Model.setInputParams(scale: 1.0, size: {416, 416}, mean: {0, 0, 0}, swapRB: true, crop: false)

return error

{:error, "cannot get `cv::dnn::Model` from `self`: mismatched type or invalid resource?"}

precompile can't install

Mix.install(
  [
    {:evision, "~> 0.1.17"},
    {:kino, "~> 0.7.0"}
  ]
)

Compiling 1 file (.ex)
Generated elixir_make app

22:24:57.041 [info] EVISION_PREFER_PRECOMPILED: true; try to download and use the precompiled library.

22:24:57.042 [info] Requested version `0.1.17` has precompiled binaries.

22:24:57.043 [info] Current target `aarch64-apple-darwin` has precompiled binaries.

22:24:57.043 [info] Current NIF version `2.16` has precompiled binaries.

22:24:57.051 [info] Precompiled binary tarball cached at /Users/shou/Library/Caches/evision-nif_2.16-aarch64-apple-darwin-0.1.17.tar.gz, sha256=fb388dd732462ad11cd0154caea8be144ab89cb00c627212a891f6df62d8486e
==> evision
could not compile dependency :evision, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile evision", update it with "mix deps.update evision" or clean it with "mix deps.clean evision"

** (FunctionClauseError) no function clause matching in IO.chardata_to_string/1    

    The following arguments were given to IO.chardata_to_string/1:

        # 1
        {"/Users/shou/Library/Caches/mix/installs/elixir-1.14.1-erts-13.0.4/7d3716c1e3be082adf0e059eed9641b4/deps/evision/checksum-evision.exs",
         "/Users/shou/Library/Caches/mix/installs/elixir-1.14.1-erts-13.0.4/7d3716c1e3be082adf0e059eed9641b4/deps/evision/checksum_evision.erl"}

    Attempted function clauses (showing 2 out of 2):

        def chardata_to_string(string) when is_binary(string)
        def chardata_to_string(list) when is_list(list)

    (elixir 1.14.1) IO.chardata_to_string/1
    (elixir 1.14.1) lib/file.ex:344: File.read/1
    /Users/shou/Library/Caches/mix/installs/elixir-1.14.1-erts-13.0.4/7d3716c1e3be082adf0e059eed9641b4/deps/evision/mix.exs:81: Mix.Tasks.Compile.EvisionPrecompiled.read_checksum_map/1
    /Users/shou/Library/Caches/mix/installs/elixir-1.14.1-erts-13.0.4/7d3716c1e3be082adf0e059eed9641b4/deps/evision/mix.exs:561: Mix.Tasks.Compile.EvisionPrecompiled.verify_checksum/3
    /Users/shou/Library/Caches/mix/installs/elixir-1.14.1-erts-13.0.4/7d3716c1e3be082adf0e059eed9641b4/deps/evision/mix.exs:490: Mix.Tasks.Compile.EvisionPrecompiled.prepare/4
    (mix 1.14.1) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
    (mix 1.14.1) lib/mix/tasks/compile.all.ex:92: Mix.Tasks.Compile.All.run_compiler/2
    /Users/shou/livebook_samples/OpenCV/opencv_dnn_ch7.livemd#cell:setup:1: (file)

below is success

Mix.install(
  [{:evision, "~> 0.1.17"}],
  system_env: [
    {"EVISION_PREFER_PRECOMPILED", "false"}
  ]
)

[thehaigo]

tried ubuntu

docker run -p 8080:8080 -p 8081:8081 --pull always livebook/livebook

mix install same error

13:35:11.611 [info] EVISION_PREFER_PRECOMPILED: true; try to download and use the precompiled library.

13:35:11.612 [info] Requested version `0.1.17` has precompiled binaries.

13:35:11.612 [info] Current target `aarch64-linux-gnu` has precompiled binaries.

13:35:11.612 [info] Current NIF version `2.16` has precompiled binaries.

13:35:15.346 [info] Precompiled binary tarball downloaded and saved to /home/livebook/.cache/evision-nif_2.16-aarch64-linux-gnu-0.1.17.tar.gz, sha256=19b59d93d15f8d9b6ec9ed5a607d50ce2fd6634d69ac6b9a7a27104254041df1
==> evision
could not compile dependency :evision, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile evision", update it with "mix deps.update evision" or clean it with "mix deps.clean evision"

** (FunctionClauseError) no function clause matching in IO.chardata_to_string/1    

    The following arguments were given to IO.chardata_to_string/1:

        # 1
        {"/home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/ee693d317f2cbd2d172de21c2538751d/deps/evision/checksum-evision.exs",
         "/home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/ee693d317f2cbd2d172de21c2538751d/deps/evision/checksum_evision.erl"}

    Attempted function clauses (showing 2 out of 2):

        def chardata_to_string(string) when is_binary(string)
        def chardata_to_string(list) when is_list(list)

    (elixir 1.14.0) IO.chardata_to_string/1
    (elixir 1.14.0) lib/file.ex:344: File.read/1
    /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/ee693d317f2cbd2d172de21c2538751d/deps/evision/mix.exs:81: Mix.Tasks.Compile.EvisionPrecompiled.read_checksum_map/1
    /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/ee693d317f2cbd2d172de21c2538751d/deps/evision/mix.exs:561: Mix.Tasks.Compile.EvisionPrecompiled.verify_checksum/3
    /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/ee693d317f2cbd2d172de21c2538751d/deps/evision/mix.exs:534: Mix.Tasks.Compile.EvisionPrecompiled.prepare/4
    (mix 1.14.0) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
    (mix 1.14.0) lib/mix/tasks/compile.all.ex:92: Mix.Tasks.Compile.All.run_compiler/2
    (mix 1.14.0) lib/mix/tasks/compile.all.ex:72: Mix.Tasks.Compile.All.compile/4

[cocoa-xu]

Oh I see, fix is on the way!

[cocoa-xu]

Hi @thehaigo, I released v0.1.18 and also added a test for Evision.DNN.DetectionModel with yolov4.

The following code should give you the expected output now.

iex> dog = "dog.jpg"
iex> weights = "yolov4.weights"
iex> config = "yolov4.cfg"
iex> mat = Evision.imread(dog)
iex> Evision.DNN.DetectionModel.detectionModel(weights, config: config)
iex> |> Evision.DNN.DetectionModel.setInputParams(scale: 1.0, size: {416, 416}, mean: {0, 0, 0}, swapRB: true, crop: false)
iex> |> Evision.DNN.DetectionModel.detect(mat)
{[0, 24, 24, 24, 0, 24, 24, 0, 47, 9, 9, 9, 9, 9, 9, 26, 24, 12, 12, 47, 24, 24,
  24, 26, 26, 26, 26, 26, 26, 26, 47, 26, 26],
 [0.6718131303787231, 0.8944537043571472, 0.5458226799964905, 0.674974799156189,
  0.9186239838600159, 0.9345014691352844, 0.8716281056404114,
  0.6807704567909241, 0.529909610748291, 0.8524839878082275, 0.7546082735061646,
  0.6172801852226257, 0.6529494524002075, 0.7684603929519653,
  0.8480220437049866, 0.9516295790672302, 0.9047489166259766,
  0.9447915554046631, 0.7290478348731995, 0.7423124313354492,
  0.8002534508705139, 0.6577810645103455, 0.6949653029441833,
  0.8610342144966125, 0.5206403136253357, 0.5880995392799377, 0.746155858039856,
  0.6399684548377991, 0.7865645289421082, 0.6371464729309082,
  0.6925340294837952, 0.7681138515472412, 0.6114254593849182],
 [
   {68, 7, 20, 2},
   {87, 7, 13, 1},
   {203, 6, 25, 3},
   {220, 5, 24, 3},
   {235, 7, 18, 2},
   {244, 6, 22, 4},
   {256, 7, 33, 3},
   {269, 7, 38, 2},
   {443, 8, 20, 1},
   {665, 4, 9, 6},
   {720, 8, 14, 4},
   {725, 6, 41, 5},
   {756, 2, 12, 5},
   {578, 76, 2, 5},
   {737, 74, 4, 7},
   {545, 132, 1, 1},
   {235, 176, 7, 9},
   {368, 211, 4, 1},
   {379, 211, 5, 1},
   {457, 213, 3, 2},
   {422, 222, 4, 1},
   {0, 274, 1, 41},
   {0, 307, 1, 38},
   {0, 420, 1, 17},
   {0, 433, 1, 13},
   {0, 443, 1, 14},
   {0, 463, 1, 17},
   {0, 495, 1, 20},
   {0, 505, 1, 16},
   {766, 518, 2, 10},
   {161, 533, 6, 3},
   {767, 531, 1, 7},
   {0, 535, 1, 28}
 ]}

[thehaigo]

@cocoa-xu yolov4 work! thank you berry much! 🙏

mage = base <> "yolov4/dog.jpg"
weights = base <> "yolov4/yolov4.weights"
config = base <> "yolov4/yolov4.cfg"
dog = Ev.imread(image) |> Ev.resize({416,416})

model =
  Ev.DNN.DetectionModel.detectionModel(weights, config: config)
  |> Ev.DNN.DetectionModel.setInputParams(
    scale: 1.0 / 255.0,
    size: {416, 416},
    mean: {0, 0, 0},
    swapRB: true,
    crop: false
  )

{class_ids, confidences, boxes} 
 = Ev.DNN.DetectionModel.detect(model, dog, confThreshold: 0.5, nmsThreshold: 0.4)

boxes |> Enum.reduce(dog, fn {x,y,w,h}, mat -> 
  Ev.rectangle(mat, {x,y}, {x+w,y+h}, {0,0,255})
end)

[cocoa-xu]

Perfect!

[jodyalbritton]

@cocoa-xu I am getting this error again with the latest version, built from source.

{:error,
 "cannot get `cv::dnn::DetectionModel` from `self`: mismatched type or invalid resource?"}

[cocoa-xu]

Hi @jodyalbritton, I built it from source, and it appeared to work here. Could you please provide more information regarding your environment and/or related lines?

Below is the full log of compiling this library from source, loading, setting input data and getting the output using YOLOv4.

https://asciinema.org/a/VGICovAqdyCphtEETvXA6BUBn

Compiling stage: starting from 0:00 to 3:40 Using YOLOv4: starting from 3:40

[jodyalbritton]

recompiled and it is working now. Not sure what I did wrong. Thanks for responding.