Nerves support - precompiled artifacts

gworkman commented 10 months ago

Hello! First, thanks for such a great library. I've primarily been working with OpenCV, and I've been enjoying using something a little lighter-weight but still full featured. However, I'm running into some issues for my Nerves deployment, and hoping to find a fix.

I am trying to use Image/Vix in a Nerves project, based on a Raspberry Pi 3B. I'm running into some issues with the precompiled artifacts - I can see that when I try to make a firmware by running MIX_ENV=prod MIX_TARGET=rpi3 mix firmware that it is trying to download libvips for Mac. I think I need would need the vix-nif-2.17-armv7l-linux-gnueabihf-0.23.1.tar.gz and libvips-8.14.2-linux-armv7.tar.gz precompiled artifacts instead.

I haven't used elixir_make before, so not sure if this is just a quick config thing to set.

I'm a little perplexed though because it look like it isn't even a compilation problem - rather the :http_util module isn't available for the download? I found this similar issue about the code load paths changing in OTP 26/Elixir 1.15

Thanks!

Erlang/OTP 26 [erts-14.0.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Elixir 1.15.4 (compiled with Erlang/OTP 26)

mix.lock:
"vix": {:hex, :vix, "0.23.1", "f0cacb0334a0b4d12fbd7d8b14c78e27bb3cb47c977f5f9abc66162499d03160"}

❯ MIX_ENV=prod MIX_TARGET=rpi3 mix firmware
==> nerves
==> pnp_connect

Nerves environment
  MIX_TARGET:   rpi3
  MIX_ENV:      prod

==> vix

23:13:17.618 [debug] Fetching https://github.com/akash-akya/sharp-libvips/releases/download/v8.14.2-rc2/libvips-8.14.2-darwin-x64.tar.gz
** (UndefinedFunctionError) function :http_util.timestamp/0 is undefined (module :http_util is not available)
    (inets 9.0.1) :http_util.timestamp()
    (inets 9.0.1) httpc.erl:750: :httpc.handle_request/9
    /Users/.../deps/vix/build_scripts/precompiler.exs:81: Vix.LibvipsPrecompiled.download/2
    /Users/.../deps/vix/build_scripts/precompiler.exs:18: Vix.LibvipsPrecompiled.fetch_libvips/0
    /Users/.../deps/vix/build_scripts/precompiler.exs:186: (file)
make[1]: *** [/Users/.../_build/rpi3_prod/lib/vix/priv/precompiled_libvips] Error 1
make: *** [all] Error 2
could not compile dependency :vix, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile vix --force", update it with "mix deps.update vix" or clean it with "mix deps.clean vix"
==> pnp_connect
** (Mix) Could not compile with "make" (exit status: 2).
You need to have gcc and make installed. Try running the
commands "gcc --version" and / or "make --version". If these programs
are not installed, you will be prompted to install them.

❯ gcc --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: x86_64-apple-darwin22.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

gworkman commented 10 months ago

Update: can confirm that these are two separate issues. I added :inets to the extra applications in devs/vix/mix.exs and it does correctly download the Mac precompiled binary. However, it brings me back to the first point - I need the nerves architecture binary instead. I noticed that it should be started during the run function of theprecompiler.exs script...

  def run do
    {:ok, _} = Application.ensure_all_started(:inets)
    {:ok, _} = Application.ensure_all_started(:ssl)

    fetch_libvips()
  end

EDIT: I should also note that I didn't have any issues with this dependency while developing locally on my Mac. Not sure what about the prod build for the target which broke it instead...

gworkman commented 10 months ago

Additionally, I poked around and saw that I can set the "CC_PRECOMPILER_CURRENT_TARGET="armv7l-linux-gnueabihf" environment variable to force compilation for a specific target. It even looks like it picked up the right toolchain:

~/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-darwin_x86_64-1.8.0/bin/armv7-nerves-linux-gnueabihf-gcc

However, now I'm getting some other compilation errors, relating to incompatible pointer types. They're treated as errors though, so it might work if I can find a way to force the compilation...

~/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-darwin_x86_64-1.8.0/bin/armv7-nerves-linux-gnueabihf-gcc -c -mabi=aapcs-linux -mfpu=fp-armv8 -marm -fstack-protector-strong -mfloat-abi=hard -mcpu=cortex-a53 -fPIE -pie -Wl,-z,now -Wl,-z,relro -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -pipe -O2 --sysroot ~/.nerves/artifacts/project-portable-1.24.1/staging -O2 -Wall -Werror -Wextra -Wno-unused-parameter -Wmissing-prototypes -std=c11 -D_POSIX_C_SOURCE=200809L -fPIC -I ~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/erts-14.1/include -I ~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/lib/erl_interface-5.4/include -pthread -I ~/project_root/_build/pcblabs_rpi3_prod/lib/vix/priv/precompiled_libvips/include -I ~/project_root/_build/pcblabs_rpi3_prod/lib/vix/priv/precompiled_libvips/lib/glib-2.0/include -I ~/project_root/_build/pcblabs_rpi3_prod/lib/vix/priv/precompiled_libvips/include/glib-2.0 -o g_object/g_value.o g_object/g_value.c
g_object/g_value.c: In function 'set_int64':
g_object/g_value.c:95:34: error: passing argument 3 of 'enif_get_int64' from incompatible pointer type [-Werror=incompatible-pointer-types]
   95 |   if (!enif_get_int64(env, term, &int64_value)) {
      |                                  ^~~~~~~~~~~~
      |                                  |
      |                                  long int *
In file included from g_object/../utils.h:4,
                 from g_object/g_value.c:3:
~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/erts-14.1/include/erl_nif_api_funcs.h:134:88: note: expected 'ErlNifSInt64 *' {aka 'long long int *'} but argument is of type 'long int *'
  134 | ERL_NIF_API_FUNC_DECL(int,enif_get_int64,(ErlNifEnv*, ERL_NIF_TERM term, ErlNifSInt64* ip));
      |                                                                          ~~~~~~~~~~~~~~^~
~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/erts-14.1/include/erl_nif.h:369:76: note: in definition of macro 'ERL_NIF_API_FUNC_DECL'
  369 | #  define ERL_NIF_API_FUNC_DECL(RET_TYPE, NAME, ARGS) extern RET_TYPE NAME ARGS
      |                                                                            ^~~~
g_object/g_value.c: In function 'set_uint64':
g_object/g_value.c:124:35: error: passing argument 3 of 'enif_get_uint64' from incompatible pointer type [-Werror=incompatible-pointer-types]
  124 |   if (!enif_get_uint64(env, term, &uint64_value)) {
      |                                   ^~~~~~~~~~~~~
      |                                   |
      |                                   long unsigned int *
~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/erts-14.1/include/erl_nif_api_funcs.h:135:89: note: expected 'ErlNifUInt64 *' {aka 'long long unsigned int *'} but argument is of type 'long unsigned int *'
  135 | ERL_NIF_API_FUNC_DECL(int,enif_get_uint64,(ErlNifEnv*, ERL_NIF_TERM term, ErlNifUInt64* ip));
      |                                                                           ~~~~~~~~~~~~~~^~
~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/erts-14.1/include/erl_nif.h:369:76: note: in definition of macro 'ERL_NIF_API_FUNC_DECL'
  369 | #  define ERL_NIF_API_FUNC_DECL(RET_TYPE, NAME, ARGS) extern RET_TYPE NAME ARGS
      |                                                                            ^~~~
cc1: all warnings being treated as errors
make[1]: *** [g_object/g_value.o] Error 1
make: *** [all] Error 2

akash-akya commented 10 months ago

Hey @gworkman, thanks for checking out the project and for sharing your attempts at fixing the compilation.

I never tested Vix with nerves yet. My first guess is that I have to update NIF compilation scripts to respect MIX_TARGET and it should be good to go. Unfortunately I am a bit occupied for few days, I'll try to check and fix it within a week.

gworkman commented 10 months ago

Hi @akash-akya! Thanks for the prompt response on this :) that totally makes sense. I'd be happy to help poke around some more to see if I can compile it, if I can get some pointers on where to start. I have to admit though I haven't the most experience with more than a basic makefile.

akash-akya commented 10 months ago

Hi @gworkman, So I was checking this, and was able to make the NIF compilation successful. But the next step, elixir code compilation fails. Because Vix generates functions dynamically during elixir code compilation (aka macros) by running the NIF code to fetch all available libvips operations. This step would obviously fail because we can't load the NIF file for rpi3 on macos. So supporting cross-compilation is a bit tricky.

Btw, you should be able to compile firmware if you run the compilation process on rpi3 itself.

I'll look into the possibility of supporting cross-compilation, but with my currently understanding it needs significant changes and might take a while.

Thanks!

gworkman commented 10 months ago

Thanks @akash-akya for looking into it! Makes sense that it's going to have some issue on the function generation if it can't read the NIF at compile time.

Have you pushed the updated script to a branch? I'd love to test building on a separate RPi3.

Additionally, one thought I have is that when creating a custom nerves system, it compiles a custom Linux kernel and Elixir dependencies in a Docker container. I'm wondering if we can put Vix in the nerves system's mix.exs and take advantage of that Docker build step?

Edit: docs to nerves custom system https://hexdocs.pm/nerves/customizing-systems.html

I'm happy to give this idea a try later today 🙂

akash-akya commented 10 months ago

Have you pushed the updated script to a branch? I'd love to test building on a separate RPi3.

No change required for this, you should be able to build NIF as it is. Let me know if you face any issues. The changes I made were to make cross-compilation work. If you are compiling on RPi3 itself, then it should work by default.

Additionally, one thought I have is that when creating a custom nerves system, it compiles a custom Linux kernel and Elixir dependencies in a Docker container. I'm wondering if we can put Vix in the nerves system's mix.exs and take advantage of that Docker build step?

Interesting. I have to check more details on this. Basically the architecture where we compile the code and where we run the code should be the same. With docker, the CPU arch would still be the host machine, right?

I'm happy to give this idea a try later today 🙂

That's great! Let me know how it goes or if you need any help 🙂

gworkman commented 9 months ago

Hi @akash-akya! Sorry for the radio silence on my end - things got busy! I had a chance to try things out today. Using the latest 0.25.0 release, I tested compilation again for my nerves target. Unfortunately, I still had to do a bunch of the workarounds listed above to even get it to try to compile things. This includes:

export the CC_PRECOMPILER_CURRENT_TARGET env variable
add :inets to the extra applications in deps/vix/mix.exs

Even then, it doesn't compile the NIF correctly, as it errors with the same as above:

~/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-darwin_x86_64-1.8.0/bin/armv7-nerves-linux-gnueabihf-gcc -c -mabi=aapcs-linux -mfpu=fp-armv8 -marm -fstack-protector-strong -mfloat-abi=hard -mcpu=cortex-a53 -fPIE -pie -Wl,-z,now -Wl,-z,relro -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -pipe -O2 --sysroot ~/.nerves/artifacts/project-portable-1.24.1/staging -O2 -Wall -Werror -Wextra -Wno-unused-parameter -Wmissing-prototypes -std=c11 -D_POSIX_C_SOURCE=200809L -fPIC -I ~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/erts-14.1/include -I ~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/lib/erl_interface-5.4/include -pthread -I ~/project_root/_build/pcblabs_rpi3_prod/lib/vix/priv/precompiled_libvips/include -I ~/project_root/_build/pcblabs_rpi3_prod/lib/vix/priv/precompiled_libvips/lib/glib-2.0/include -I ~/project_root/_build/pcblabs_rpi3_prod/lib/vix/priv/precompiled_libvips/include/glib-2.0 -o g_object/g_value.o g_object/g_value.c
g_object/g_value.c: In function 'set_int64':
g_object/g_value.c:95:34: error: passing argument 3 of 'enif_get_int64' from incompatible pointer type [-Werror=incompatible-pointer-types]
   95 |   if (!enif_get_int64(env, term, &int64_value)) {
      |                                  ^~~~~~~~~~~~
      |                                  |
      |                                  long int *
In file included from g_object/../utils.h:4,
                 from g_object/g_value.c:3:
~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/erts-14.1/include/erl_nif_api_funcs.h:134:88: note: expected 'ErlNifSInt64 *' {aka 'long long int *'} but argument is of type 'long int *'
  134 | ERL_NIF_API_FUNC_DECL(int,enif_get_int64,(ErlNifEnv*, ERL_NIF_TERM term, ErlNifSInt64* ip));
      |                                                                          ~~~~~~~~~~~~~~^~
~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/erts-14.1/include/erl_nif.h:369:76: note: in definition of macro 'ERL_NIF_API_FUNC_DECL'
  369 | #  define ERL_NIF_API_FUNC_DECL(RET_TYPE, NAME, ARGS) extern RET_TYPE NAME ARGS
      |                                                                            ^~~~
g_object/g_value.c: In function 'set_uint64':
g_object/g_value.c:124:35: error: passing argument 3 of 'enif_get_uint64' from incompatible pointer type [-Werror=incompatible-pointer-types]
  124 |   if (!enif_get_uint64(env, term, &uint64_value)) {
      |                                   ^~~~~~~~~~~~~
      |                                   |
      |                                   long unsigned int *
~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/erts-14.1/include/erl_nif_api_funcs.h:135:89: note: expected 'ErlNifUInt64 *' {aka 'long long unsigned int *'} but argument is of type 'long unsigned int *'
  135 | ERL_NIF_API_FUNC_DECL(int,enif_get_uint64,(ErlNifEnv*, ERL_NIF_TERM term, ErlNifUInt64* ip));
      |                                                                           ~~~~~~~~~~~~~~^~
~/.nerves/artifacts/project-portable-1.24.1/staging/usr/lib/erlang/erts-14.1/include/erl_nif.h:369:76: note: in definition of macro 'ERL_NIF_API_FUNC_DECL'
  369 | #  define ERL_NIF_API_FUNC_DECL(RET_TYPE, NAME, ARGS) extern RET_TYPE NAME ARGS
      |                                                                            ^~~~
cc1: all warnings being treated as errors
make[1]: *** [g_object/g_value.o] Error 1
make: *** [all] Error 2

I noticed that the commit you referenced from a few weeks back added the precompiled versions, but it doesn't seem like it worked for me? At least, from above it seems like it isn't picking up the correct toolchain automatically.

Also, as you suspected, the attempt to compile in Docker didn't work either. The architecture stuff makes sense, I don't use it too much so not an expert.

For reference, I'm working on an intel-based Mac, with Elixir version 1.15.4-otp-26 and Erlang 26.0.2 (I believe these are the current Nerves supported versions).

Thanks for your time on this - I'd love to migrate away from OpenCV for this project. Let me know where I should poke around next 🙂

akash-akya commented 9 months ago

Hi @gworkman, thanks for testing it! how are you testing this? It seems like you are trying to cross-compile?

Can you ssh to RPi3 and test this and share result?

create vix.exs with this content on RPi3


Mix.install([{:vix, "~> 0.25.0"}])
alias Vix.Vips.{Image, Operation}

:ok = Operation.black!(500, 500, bands: 3) |> Image.write_to_file("black.jpg")


* run the script with command `$ elixir mix.exs`
* you should see a new image `black.jpg` created

If you see any errors, please share.

> Sorry for the radio silence on my end - things got busy!

I can understand, I am on a similar boat. Please take your time

gworkman commented 9 months ago

Ah, this looks like where we are differing! I'm specifically targeting Nerves, which compiles the firmware (including Elixir release and NIFs) on the host device, in my case an Intel-based MacBook. So yea, I think that would be cross compiling, because Nerves ships a toolchain to compile the native code for the target platform.

EDIT Does Mix.install work on Nerves SSH console?? I'll need to test that later when I get connected to a device if so! I assumed looking at the above that you had installed Elixir on a Raspbian image and accessed the iex console from there, but I'm realizing after re-reading that it may not be the case. END EDIT

Here's the steps to reproduce what I'm doing:

Install Nerves according to the docs. Note the Elixir/Erlang versions: asdf install erlang 26.0.2, asdf install elixir 1.15.4-otp-26
Create a new nerves project: mix nerves.new vix_test, install dependencies, etc
export MIX_TARGET=rpi3
Add vix to mix.exs and mix deps.get
mix firmware
This is where I run into issues, since when it goes to compile Vix, the above mentioned issues occur (fails to download precompiled binaries for right architecture, cross compile emits errors)

I'd be happy to do the above steps and push it to a minimum reproducible example repo if you'd like 🙂

I can understand, I am on a similar boat. Please take your time

I appreciate the help (and enthusiasm!) you've provided so far, be sure to take care of yourself as well!

akash-akya commented 9 months ago

compiles the firmware on the host device, in my case an Intel-based MacBook

I see. Yeah this won't work because Vix library must load the NIF (which is compiled for RPi3) to generate elixir code on macos like I mentioned before. But you could be able to do this on RPi itself though. I mean build the nerves firmware on the RPi3 itself.

There are few others options, such as building firmware using github actions, or using a virtual machine.

I'll look into if it is possible to compile multiple NIF and load different nifs based on target

gworkman commented 9 months ago

Yeah this makes complete sense. I think I intuitively understand this, but was excited that you mentioned you got it to work. I probably don't want to compile on GHA/a separate RPi3 because it will significantly reduce my workflow speed, but the approach to compile multiple NIFs and load different ones at runtime seems promising! Is there a way I can contribute?

akash-akya / vix

Nerves support - precompiled artifacts #130