search

zeam-vm / pelemay

Pelemay is a native compiler for Elixir, which generates SIMD instructions. It has a plan to generate for GPU code.
Apache License 2.0
186 stars 13 forks source link

Makefile isn't generated in case of Arch Linux #128

Closed thantez closed 4 years ago

thantez commented 4 years ago

Describe the bug Hi. I made a Test module to test Pelemay's features but I see an error.

To Reproduce First I update GCC and Make by sudo pacman -S make gcc

My PC Specifications:

This is Test module:

defmodule Test do
  require Pelemay
  import Pelemay

  defpelemay do
    def map_square (list) do
      list
      |> Enum.map(& &1 * &1)
    end

    def string_replace(list) do
      list
      |> Enum.map(& String.replace(&1, "Fizz", "Buzz"))
    end
  end
end

after adding {:pelemay, "~> 0.0.10"} to mix.exs, I wanted to be run and compile Test by iex -S mix that I see this error:

== Compilation error in file lib/test.ex ==
** (MatchError) no match of right hand side value: {"make: /run/media/thantez/Projects/elixir/test/_build/dev/lib/pelemay/build/libnifelixirtest.mk: No such file or directory\nmake: *** No rule to make target '/run/media/thantez/Projects/elixir/test/_build/dev/lib/pelemay/build/libnifelixirtest.mk'.  Stop.\n", 2}
    lib/pelemay/generator/builder.ex:196: Pelemay.Generator.Builder.generate/1
    expanding macro: Pelemay.defpelemay/1
    lib/test.ex:5: Test (module)
zacky1972 commented 4 years ago

Would you show me the error log from Logger tagged [Pelemay]? @thantez

zacky1972 commented 4 years ago

I've just released version 0.0.11. Would you please test your code on this version?

thantez commented 4 years ago

I updated pelemay to 0.0.11 and compiled but I got a new error:

Erlang/OTP 22 [erts-10.6.4] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Compiling 1 file (.ex)

== Compilation error in file lib/test.ex ==
** (RuntimeError) Build error.
    lib/pelemay/generator/builder.ex:179: Pelemay.Generator.Builder.generate_makefile/4
    lib/pelemay/generator/builder.ex:230: Pelemay.Generator.Builder.generate/1
    expanding macro: Pelemay.defpelemay/1
    lib/test.ex:6: Test (module)

I can't understand what you mean about Logger tagged [Pelemay]. Tags in logger are the same as metadata? Please tell me how I do that to I do.

zacky1972 commented 4 years ago

I see.

It shows a failure to generate a makefile to compile native code generated by Pelemay.

Would you perform the following commands and show me the results?

result 1

$ iex 
iex> "#{:code.root_dir()}/erts-#{:erlang.system_info(:version)}/include"

Keep the result 1 into an environment variable ERLANG_INCLUDE_PATH

result 2

$ cd path_to_the_project_of_Test
$ gcc -MM -I$(ERLANG_INCLUDE_PATH) -Ilib/pelemay/generator/native _build/dev/lib/pelemay/src/libnifelixirtest.c

result 3

$ cd path_to_the_project_of_Test
$ gcc -MM -I$(ERLANG_INCLUDE_PATH) -Ilib/pelemay/generator/native lib/pelemay/generator/native/basic.c

result 4

$ cd path_to_the_project_of_Test
$ gcc -MM -I$(ERLANG_INCLUDE_PATH) -Ilib/pelemay/generator/native lib/pelemay/generator/native/lsm.c
zacky1972 commented 4 years ago

I found the phenomenon you said:

I can't understand what you mean about Logger tagged [Pelemay]. Tags in logger are the same as metadata? Please tell me how I do that to I do.

That is, Pelemay doesn't show anything on Logger in this case.

Thus, I create another issue: #133

thantez commented 4 years ago

Ok. Here we go!

Result 1

iex(1)> "#{:code.root_dir()}/erts-#{:erlang.system_info(:version)}/include"
"/usr/lib/erlang/erts-10.6.4/include"

Result 2

$ gcc -MM -I$(ERLANG_INCLUDE_PATH) -Ilib/pelemay/generator/native _build/dev/lib/pelemay/src/libnifelixirtest.c
_build/dev/lib/pelemay/src/libnifelixirtest.c:8:10: fatal error: basic.h: No such file or directory
    8 | #include "basic.h"
      |          ^~~~~~~~~
compilation terminated.

Result 3

$ gcc -MM -I$(ERLANG_INCLUDE_PATH) -Ilib/pelemay/generator/native lib/pelemay/generator/native/basic.c
gcc: error: lib/pelemay/generator/native/basic.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.

Result 4

$ gcc -MM -I$(ERLANG_INCLUDE_PATH) -Ilib/pelemay/generator/native lib/pelemay/generator/native/lsm.c
gcc: error: lib/pelemay/generator/native/lsm.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.

To ensure the result, I once ran all of these commands without using environments vars. Like this:

$ gcc -MM -I/usr/lib/erlang/erts-10.6.4/include -Ilib/pelemay/generator/native _build/dev/lib/pelemay/src/libnifelixirtest.c

And I got same results. There are files inside erts includes:

$ ls /usr/lib/erlang/erts-10.6.4/include 
driver_int.h  erl_driver.h  erl_drv_nif.h  erl_fixed_size_int_types.h  erl_int_sizes_config.h  erl_memory_trace_parser.h  erl_nif_api_funcs.h  erl_nif.h  internal
zacky1972 commented 4 years ago

Would you change the command as follows? :

$ gcc -MM -I/usr/lib/erlang/erts-10.6.4/include -I./lib/pelemay/generator/native _build/dev/lib/pelemay/src/libnifelixirtest.c

If you'll get the same error, would you test the same command with modifing _build/dev/lib/pelemay/src/libnifelixirtest.c as follows?

...
#include <basic.h> // # "basic.h"
...
thantez commented 4 years ago 
$ gcc -MM -I/usr/lib/erlang/erts-10.6.4/include -I./lib/pelemay/generator/native _build/dev/lib/pelemay/src/libnifelixirtest.c

Have same error, but when I modified libnifelixirtest.c, I got another error:

$ gcc -MM -I/usr/lib/erlang/erts-10.6.4/include -I./lib/pelemay/generator/native _build/dev/lib/pelemay/src/libnifelixirtest.c
_build/dev/lib/pelemay/src/libnifelixirtest.c:9:10: fatal error: lsm.h: No such file or directory
    9 | #include "lsm.h"
      |          ^~~~~~~
compilation terminated.

So I modified that again to this:

#include "lsm.h"
// to
#include <lsm.h>

Then I run $ gcc -MM -I/usr/lib/erlang/erts-10.6.4/include -I./lib/pelemay/generator/native _build/dev/lib/pelemay/src/libnifelixirtest.c and it prints this:

$ gcc -MM -I/usr/lib/erlang/erts-10.6.4/include -Ilib/pelemay/generator/native _build/dev/lib/pelemay/src/libnifelixirtest.c 

libnifelixirtest.o: _build/dev/lib/pelemay/src/libnifelixirtest.c \
 /usr/lib/erlang/erts-10.6.4/include/erl_nif.h \
 /usr/lib/erlang/erts-10.6.4/include/erl_drv_nif.h \
 /usr/lib/erlang/erts-10.6.4/include/erl_int_sizes_config.h \
 /usr/lib/erlang/erts-10.6.4/include/erl_nif_api_funcs.h \
 _build/dev/lib/pelemay/src/map_elem1_mult_elem1_kernel.h \
 _build/dev/lib/pelemay/src/map_elem1_mult_elem1_kernel_driver.h

Ok. So I ran result 3 and result 4 again and I got the same errors that I had taken before. As a comment, I saw -I./lib/pelemay/generator/native have not any difference with -Ilib/pelemay/generator/native in these cases!

zacky1972 commented 4 years ago

Would you modify pelemay in deps in mix.exs as follows:

{:pelemay, git: "https://github.com/zeam-vm/pelemay.git", branch: "bug/128"}

And then, mix clean and iex -S mix, again?

thantez commented 4 years ago

I do:

# modify mix.exs
$ mix deps.clean --all
$ mix deps.get
$ mix clean
$ iex -S mix

There is still the build error.

== Compilation error in file lib/test.ex ==
** (RuntimeError) Build error.
    lib/pelemay/generator/builder.ex:179: Pelemay.Generator.Builder.generate_makefile/4
    lib/pelemay/generator/builder.ex:230: Pelemay.Generator.Builder.generate/1
    expanding macro: Pelemay.defpelemay/1
    lib/test.ex:6: Test (module)
zacky1972 commented 4 years ago

I updated the branch bug/128. Would you test it as follows?:

$ mix deps.clean --all
$ mix deps.get
$ mix clean
$ iex -S mix

It may create some files on _build/dev/lib/pelemay/build. if successfully, it should create the following files:

$ ls _build/dev/lib/pelemay/build
basic.d             libnifelixirtest.d  lsm.d
kernels.d           libnifelixirtest.mk

Would you please show me the result of ls and the contents of them?

thantez commented 4 years ago 
$ ls -a _build/dev/lib/pelemay/build
.  ..

No files have been created!

zacky1972 commented 4 years ago

I guess updating the branch bug/128 doesn't work well.

Would you please create another mix new test project, set mix.exs to add the branch bug/128 of Pelemay, set lib/test.ex, and iex -S mix, again?

thantez commented 4 years ago

I updated Manjaro with sudo pacman -Syu and there is a new result! After updating, I ran this commands:

$ mix deps.clean
$ mix deps.clean --all
$ mix deps.update --all
$ iex -S mix
# compiled successfully.
$ la _build/dev/lib/pelemay/build
total 13K
-rwxrwxrwx 1 root root  423 May 12 17:05 basic.d*
-rwxrwxrwx 1 root root 1.1K May 12 17:05 kernels.d*
-rwxrwxrwx 1 root root  731 May 12 17:05 libnifelixirtest.d*
-rwxrwxrwx 1 root root 1.1K May 12 17:05 libnifelixirtest.mk*
-rwxrwxrwx 1 root root  511 May 12 17:05 lsm.d*

Maybe mix deps.update --all works.

thantez commented 4 years ago

contents

$ cat basic.d
../obj/basic.o: \
  /run/media/thantez/Projects/elixir/test/deps/pelemay/lib/pelemay/generator/native/basic.c \
  /usr/lib/erlang/erts-10.7/include/erl_nif.h \
  /usr/lib/erlang/erts-10.7/include/erl_drv_nif.h \
  /usr/lib/erlang/erts-10.7/include/erl_int_sizes_config.h \
  /usr/lib/erlang/erts-10.7/include/erl_nif_api_funcs.h \
  /run/media/thantez/Projects/elixir/test/deps/pelemay/lib/pelemay/generator/native/basic.h
$ cat kernels.d 
../obj/map_elem1_mult_elem1_kernel.o: \
  /run/media/thantez/Projects/elixir/test/_build/dev/lib/pelemay/src/map_elem1_mult_elem1_kernel.c \
  /usr/lib/erlang/erts-10.7/include/erl_nif.h \
  /usr/lib/erlang/erts-10.7/include/erl_drv_nif.h \
  /usr/lib/erlang/erts-10.7/include/erl_int_sizes_config.h \
  /usr/lib/erlang/erts-10.7/include/erl_nif_api_funcs.h \
  /run/media/thantez/Projects/elixir/test/deps/pelemay/lib/pelemay/generator/native/basic.h \
  /run/media/thantez/Projects/elixir/test/_build/dev/lib/pelemay/src/map_elem1_mult_elem1_kernel.h

../obj/map_elem1_mult_elem1_kernel_driver.o: \
  /run/media/thantez/Projects/elixir/test/_build/dev/lib/pelemay/src/map_elem1_mult_elem1_kernel_driver.c \
  /run/media/thantez/Projects/elixir/test/deps/pelemay/lib/pelemay/generator/native/clockcycle.h \
  /usr/lib/erlang/erts-10.7/include/erl_nif.h \
  /usr/lib/erlang/erts-10.7/include/erl_drv_nif.h \
  /usr/lib/erlang/erts-10.7/include/erl_int_sizes_config.h \
  /usr/lib/erlang/erts-10.7/include/erl_nif_api_funcs.h \
  /run/media/thantez/Projects/elixir/test/_build/dev/lib/pelemay/src/map_elem1_mult_elem1_kernel.h
$ cat libnifelixirtest.d
../obj/libnifelixirtest.o: \
  /run/media/thantez/Projects/elixir/test/_build/dev/lib/pelemay/src/libnifelixirtest.c \
  /usr/lib/erlang/erts-10.7/include/erl_nif.h \
  /usr/lib/erlang/erts-10.7/include/erl_drv_nif.h \
  /usr/lib/erlang/erts-10.7/include/erl_int_sizes_config.h \
  /usr/lib/erlang/erts-10.7/include/erl_nif_api_funcs.h \
  /run/media/thantez/Projects/elixir/test/deps/pelemay/lib/pelemay/generator/native/basic.h \
  /run/media/thantez/Projects/elixir/test/deps/pelemay/lib/pelemay/generator/native/lsm.h \
  /run/media/thantez/Projects/elixir/test/_build/dev/lib/pelemay/src/map_elem1_mult_elem1_kernel.h \
  /run/media/thantez/Projects/elixir/test/_build/dev/lib/pelemay/src/map_elem1_mult_elem1_kernel_driver.h
$ cat libnifelixirtest.mk 
.phony: all clean

CC = /usr/bin/clang
CFLAGS = 
ERL_CFLAGS = 
ERL_EI_INCLUDE_DIR = 
ERL_EI_LIBDIR = 
ERL_LDFLAGS = 
LDFLAGS = 

CFLAGS += -Ofast -g -ansi -pedantic -I/run/media/thantez/Projects/elixir/test/deps/pelemay/lib/pelemay/generator/native
ifdef CROSSCOMPILE
  CFLAGS += $(ERL_CFLAGS)
  LDFLAGS += $(ERL_LDFLAGS)
else
  CFLAGS += -I/usr/lib/erlang/erts-10.7/include
endif
CFLAGS += -std=c11 -Wno-unused-function

ifeq ($(OS), Windows_NT)
  TARGET = ../priv/libnifelixirtest.dll
else
  TARGET = ../priv/libnifelixirtest.so
  CFLAGS += -fPIC
  ifeq ($(shell uname),Darwin)
    ifndef CROSSCOMPILE
      LDFLAGS += -dynamiclib -undefined dynamic_lookup
    endif
  endif
endif

OBJS = ../obj/libnifelixirtest.o \
  ../obj/map_elem1_mult_elem1_kernel.o \
  ../obj/map_elem1_mult_elem1_kernel_driver.o \
  ../obj/basic.o \
  ../obj/lsm.o

all: $(TARGET)

$(TARGET): $(OBJS)
        $(CC) $^ -o $@ -shared $(LDFLAGS)

include $(shell ls *.d 2>/dev/null)

%.o %.c:
        $(CC) -S $< -o $*.s $(CFLAGS)
        $(CC) -c $< -o $@ $(CFLAGS)

clean:
        $(RM) $(TARGET) $(OBJS)
$ cat lsm.d              
../obj/lsm.o: \
  /run/media/thantez/Projects/elixir/test/deps/pelemay/lib/pelemay/generator/native/lsm.c \
  /run/media/thantez/Projects/elixir/test/deps/pelemay/lib/pelemay/generator/native/lsm.h \
  /usr/lib/erlang/erts-10.7/include/erl_nif.h \
  /usr/lib/erlang/erts-10.7/include/erl_drv_nif.h \
  /usr/lib/erlang/erts-10.7/include/erl_int_sizes_config.h \
  /usr/lib/erlang/erts-10.7/include/erl_nif_api_funcs.h \
  /run/media/thantez/Projects/elixir/test/deps/pelemay/lib/pelemay/generator/native/basic.h
thantez commented 4 years ago

I backed to {:pelemay, "~> 0.0.11"} and compiled again and worked. Now I run Test.map_square(1..100000000) inside iex and it returns correct result. I don't understand why this time it works but I see in libnifelixirtest.mk we have this line:

CC = /usr/bin/clang

That I think it will say I set cc to clang 🤔 but I'm not! There is my cc's output:

$ cc -v
Using built-in specs.
COLLECT_GCC=cc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-pkgversion='Arch Linux 9.3.0-1' --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++,d --enable-shared --enable-threads=posix --with-system-zlib --with-isl --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-gnu-indirect-function --enable-multilib --disable-werror --enable-checking=release --enable-default-pie --enable-default-ssp --enable-cet=auto gdc_include_dir=/usr/include/dlang/gdc
Thread model: posix
gcc version 9.3.0 (Arch Linux 9.3.0-1)

PS: when I started this issue, I had a problem with clang that had caused an error, so I set cc to gcc that had been worked correctly. When I ran clang -v I saw clang: error while loading shared libraries: libLLVM-10.so error. Now that error solved and this is result of clang -v:

$ clang -v
clang version 10.0.0 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/9.3.0
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/9.3.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0
Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/9.3.0
Selected GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/9.3.0
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
zacky1972 commented 4 years ago

I see.

I guess you'll get the result CC=/usr/bin/gcc (or somewhere right path) written in the makefile if you set the environment variable CC by the command export CC=gcc before run mix clean; mix compile.

zacky1972 commented 4 years ago

I've just released Pelemay 0.0.12, which includes this bug fix. Would you try it! @thantez

thantez commented 4 years ago

Yes, fixed any problem. It works well. Thank you.

zacky1972 commented 4 years ago

You're welcome. Thank you for your corporation!