search

IntelLabs / ParallelAccelerator.jl

The ParallelAccelerator package, part of the High Performance Scripting project at Intel Labs
BSD 2-Clause "Simplified" License
294 stars 32 forks source link

Issue about ParallelAccelerator.embed() #105

Closed jihoonseo closed 8 years ago

jihoonseo commented 8 years ago

Hi I want to report a issue about ParallelAccelerator.embed().

My environment:

commit bfe9026889832dd76ff9d307dc5e13b9b61dcf21 Author: Ehsan Totoni ehsan.totoni@intel.com Date: Mon Jul 11 15:42:55 2016 -0700 minor omp include

I wanted to speed up the package load time, so I followed the instruction, and faced an issue.

1st try.

  1. On $(non-root shell), I executed $ julia.
  2. In Julia REPL, I executed julia> importall ParallelAccelerator.
  3. In Julia REPL, I executed julia> ParallelAccelerator.embed(), and got an error:

The specified path to the Julia source code, /usr/bin/../.., does not seem to be a Julia source distribution; base dir not found.

(My suggestion: It would be better if the Julia source distribution directory is specified by default, in the form of environment variable.)

2nd try.

  1. On $(non-root shell), I executed $ julia.
  2. In Julia REPL, I executed julia> importall ParallelAccelerator.
  3. In Julia REPL, I executed julia> ParallelAccelerator.embed("/usr/share/julia"), and got an error:

ERROR: SystemError: opening file /usr/share/julia/base/userimg.jl: Permission denied

(My question is that, is there any way to avoid Permission denied error, while remaining in non-root user and without being root?)

3rd try.

  1. On #(root shell), I executed # julia.
  2. I had to install ParallelAccelerator and switch to master branch again, since ParallelAccelerator is not installed in /root/.julia/v0.4.
  3. In Julia REPL, I executed julia> importall ParallelAccelerator.
  4. In Julia REPL, I executed julia> ParallelAccelerator.embed("/usr/share/julia"), and got an error:

make: *\ No targets specified and no makefile found. Stop. ERROR: failed process: Process(make, ProcessExited(2)) [2]

How can I resolve this stuck?

I would appreciate your comments.

DrTodd13 commented 8 years ago

apt-get install julia doesn't install Julia source does it? I thought apt-get install would only put a Julia binary distribution on the machine. See if you have a julia/base directory. If not then you don't have a source distribution of Julia and embed() won't work. The way embed works is to build a pre-compiled version of ParallelAccelerator into Julia itself and that isn't possible without sources.

Have you built Julia before? Don't forget the "-j 8" flag to make. :)

jihoonseo commented 8 years ago

Hi

1) It seems that the command # apt-get install julia installs Julia(0.4.5) source code.

root@u1604d:/usr/share/julia/base# find . -type f | wc -l
250

root@u1604d:/usr/share/julia/base# du -hs
4.0M    .

root@u1604d:/usr/share/julia/base# ls Makefile -alh
-rw-r--r-- 1 root root 3.9K  4월 18 21:14 Makefile

and the above can be compared with Generic Linux binaries of Julia(0.4.6):

jhseo@u1604d:~/Downloads/julia-2e358ce975/share/julia/base$ find . -type f | wc -l
249

jhseo@u1604d:~/Downloads/julia-2e358ce975/share/julia/base$ du -hs
4.0M    .

jhseo@u1604d:~/Downloads/julia-2e358ce975/share/julia/base$ ls Makefile -alh
-rw-r--r-- 1 jhseo jhseo 4.0K  6월 20 06:42 Makefile

2) I failed to embed() with Generic Linux binaries of Julia(0.4.6):

# miss
julia> ParallelAccelerator.embed("/home/jhseo/Downloads/julia-2e358ce975/share/julia/base")
The specified path to the Julia source code, /home/jhseo/Downloads/julia-2e358ce975/share/julia/base, does not seem to be a Julia source distribution; base dir not found.

# miss
julia> ParallelAccelerator.embed("/home/jhseo/Downloads/julia-2e358ce975")
The specified path to the Julia source code, /home/jhseo/Downloads/julia-2e358ce975, does not seem to be a Julia source distribution; base dir not found.

# hit but got an error
julia> ParallelAccelerator.embed("/home/jhseo/Downloads/julia-2e358ce975/share/julia")
make: *** No targets specified and no makefile found.  Stop.
ERROR: failed process: Process(`make`, ProcessExited(2)) [2]
 in run at ./process.jl:531
 in embed at /home/jhseo/.julia/v0.4/ParallelAccelerator/src/ParallelAccelerator.jl:216

Is there any wrong or missing steps in my try?

DrTodd13 commented 8 years ago

Can you show the contents of /usr/share/julia here? I think your approach 3 above was the closest to working. Since apt-get installs Julia in a system directory you would need to be root to modify it. The problem there seems to be there was no Makefile in /usr/share/julia which one would expect if it were a Julia source distribution.

I really would suggest getting Julia sources from GitHub, compile that version of Julia and then run that version and do the ParallelAccelerator.embed() (which should recompile Julia yet again). We don't typically do it but I suppose there's a make install that loads the version you compile to a system location.

jihoonseo commented 8 years ago

Okay. I will uninstall the Ubuntu package repository version of Julia and get-and-install from Github, referencing this manual: https://github.com/JuliaLang/julia/blob/master/README.md

FYI

root@u1604d:/usr/share/julia# ls -alh
total 56K
drwxr-xr-x   4 root root 4.0K  7월 14 08:50 .
drwxr-xr-x 296 root root  12K  7월 12 14:03 ..
drwxr-xr-x  13 root root  12K  7월 14 08:58 base
-rwxr-xr-x   1 root root 9.3K  6월 20 08:20 build_executable.jl
-rwxr-xr-x   1 root root 7.4K  6월 20 08:20 build_sysimg.jl
-rwxr-xr-x   1 root root 1.9K  6월 20 08:20 julia-config.jl
drwxr-xr-x   8 root root 4.0K  7월 14 08:50 test

root@u1604d:/usr/share/julia# cd base

root@u1604d:/usr/share/julia/base# ls
abstractarray.jl      cartesian.jl      deepcopy.jl       error.jl           fs.jl               intset.jl         loading.jl           multimedia.jl                   pcre.jl        random.jl           rounding.jl     statistics.jl  unicode
abstractarraymath.jl  channels.jl       deprecated.jl     essentials.jl      functors.jl         iobuffer.jl       lock.jl              nofloat_hashing.jl              pkg            range.jl            serialize.jl    stat.jl        unicode.jl
array.jl              char.jl           dft.jl            exports.jl         gmp.jl              io.jl             Makefile             nullable.jl                     pkg.jl         rational.jl         set.jl          stream.jl      userimg.jl
arraymath.jl          c.jl              dict.jl           expr.jl            grisu               iostream.jl       managers.jl          number.jl                       pointer.jl     reducedim.jl        sharedarray.jl  string.jl      util.jl
ascii.jl              client.jl         docs              fastmath.jl        grisu.jl            irrationals.jl    markdown             operators.jl                    poll.jl        reduce.jl           shell.jl        strings        uv_constants.jl
base64.jl             collections.jl    dSFMT.jl          fenv_constants.jl  hashing2.jl         iterator.jl       math.jl              options.jl                      precompile.jl  reflection.jl       show.jl         subarray.jl    version_git.jl
base.jl               combinatorics.jl  dsp.jl            fft                hashing.jl          latex_symbols.jl  meta.jl              ordering.jl                     primes.jl      refpointer.jl       simdloop.jl     sysimg.jl      version_git.jl.backup
bitarray.jl           complex.jl        emoji_symbols.jl  file_constants.jl  i18n.jl             libc.jl           methodshow.jl        osutils.jl                      printf.jl      regex.jl            socket.jl       sysinfo.jl     version.jl
bool.jl               coreimg.jl        Enums.jl          file.jl            inference.jl        libdl.jl          mmap.jl              ParallelAccelerator_userimg.jl  process.jl     REPLCompletions.jl  sort.jl         task.jl
boot.jl               datafmt.jl        env.jl            float16.jl         interactiveutil.jl  linalg            mpfr.jl              parse.jl                        profile.jl     REPL.jl             sparse          Terminals.jl
broadcast.jl          dates             errno_h.jl        floatfuncs.jl      intfuncs.jl         linalg.jl         multidimensional.jl  path.jl                         promotion.jl   replutil.jl         sparse.jl       test.jl
build_h.jl            Dates.jl          errno.jl          float.jl           int.jl              LineEdit.jl       multi.jl             pcre_h.jl                       quadgk.jl      require.jl          special         tuple.jl

You can see that there exists a Makefile in /usr/share/julia/base.

P.S. Just for your information, I uninstalled the Ubuntu package repository version of Julia (which is 0.4.5-3) and tried with the PPA version of Julia (which is 0.4.6)

sudo add-apt-repository ppa:staticfloat/juliareleases
sudo add-apt-repository ppa:staticfloat/julia-deps
sudo apt-get update
sudo apt-get install julia

but got the same unsuccessful result.

DrTodd13 commented 8 years ago

Here are the contents of the top of a Julia 0.5 src distribution.

appveyor.yml
base
contrib
CONTRIBUTING.md
DEBUGGER.md
deps
DISTRIBUTING.md
doc
etc
examples
HISTORY.md
julia
LICENSE.md
Makefile
Make.inc
Make.user
NEWS.md
README.arm.md
README.md
README.windows.md
src
test
ui
usr
usr-staging
VERSION
Windows.inc

So, I don't think you have a source distribution. A makefile in base but no Makefile in the top directory is confusing. Still, I suggest you just get a source installation by cloning from github. Then, build Julia then run embed() again.

jihoonseo commented 8 years ago

Hi As you wrote, I was able to embed() with a source distribution of Julia 0.4.

(You exampled the files in top directory of Julia 0.5, but I failed to Pkg.test("CompilerTools") with Julia 0.5, so I used Julia 0.4.)

And I wrote a small Julia installation guide: https://github.com/jihoonseo/notes/blob/master/installing_Julia_with_sourcecode.md

Thank you.