search

SRI-CSL / whole-program-llvm

A toolkit for building whole-program LLVM bitcode files
MIT License
72 stars 11 forks source link

extract-bc mistakenly truncating paths on macOS #28

Closed carlocab closed 3 years ago

carlocab commented 3 years ago

I'm running macOS Big Sur, and have installed wllvm using pip into a virtual environment. I installed llvm 11.1 and python 3.9.2 using Homebrew, and have set LLVM_COMPILER=clang. llvm-link and llvm-ar are in my PATH.

I tried to compile a small "hello, world" program and extract the bitcode by running

wllvm hello.c -o hello
extract-bc hello

This produces the following error:

❯ extract-bc ./hello
llvm-link: ders/kb/pf4wbfc512z47l56g5072l900000gn/T/tmp.Kke5QqgU49/.hello.o.bc: error: Could not open input file: No such file or directory
llvm-link: error:  loading file 'ders/kb/pf4wbfc512z47l56g5072l900000gn/T/tmp.Kke5QqgU49/.hello.o.bc'

The issue seems to be that the path to .hello.o.bc is getting truncated for some reason:

❯ pwd
/var/folders/kb/pf4wbfc512z47l56g5072l900000gn/T/tmp.Kke5QqgU49
❯ ls -la
.rw-r--r-- 1.0k carlocab 28 Feb 12:20 .hello.o
.rw-r--r-- 3.0k carlocab 28 Feb 12:20 .hello.o.bc
.rwxr-xr-x  65k carlocab 28 Feb 12:20 hello
.rw-r--r--   80 carlocab 28 Feb 12:20 hello.c
drwxr-xr-x    - carlocab 28 Feb 12:19 venv

Am I doing something wrong here?

I'm getting the exact same error with the built-in macOS python (3.8.2), in case that helps pin down the problem.

I also tried this example in your README, but the configure script complained that the C compiler wasn't working. It gives me the same error even if I set WLLVM_CONFIGURE_ONLY=1.

ianamason commented 3 years ago

I am pretty busy at the moment. Might take me a few days before I get time to look at this.

Did you try gllvm?

What was your cwd when you did the compile?

What does wllvm-sanity-checker say?

What did clang do with the wrapper?

carlocab commented 3 years ago

Thanks for getting back to me.

I am pretty busy at the moment. Might take me a few days before I get time to look at this.

No problem; please take all the time you need.

Did you try gllvm?

I had not, but I was unable to install gllvm when I tried:

❯ go get github.com/SRI-CSL/gllvm/cmd/...
go: downloading github.com/SRI-CSL/gllvm v1.3.0
github.com/SRI-CSL/gllvm/cmd/gclang++: malformed import path "github.com/SRI-CSL/gllvm/cmd/gclang++": invalid char '+'

What was your cwd when you did the compile?

The same as when I did extract-bc: /var/folders/kb/pf4wbfc512z47l56g5072l900000gn/T/tmp.Kke5QqgU49

What does wllvm-sanity-checker say?

❯ wllvm-sanity-checker
wllvm version: 1.2.9
wllvm released: February 20 2020

Logging output to standard error.
Logging level not set, defaulting to WARNING.

We are using clang.

The C compiler clang is:

        clang version 11.1.0

The C++ compiler clang++ is:

        clang version 11.1.0

The bitcode linker llvm-link is:

        LLVM version 11.1.0

The bitcode archiver llvm-ar is:

        LLVM version 11.1.0

Not using a bitcode store.

What did clang do with the wrapper?

Not sure what you mean by this.

carlocab commented 3 years ago

I built gllvm from source using your release tarball instead of by go get. I tried it out, and it works for me; thanks.

It would be nice to get wllvm working, though.

ianamason commented 3 years ago

I will look into wllvm when I get a chance. I built a new release just the other day, it build apache and musl without any problems, but ....

The go thing is annoying too, that complaint is new, sigh. What version of go were you using?

carlocab commented 3 years ago

I'm using go1.16, which I should mention takes some coaxing into building gllvm, because it expects builds to use Go modules now: https://blog.golang.org/go116-module-changes

You may wish to look into using that, because I think anything that doesn't use modules will just not build with go1.17.

ianamason commented 3 years ago

Great, go going the way of python.

ianamason commented 3 years ago

OK so I looked into this, and it appears in the "evolution" of python, the things we get out of the section have morphed from strings to bytes. So I put a decode in there, and this make things look like strings. Maybe that was why they were being truncated? Who knows. I can't reproduce your issue. But:

CSL-CAS7352:whole-program-llvm e27658$ pwd
/Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm
CSL-CAS7352:whole-program-llvm e27658$ cd test/test_files/
CSL-CAS7352:test_files e27658$ CC=wllvm make
wllvm foo.c bar.c baz.c main.c -o main
wllvm foo.c -c
wllvm bar.c -c
wllvm baz.c -c
wllvm main.c -c
ar cr libfoo.a foo.o bar.o baz.o
ranlib libfoo.a
CSL-CAS7352:test_files e27658$ extract-bc -m main
WARNING:Manifest written to /Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/main.llvm.manifest
CSL-CAS7352:test_files e27658$ more main.llvm.manifest
/Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/.foo.o.bc
/Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/.bar.o.bc
/Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/.baz.o.bc
/Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/.main.o.bc
CSL-CAS7352:test_files e27658$ extract-bc -m libfoo.a
WARNING:Manifest written to /Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/libfoo.a.llvm.manifest
WARNING:Manifest written to /Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/libfoo.a.llvm.manifest
Writing output to /Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/libfoo.bca
/usr/local/opt/llvm/bin/llvm-ar: warning: creating /Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/libfoo.bca
Generated LLVM bitcode archive /Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/libfoo.bca
CSL-CAS7352:test_files e27658$ more libfoo.a.llvm.manifest
/Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/.bar.o.bc
/Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/.foo.o.bc
/Users/e27658/Repositories/GitHub/SRI-CSL/whole-program-llvm/test/test_files/.baz.o.bc
CSL-CAS7352:test_files e27658$

My setup looks similar to yours:

CSL-CAS7352:test_files e27658$ wllvm-sanity-checker
wllvm version: 1.2.9
wllvm released: February 20 2020

Logging output to standard error.
Logging level not set, defaulting to WARNING.

We are using clang.

The C compiler /usr/local/opt/llvm/bin/clang is:

    clang version 11.1.0

The C++ compiler /usr/local/opt/llvm/bin/clang++ is:

    clang version 11.1.0

The bitcode linker /usr/local/opt/llvm/bin/llvm-link is:

    LLVM version 11.1.0

The bitcode archiver /usr/local/opt/llvm/bin/llvm-ar is:

    LLVM version 11.1.0

Not using a bitcode store.

The fix is in eb6e0f0744769761d08ddc111a4d650df16212a7

carlocab commented 3 years ago

I tried cloning this repo and reproducing what you just did:

❯ wllvm-sanity-checker
wllvm version: 1.2.9
wllvm released: February 20 2020

Logging output to standard error.
Logging level not set, defaulting to WARNING.

We are using clang.

The C compiler clang is:

        clang version 11.1.0

The C++ compiler clang++ is:

        clang version 11.1.0

The bitcode linker llvm-link is:

        LLVM version 11.1.0

The bitcode archiver llvm-ar is:

        LLVM version 11.1.0

Not using a bitcode store.

❯ pwd
/tmp/whole-program-llvm
❯ cd test/test_files
❯ CC=wllvm make
wllvm foo.c bar.c baz.c main.c -o main
wllvm foo.c -c
wllvm bar.c -c
wllvm baz.c -c
wllvm main.c -c
ar cr libfoo.a foo.o bar.o baz.o
ranlib libfoo.a

❯ extract-bc -m main
WARNING:Manifest written to /private/tmp/whole-program-llvm/test/test_files/main.llvm.manifest
llvm-link: le-program-llvm/test/test_files/.foo.o.bc: error: Could not open input file: No such file or directory
llvm-link: error:  loading file 'le-program-llvm/test/test_files/.foo.o.bc'

I installed wllvm after cloning your repo with

❯ python3 -m venv venv
❯ source venv/bin/activate
❯ pip install -e .

I'll try to find time to poke around the changes you made to see if that fixes anything.

carlocab commented 3 years ago

Which version of Python are you using?

ianamason commented 3 years ago

I am using 3.8.8 but hopefully that should not matter. What did the manifest contain?

carlocab commented 3 years ago

The manifest contains

le-program-llvm/test/test_files/.foo.o.bc
/private/tmp/whole-program-llvm/test/test_files/.bar.o.bc
/private/tmp/whole-program-llvm/test/test_files/.baz.o.bc
/private/tmp/whole-program-llvm/test/test_files/.main.o.bc
ianamason commented 3 years ago

So just the first is broken. That is very ODD. Can you zoom @carlocab?

ianamason commented 3 years ago

Let me know when a good time to help you debug this is @carlocab. I can guide you through so we can see where the truncation is happening.

carlocab commented 3 years ago

Thanks for going out of your way for this. It's quite late now, and I'm a bit busy in the next two days. Do you have any time over the weekend, or perhaps next week? My timezone is GMT.

ianamason commented 3 years ago

Yes. Weekend is fine.

carlocab commented 3 years ago

Excellent, thank you. I will send you an email.

ianamason commented 3 years ago

If you @carlocab have time today could you do your hello world example with logging on and out to a file?

export WLLVM_OUTPUT_LEVEL=DEBUG
export WLLVM_OUTPUT_FILE=/tmp/wllvm.log
wllvm hello.c -o hello

Don't think I need to see the extract-bc, but does get-bc give you the correct results? Grab the log file before you do try out the get-bc.

carlocab commented 3 years ago

Here's the log file requested: wllvm.log

get-bc does seem to correctly generate hello.bc. I should have time to speak in about an hour if you're able.

ianamason commented 3 years ago

So if get-bc extracts the bitcode, then there is nothing wrong with the compile phase, it is simply that extracting the section is not working properly in the python code. Which narrows it down substantially.

I am going to dash out for an hour, so I will touch base when I get back.

ianamason commented 3 years ago

OK back. So lets look at how we get the section out:

otool -X -s __WLLVM __llvm_bc hello

In my case (working with the test_file example) I get:

test_files e27658$ otool -X -s __WLLVM __llvm_bc main.o
Contents of (__WLLVM,__llvm_bc) section
0000000000000070    2f 55 73 65 72 73 2f 65 32 37 36 35 38 2f 52 65
0000000000000080    70 6f 73 69 74 6f 72 69 65 73 2f 47 69 74 48 75
0000000000000090    62 2f 53 52 49 2d 43 53 4c 2f 77 68 6f 6c 65 2d
00000000000000a0    70 72 6f 67 72 61 6d 2d 6c 6c 76 6d 2f 74 65 73
00000000000000b0    74 2f 74 65 73 74 5f 66 69 6c 65 73 2f 2e 6d 61
00000000000000c0    69 6e 2e 6f 2e 62 63 0a

FYI

CSL-CAS7352:test_files e27658$ otool --version
llvm-otool(1): Apple Inc. version cctools-949.0.1
Apple LLVM version 11.0.0 (clang-1100.0.33.17)
  Optimized build.
  Default target: x86_64-apple-darwin18.7.0
  Host CPU: haswell

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_be - AArch64 (big endian)
    arm        - ARM
    arm64      - ARM64 (little endian)
    armeb      - ARM (big endian)
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
carlocab commented 3 years ago

Not sure this looks right:

❯ otool -X -s __WLLVM __llvm_bc hello
000000010000c000        2f 70 72 69 76 61 74 65 2f 74 6d 70 2f 68 65 6c
000000010000c010        6c 6f 2f 2e 68 65 6c 6c 6f 2e 6f 2e 62 63 0a
❯ otool --version
llvm-otool(1): Apple Inc. version cctools-977.1
otool(1): Apple Inc. version cctools-977.1
disassmbler: LLVM version 12.0.0, (clang-1200.0.32.29)

By the way, if you'd rather chat on Zoom, feel free to mail me an invitation. Or I can set up a meeting as well if you prefer.

ianamason commented 3 years ago

OK thanks for zooming @carlocab. Conclusion: otool version 12 differs from previous otools in that it omits the

Contents of (__WLLVM,__llvm_bc) section

line in it's output, so dropping the first line of output (which is what happens) is not a good idea. Will have to investigate the best way to fix this.

ianamason commented 3 years ago

I published a new pip package 1.3.0 with this fix.

carlocab commented 3 years ago

Amazing! Thanks very much. I'll test it and give you an update.

carlocab commented 3 years ago

Works great. Thanks again.