Improve shared linking support in LLVM

[GoogleCodeExporter]

python-dev wants us to link shared against LLVM, rather than the current 
static link. This will simplify things for downstream packagers and 
maintainers.

http://llvm.org/PR3201 seems to be the appropriate upstream bug.

Nick, I recall you saying you had an idea of what would need to change for 
this to happen; feel free to bounce this to me or Jeffrey if you don't have 
cycles for it.

Original issue reported on code.google.com by collinw on 27 Jan 2010 at 7:12

• Blocking: #131

[GoogleCodeExporter]

Yep, I've done some of the work needed to make this happen already. This is
http://llvm.org/PR3201 .

Original comment by nlewy...@google.com on 27 Jan 2010 at 9:39

[GoogleCodeExporter]

Do you have an idea of how much more needs to be done? Is it just a matter of 
reviewing/committing Dan's patches from http://llvm.org/PR3201?

Original comment by collinw on 27 Jan 2010 at 9:50

[GoogleCodeExporter]

Those patches grew waaay too big and have likely bitrotted by now. Perhaps it 
would 
help if I summarised the conclusions that could be drawn from the effort? If 
so, I'll try 
to write something up tomorrow.

Original comment by danchr on 27 Jan 2010 at 10:41

[GoogleCodeExporter]

That would be great, Dan, thanks!

Original comment by collinw on 27 Jan 2010 at 10:42

[GoogleCodeExporter]

Okay, so basically, there are three approaches to building LLVM as 
dynamic/shared libraries:

1) Build the same library components as now, but as dynamic libraries.
2) Build all of LLVM (and all of Clang) as one library each.
3) Reorganise the LLVM (and possibly Clang) directory hierarchies to be more 
hierarchic, combining 
for example libLLVMSystem & libLLVMSupport into one.

At first, I took the easy path, option #1. However, this results in two issue: 
1) Rebasing can cause 
massive increases in startup time. 2) Extensive calls across library boundaries 
— meaning stubs will 
be used — causes a major slowdown. I managed to remedy the first issue by 
rebasing on Darwin, but 
the second one is a dealbreaker; IIRC the slowdown was in the order of 30% in 
Clang. Additionally, I 
believe there were be problems with the LLVM command library.

Then, I combined all LLVM components[1] into one library, and all Clang 
components into another. 
However, this requires modifying some source files; for example, the execution 
engine defines a few 
command line options — such as ‘mcpu’ & ‘march’ — which cause a 
conflict when linked into the 
same binary. I just removed them, but a proper fix would probably be to move 
the option definitions 
into ‘lli’ and have the JIT constructor take them as arguments.

The single LLVM and single Clang libraries are truly huge, and there were some 
problems with how I 
built them. As TableGen links against libLLVMSystem and libLLVMSupport, they 
must be compiled 
before it. Hence, I compiled them twice; once as static archives which were 
used for TableGen, and 
then once again for ‘libLLVM.dylib’. TableGen would then be relinked on 
installation to have it linked 
against ‘libLLVM.dylib’. (On non-Darwin systems, this would apply to all 
other binaries as well, as I 
didn't know how to set the install paths of them.) As another possible 
annoyance, compiling LLVM as 
both static and dynamic libraries caused all sources to be compiled twice, as 
the two forms of library 
used different locations for intermediate objects.

It seems to me that the third option would be preferable overall, and also mean 
that static and 
dynamic builds wouldn't be so entirely different. Unfortunately, I don't have 
sufficient understanding 
of the LLVM sources to determine a sensible hierarchy. In addition, compiling 
and testing LLVM is 
quite demanding on hardware resources, and even though I'm no longer stuck on a 
G4, it's still quite 
tiring in the long run…

I've attached the patches I had against LLVM 2.6 & Clang 1.0. Please note that 
these aren't cleaned up 
in any way whatsoever ;-) They contain unrelated changes, such as a drastic 
overhaul of the 
Makefiles, support for colour output from them, and a merger of the individual 
target libraries into 
the Target library. Also note that the patches don't include ‘configure’, 
so it'd have to be 
regenerated.

[1] With the exception of the LLVMC compiler driver, which I find weird.

Original comment by danchr on 30 Jan 2010 at 12:21

Attachments:

• clang-dynamic-build.diff
• llvm-dynamic-build.diff

[GoogleCodeExporter]

Reassigning to Jeffrey, since he's actively working on this in upstream LLVM.

Original comment by collinw on 9 Feb 2010 at 10:33

[GoogleCodeExporter]

Jeffrey, can you update this with the current status of shared linking?

Original comment by collinw on 21 Feb 2010 at 6:07

[GoogleCodeExporter]

I committed the shared library patch to LLVM, but it broke two platforms and 
was rolled 
back. I should be able to roll that forward in the next couple days. Then we 
can merge 
LLVM in and try linking it shared.

Original comment by jyass...@gmail.com on 21 Feb 2010 at 6:30

[GoogleCodeExporter]

This is done, right?

Original comment by collinw on 22 Mar 2010 at 8:31

• Changed state: Started