Closed GoogleCodeExporter closed 9 years ago
Hello,
Here's the rationale: before plv8js and non-ancient versions of v8 are picked
up by most operating systems (for example: I could not find a 'd8' binary in
Lucid's repositories) and for the sake of those experimenting with new versions
of plv8 and v8 it's really handy to have a self-contained binary that contains
the entire system.
So, here's what I did, although it needs some polishing:
1) Made the 'd8' command more explicit, instead of relying on PATH
diff --git a/Makefile b/Makefile
index f03c0df..6c265fe 100644
--- a/Makefile
+++ b/Makefile
@@ -3,6 +3,8 @@ V8DIR = ../v8
# set your custom C++ compler
CUSTOM_CC = g++
+D8_CMD = $(V8DIR)/out/x64.release/d8
+
JSS = coffee-script.js
# .cc created from .js
JSCS = $(JSS:.js=.cc)
@@ -14,11 +16,14 @@ EXTVER = 1.1.0
DATA = plv8.control plv8--$(EXTVER).sql
DATA_built = plv8.sql
REGRESS = init-extension plv8 inline json startup_pre startup varparam
2) Changed the linking to point to a couple .as that are built by building v8.
It's important to include CXX='g++ -fPIC' make x64 when building v8. Note that
there are both 'snapshot' and 'non-snapshot' versions of v8, the tradeoff being
binary size vs. startup time. I chose ostensibly larger binary size and lower
startup time.
-SHLIB_LINK := $(SHLIB_LINK) -lv8
+
+V8STATIC = $(V8DIR)/out/x64.release/obj.target/tools/gyp
+
+SHLIB_LINK := $(SHLIB_LINK) $(V8STATIC)/libv8_base.a
$(V8STATIC)/libv8_snapshot.a
META_VER := $(shell v8 -e 'print(JSON.parse(read("META.json")).version)' 2>/dev/null)
ifndef META_VER
3) (More d8 stuff)
-META_VER := $(shell d8 -e 'print(JSON.parse(read("META.json")).version)')
+META_VER := $(shell $(D8_CMD) -e
'print(JSON.parse(read("META.json")).version)')
endif
CCFLAGS := $(filter-out -Wmissing-prototypes, $(CFLAGS))
CCFLAGS := $(filter-out -Wdeclaration-after-statement, $(CCFLAGS))
This emits a plv8 shared object that has exactly the v8 in ../v8.
I think it'd be nice if this could be made a one-step process, for both debug
and release builds -- it'd be easier for people to play with new versions and
make sure their debugging symbols match up. Full-blown operating system
projects like Debian may or may not choose to dynamically link plv8 to libv8
and QA that integration, making a tradeoff on if the package maintainer wants
to hurry out a plv8 release every time libv8 has a major problem.
Original comment by drfar...@gmail.com
on 20 Sep 2012 at 9:46
One of my concerns to do static linking is I don't have a good sense around
platform portability. Does plv8 binary work in all Linux platform once you
link it to specific build of v8 statically? I guess we should support at least
x86 and x64 (until arm server becomes popular!) versions of pre-built binary.
In addition, we plv8js currently don't have QA or CI at all. I'm just freaking
out...
d8 stuff is definitely something we can improve. I assumed everyone who builds
plv8 should have built v8 from source thus he must have v8 or d8, but it seems
growing number of people start using it and someone installs v8 from source
while other people install it from binary package. At one time I guessed we
should provide configure script because we have such dependency pain like v8.
Fortunately, pgxnclient now supports configure before make, so we can try it.
For the ease of use, I was thinking that is the packaging matter. I know of
only RPM, but it resolves dependency if it's separated. And as far as we link
dynamically users can choose their favorite version of v8. Of course you want
different versions of v8 installed in your system and try one with plv8 while
the other is used by, say, node.js. Then the problem should be LD_LIBRARY_PATH
when you kick off postgres. Aside from packaging issue, I don't see the
benefit of static linking yet.
Again, I agree with the build issue so I'll fix that. But maybe we can build
our buildfarm and packaging system for nightly build package for all supported
platform or something.
Original comment by umi.tan...@gmail.com
on 21 Sep 2012 at 6:22
So I concluded the issue can be broken down into 3.
1) Distribution problem
We don't have any distribution package system at the moment, and probably this is what we should do next. But it seems this issue is slightly different from others and we still need to keep dynamic link when building from source.
2) Support static link when building from source
Regardless whether we will distribute static link or not, we should need an easy way to build static linking. We could use configure-like script, and I at one time thought about it, but Makefile can handle it apparently.
3) V8DIR and d8 stuff
V8DIR is definitely a bad coding from my old age when this is quite personal. It's not portable at all and let users edit Makefile makes no sense. Also, v8/d8 were used to parse META.json, but it's only a development issue that keeping a single version string in one place.
I somehow tried to solve 2 and 3 in a separated repository, named staticlink.
Can you try it and see if it works for you?
Original comment by umi.tan...@gmail.com
on 24 Sep 2012 at 8:43
I put this change on master. Feel free to claim if it doesn't work for you.
Original comment by umi.tan...@gmail.com
on 26 Sep 2012 at 9:40
Hello,
Sorry for the delay. I think fiddling around with LD_LIBRARY_PATH and so on
is, as of yet, not worth it. I have seen so many knowledgable people have
completely inane issues that waste their time that involve dynamic linking
problems. This is because things can kind of work until they don't and the
error messages in such cases are opaque (C++ mangled symbols spat out by ld,
for example) that I think it's a recipe for disaster. It's counter-intuitively
worsened by greater availability of versions of a library, because a "could not
link" error is much better than "symbol is missing" error, if not an outright
crash, and some not always easy to determine later point.
Notably, all these people compile plv8 from source. If apt-get install
postgresql-9.2-plv8 worked great while using dynamic linking, I think there is
no cause for concern, so I think distributors should basically choose what kind
of build they want to use for this.
I'd like to have something like Ruby's bundler for dynamic linking (zeroinstall
may come closest) .sos, but I don't think this is plv8's problem to solve.
What about having plv8 presume that when doing a static build that libv8 is in
a particular path? It can even download it for the user so they can get
started quickly.
Basically, I think the friction to even experiment with plv8 is rather high for
those building from source, and I think static linking cuts through a lot of
those issues rather nicely.
Another interesting pattern I've seen that works reasonably well is to use a
directory of symbolic links, which can be built or assigned by a program. The
dpkg-alternatives system uses a technique like this, and so does rbenv. I have
come to think this approach has some merit.
Original comment by dan...@heroku.com
on 26 Sep 2012 at 10:50
The main use case that I see most (, and this is what I assume people around
postgres mostly expect) is make install easily with pg_config, meaning PGXN
setup. So in my opinion, the default behavior should be dynamic link with
system-installed libv8.so for the first place. Packaging is great, but it's
also great "pgxn install plv8" works fine. I thought you'd agree with this,
but do you still concern on this?
For people build plv8 with v8 from source, yes, we could offer easier way.
That is exactly what I meant by the recent change about V8_STATIC_SNAPSHOT. If
you know where the v8 source and libraries because you have built it, it is
your choice to statically link it. And it is also the packager's choice
whether plv8 should be statically or dynamically linked with v8. Someone would
argue apt-get install postgresql-9.2-plv8 with libv8 dependency is the best
way, while another packager chooses it without dependency.
Ruby bundler is a different story. Gems work fine within RVM, but v8 should be
built in the system and only portable within the same architecture, maybe the
same version of libstdc++?
We may be able to add another make target "v8" in plv8 build which svn co &&
make dependencies && make native library=shared in v8 directory and make plv8
with this static libraries.
Ok, I think I'm missing your point. With V8_SNAPSHOT=1, what is your concern?
Do you agree pgxn-style is the first place? A new make target "v8" may solve
your problem??
Original comment by umi.tan...@gmail.com
on 27 Sep 2012 at 5:58
I had problems building plv8 naively on Lucid with its relatively ancient v8,
because of a missing d8 binary. This feels a bit like a regression (even
though I saw how you use d8 and it looks fine to me in principle) because I
remember doing this before somehow without problems. But practically, Lucid,
CentOS 5, etc users will be in some pretty arcane place if you want to rely on
a 'system installed version', and the obvious alternative to get new ones
(which are not yet integration-tested by the vendor and may break other
packages).
This arcane-v8 situation can be resolved by adding some dodgy repository or
other, hand installed RPMs/debs, self-rolled backports from debian unstable,
'make install', et al. However, those can potentially break the distribution
install if there are introduced ABI compatibility issues or bugs and, being
from a "dodgy" source of questionable helpfulness in the security-update and
integration testing against other dependent packages territory.
You could support all the ancient v8s, but it seems like it'd make your life
more difficult in dimensions might not add as much value to the project,
depending on how much grunting it takes to support and test old (even obsolete
major versions of) v8.
If you elect to not burden yourself supporting ancient v8s as-packaged on a
bunch of distributions, this situation would negate one major advantage of
dynamic linking: 'free' help in managing security updates in dependencies.
The other primary advantage is code image size. Static plv8 is not that big
and I'm not sure how many database servers are going to be loading v8
independently in even more than zero other applications, so I don't see any
major advantage there.
As far I can tell, that eliminates all the advantages of dynamic linking.
I make the bundler comparison (ZeroInstall might be closer, but it's also more
obscure) because it'd be nice to link plv8 against some arbitrary exact version
of libv8. This would be more useful in extension with more than one
dependency, the aggregation of which are large: PostGIS comes to mind, where
libgeos alone is something like 50MB, and may actually be worth saving if one
runs three or four Postgres versions at a time.
I think it might behoove a pgxn release to have a copy of all of v8, and then
statically link, and the developer copy is fine with V8_STATIC_SNAPSHOT.
That's because pgxn can't really be responsible for managing v8 versions and
without the security update angle I don't see the point.
All in all, being able to rely on V8 from the environment requires an
incredibly wide dissemination of v8 with vast stability/compatibility, and as
compared to your other dependencies it is anything but those.
Original comment by dan...@heroku.com
on 29 Sep 2012 at 5:37
Hmm, it's still not clear to me what you are proposing here... I see both ways
have their benefits and I agree bundling v8 is a huge opportunity to mitigate
users installing plv8. They way we could choose are:
1) by default, pgxn-style and provide a way to static link to v8 (as master as
of today)
2) include specific version of v8 source in plv8 source and by default let
users build that v8 when typing make for plv8
3) include pre-built v8 libraries in plv8 source and let users build plv8
linking those v8 static libraries
4) provide plv8 binary package statically linking to v8
5) something else
To me, it seems more difficult to choose #3 than others because we cannot
support complicate matrix of various OSes and versions. It is much harder than
support different versions of v8. I can build mac v8 library in may laptop but
I'm not sure if it works fine in other versions of mac, and I have no idea what
happens about linux if I built v8 static libraries in my AWS CentOS. It sounds
there's no reason to distribute pre-built v8 with plv8 source. If we could do
that, we could also built plv8 and just distribute it.
So what's your proposal?
Original comment by umi.tan...@gmail.com
on 29 Sep 2012 at 6:31
I agree with you about #3. If I understand pgxn style packaging (is that just
how external contribs are also laid out, PGXS=1?), I think #2 is probably
closer, so the build only requires Make, a C++ compiler, Postgres, and some
low-impact dependencies like those.
The problem is v8 itself is pretty enormous, and so it probably doesn't make
much sense to check it into plv8. So, I think similar to the generation of
gram.c in Postgres when generating source releases (as to relieve the bison
dependency) so it should be for some version of the v8 source.
The general idea is that with only enough knowledge to find a C++ compiler and
Postgres that one can emit a useful binary.
Original comment by drfar...@gmail.com
on 6 Oct 2012 at 9:40
That's kind of modern dependency resolution, such as ivy, etc. When the
dependency is not ready, we can download the source from the internet and build
it locally. Then we can build a statically-linked plv8. I still wonder how
much people would like to do that instead of using brew/yum/apt
package-installed version of libv8. I'm also concern about building v8 would
require additional dev tool dependency or something. I'd be annoyed to answer
to the questions like "why can't I build v8 in my environment?" from plv8
users...
Original comment by umi.tan...@gmail.com
on 29 Nov 2012 at 8:32
From what I've seen, the qualitative sense I have is that libv8 iterates very
quickly and distro's versions are comparatively ancient, but of different
stripes of ancient on each distro. If libv8 had complex dependencies I'd
probably be more concerned about problems with people building v8, but it so
happens it only has libstdc++ as a dependency. Basically, all the mistakes
I've seen people make are somehow getting the linking screwed up or, more
recently, reliance on libv8 features that don't exist in old libv8s (ala d8 on
ubuntu lucid).
Basically, I don't think libv8 is pervasive enough and slow-moving enough to
have plv8 be compiled by hand against system libraries be a very attractive
option. If a distributor (debian, redhat) wants to dynamically link a release
of a plv8 package to their version of libv8 and makes sure it works, I see no
problem with them doing so, but I think for a build for environments where no
particular v8 version can be assumed that attempts to dynamically link v8 is
more pain than gain.
Original comment by dan...@heroku.com
on 29 Nov 2012 at 10:40
In 90190dc, I introduced 'make static' which downloads, builds v8 and do static
link all together. Still 'make' is for dynamic link, but I guess it's
reasonable for now considering some people want to have the current behavior
being there.
I tried it only in my mac. Can you try in linux or elsewhere?
Original comment by umi.tan...@gmail.com
on 1 Dec 2012 at 10:25
I tried this on linux. It has some problems, fixed by this:
diff --git a/Makefile.v8 b/Makefile.v8
index f08c7de..1cd8bd2 100644
--- a/Makefile.v8
+++ b/Makefile.v8
@@ -9,7 +9,7 @@
#-----------------------------------------------------------------------------#
AUTOV8_VERSION = 3.14.5
AUTOV8_DIR = build/v8-$(AUTOV8_VERSION)
-AUTOV8_OUT = build/v8-$(AUTOV8_VERSION)/out/native
+AUTOV8_OUT = build/v8-$(AUTOV8_VERSION)/out/native/obj.target/tools/gyp
AUTOV8_LIB = $(AUTOV8_OUT)/libv8_snapshot.a
AUTOV8_STATIC_LIBS = libv8_base.a libv8_snapshot.a
@@ -28,8 +28,8 @@ $(AUTOV8_DIR):
| tar zxf - -C build
$(AUTOV8_LIB): $(AUTOV8_DIR)
- $(MAKE) -C build/v8-$(AUTOV8_VERSION) dependencies
- $(MAKE) -C build/v8-$(AUTOV8_VERSION) native
+ env CXXFLAGS=-fPIC $(MAKE) -C build/v8-$(AUTOV8_VERSION) dependencies
+ env CXXFLAGS=-fPIC $(MAKE) -C build/v8-$(AUTOV8_VERSION) native
include Makefile
Basically, for some reason the build output seems to be in a different place
than originally expected, and GCC demands static libraries be built with -fPIC
if they are going to eventually end up being included in a shared object. This
has the effect of V8 building without complaint but then dying when it attempts
to link into the plv8 .so.
Original comment by dan...@heroku.com
on 26 Feb 2013 at 6:56
Sounds good, although this breaks MacOS build. I hacked the path issue by
808a254. Take a look.
Original comment by umi.tan...@gmail.com
on 28 Feb 2013 at 8:45
This seems to work for me. I have a semi-credible debian package now:
https://github.com/fdr/plv8-static-debian
Original comment by dan...@heroku.com
on 7 Mar 2013 at 12:51
Great. Close this.
Original comment by umi.tan...@gmail.com
on 7 Mar 2013 at 7:45
Original issue reported on code.google.com by
umi.tan...@gmail.com
on 13 Sep 2012 at 6:40