search

twitter-archive / commons

Twitter common libraries for python and the JVM (deprecated)
http://twitter.github.com/commons
Other
2.1k stars 565 forks source link

Build configuration missing. #1

Open sheki opened 13 years ago

sheki commented 13 years ago

Is any configuration / build scripts /ant OR maven going to be released for building these tools any time soon? Very much interested in looking at how Netty is used with thrift.

Thanks for open sourcing.

jsirois commented 13 years ago

We need to update the README and will be blogging about pants and sapling by ~end of May, but the build already works - but is custom (uses ant/ivy behind the scenes). You need python 2.6 and ant 1.8+ installed and then, from the root of the repo, you run: $ ./pants tests/java/com/twitter/common:all-tests!

This will likely fail on the 1st run and explain steps for installing a python templating library (mako) dependency. After installing mako you should be good to go.

For example to compile all code and run all tests. $ ./pants tests/java/com/twitter/common:all-tests!

You can learn a bit more using: $ ./pants help Tjis will list subcommands available.

$ ./pants help [subcommand] This will give even more info about a particular subcommand.

In short - the repo is a monolithic source tree, but it produces many jars - vs a typical gradle/buildr/maven multi-module project where there would be a top-level project directory for each artifact-producing project. Currently pants provides 40 thin jars from its one source tree: $ ./pants list --provides --provides-columns=artifact_id com.twitter.common#application-action com.twitter.common#application com.twitter.common#application-http com.twitter.common#application-modules com.twitter.common#args com.twitter.common#base com.twitter.common#collections com.twitter.common#inject com.twitter.common#inject-timed com.twitter.common#io com.twitter.common#io-thrift com.twitter.common#logging com.twitter.common#dynamic-host-set com.twitter.common#net-util com.twitter.common#net-pool com.twitter.common#net-http com.twitter.common#net-http-handlers-text com.twitter.common#net-http-handlers-string-template com.twitter.common#net-http-handlers com.twitter.common#net-http-handlers-thrift com.twitter.common#net-http-handlers-time-series com.twitter.common#quantity com.twitter#serverset com.twitter.common#stats-provider com.twitter.common#stats com.twitter.common#stats-jvm com.twitter.common#stats-time-series com.twitter.common#stats-util com.twitter.common#thrift com.twitter.common#jdk-logging com.twitter.common#util com.twitter.common#util-system-mocks com.twitter.common#util-sampler com.twitter.common#util-testing com.twitter.common#util-caching com.twitter.common#zookeeper com.twitter.common#zookeeper-testing com.twitter.common#stats-ostrich-adapter com.twitter.common#service-thrift org.apache.scribe#scribe_client

And some of these jars are very thin in and of themselves, leveraging internal and external jars: $ ./pants files src/java/com/twitter/common/quantity com/twitter/common/quantity/Amount.java com/twitter/common/quantity/Data.java com/twitter/common/quantity/Time.java com/twitter/common/quantity/Unit.java

$ ./pants depmap -m src/java/com/twitter/common/quantity com.twitter.common-quantity com.twitter.common-collections commons-lang-commons-lang-2.4 com.google.guava-guava-r08 com.google.code.findbugs-jsr305-1.3.9

The goal is to make it very easy to craft lean targets that export minimal APIs and compose these.

caniszczyk commented 13 years ago

Any reason Maven isn't used here? It's pretty much the standard build system for open source java stuff.

jsirois commented 13 years ago

Yes - scalability on 2 fronts: 1.) What you see is a small slice of a much larger repo (using saplling: http://pypi.python.org/pypi/sapling https://github.com/jsirois/sapling) with >20 leaf projects and some common_internal libraries they share that are specific to internal twitter infrastucture. In total, using a maven-like build system, we'd have > 120 top level projects in a classic maven multi-project build. We were motivated by a strong desire to maintain a single source tree allowing for easy refactors across the global code base and easy discoverability of common code amongst 100s of developers scaling towards 1000s. 2.) We needed to support several languages 1st class. In particular - in this open source split you can see and use the excellent python support

The artifacts this build system exports are fully maven and ivy compatible, but a big shortcoming is that all the build documentation is still on internal wikis. This will be remedied at some point.

fern commented 13 years ago

Any way to vote to change it to use maven? :)

ewhauser commented 13 years ago

Have you actually tried to compile and build with pants? It's exponentially faster than a Maven build would be with 120 projects in a multi-project build.

Stop complaining because you don't want to take the time to understand their tool.

jsirois commented 13 years ago

You could vote to change to maven or anything else if it fits the requirements described. In particular, besides the shortcoming of scant (non-existant!) docs right now on the public wiki for operating the pants build tool, what additional pain points are you encountering?

caniszczyk commented 13 years ago

My pain point is that if you want people to contribute, use a standard build technology. I understand I'm saying this as an outsider though so take it like you wish.

Maven3 is quite a bit faster than maven2 these days, even supports OSGi. As a person who spent migrating the majority of eclipse.org's infrastructure to maven3 it doesn't suck that bad anymore. It has a large ecosystem where you can easily integrate with tools like Hudson/Jenkins, code coverage etc... It would be nice if this would be investigated in the future again.

jsirois commented 13 years ago

It does not support a single source tree though in any reasonable way afaict and does not support python. I understand the pain point on contribution though and its on us to update the wiki with good docs. That said - if you are an eclipse or intellij user, the following works well for development: $ ./pants (idea|eclipse) tests/java/com/twitter/common:all -n common -c See pants help (eclipse|idea) for more info.

cscotta commented 12 years ago

Hi John,

I noticed about six months ago you mentioned plans to add documentation for your in-house build tool to the wiki, but I'm still not seeing anything here.

Probably just going to fork and start mavenizing a few things here.

Thanks,

– Scott

jsirois commented 12 years ago

Sorry for the large delay.

There will be a push this week with a much improved pants (no longer uses ant in the backend, uses compiler daemons for java and scala...). I'll be updating the wiki at that time with our internal pants wiki.

In the meantime the following may be helpful to you: