search

twitter-archive / snowflake

Snowflake is a network service for generating unique ID numbers at high scale with some simple guarantees.
http://twitter.com/
7.68k stars 1.13k forks source link

can not build by maven #8

Closed snageyang closed 10 years ago

snageyang commented 12 years ago

its repository can not access.http://artifactory.local.twitter.com/libs-releases-local.

davegardnerisme commented 12 years ago

Any plans on making it possible to build this project?

ryanking commented 12 years ago

Working on it.

snageyang commented 12 years ago

im try it.thks

billmiller1001 commented 12 years ago

I still can't get it to build. Fails on the maven-finagle-thrift-plugin-0.0.5 from the scala-parent-292.0.0.2.pom. The latest on http://maven.twttr.com is maven-finagle-thrift-plugin-0.0.2

ryanking commented 12 years ago

Still working on it, sorry.

shijinkui commented 12 years ago

i have this problem also. shit

danigiri commented 12 years ago

Guys, got to compile it by specifying a slightly older version of 'maven-finagle-thrift-plugin' which as @billmiller1001 commented, is on the public twitter maven repo.

  <build>
    <plugins>
      <plugin>
        <groupId>com.twitter</groupId>
        <artifactId>maven-finagle-thrift-plugin</artifactId>
        <version>0.0.2</version>
        <configuration>
          <thriftGenerators>
            <thriftGenerator>java</thriftGenerator>
            <thriftGenerator>rb</thriftGenerator>
          </thriftGenerators>
        </configuration>
      </plugin>
    </plugins>
  </build>

Builds (like, with one warning) and tests pass. Can't say if using older version has any real impact.

Hope this helps

danigiri commented 12 years ago

Mmmm.. Have been fiddling a bit. The project builds and tests using maven and the small change, but the generated jar doesn't have the Snowflake server, which does get generated with the ./sbt script. Once sbt is run, tweaking the startup script is all is needed, then works like a charm.

edgurgel commented 12 years ago

Same problem

ryanking commented 12 years ago

This should be working now. let me know if there are any problems.

ryanking commented 12 years ago

sorry, I was wrong :(

dokipen commented 12 years ago

[ERROR] Failed to execute goal on project snowflake: Could not resolve dependencies for project com.twitter.service:snowflake:jar:1.0.1-SNAPSHOT: Could not find artifact org.scala-tools.testing:specs_2.9.2:jar:1.6.9 in Twitter public Maven repo (http://maven.twttr.com) -> [Help 1]

timperrett commented 12 years ago

@ryanking Nope, it still doesn't build, and the SBT build configuration is also missing from the source tree.

benvp commented 12 years ago

Youre still on it? Getting the same error like dokipen.

paulwilton commented 12 years ago

also there is a dependency on com.twitter:util-core:jar:4.0.1 which doesn't exist in the twitter public maven repo

robertdolca commented 12 years ago

com.twitter maven-finagle-thrift-plugin version 0.0.8

I downloaded org.scala-tools.testing:specs_2.9.2:jar:1.6.9 manually.

chka commented 11 years ago

Solved:

cd ~/.m2/repository vi com/twitter/util-logging/5.3.0/util-logging-5.3.0.pom ==> find specs_2.9.2 and change it to specs_2.9.1

vi com/twitter/scala-parent-292/0.0.4/scala-parent-292-0.0.4.pom ==> same here: find specs_2.9.2 and change it to specs_2.9.1

mvn test

caniszczyk commented 11 years ago

bump @ryanking, looks like the build is failing still...

https://travis-ci.org/twitter/snowflake

Looks like the solution is to bump maven-finagle-thrift-plugin but not sure.

yourabi commented 11 years ago

I got the build to succeed. This is a short term work around, the real fix needs to happen in "scala-parent"

1) checkout snowflake 2) edit pom.xml to bump maven-finagle-thrift-plugin to 0.0.9 from 0.0.7 (see pull request #13) 3) mvn test (will fail) but create needed directory structure in $HOME/.m2 4) cd $HOME/.m2/repository/org/scala-tools/testing/specs_2.9.2/1.6.9 and download missing specs_2.9.2-1.6.9.jar from typesafe cache: wget http://repo.typesafe.com/typesafe/scala-tools-releases-cache/org/scala-tools/testing/specs_2.9.2/1.6.9/specs_2.9.2-1.6.9.jar 5) mvn -U test

At this point build should succeed. It it worth pointing out that I haven't had a chance to test this out live yet since I've had to jump through these hoops just to get the build to pass.

satoshi75nakamoto commented 11 years ago

bump @ryanking, looks like the build is failing still... ;-)

bixu commented 11 years ago

Also seeing this.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.177s
[INFO] Finished at: Fri Jun 28 00:35:20 UTC 2013
[INFO] Final Memory: 5M/147M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project snowflake: Could not resolve dependencies for project com.twitter.service:snowflake:jar:1.0.2-SNAPSHOT: Could not find artifact org.scala-tools.testing:specs_2.9.2:jar:1.6.9 in Twitter public Maven repo (http://maven.twttr.com) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
satoshi75nakamoto commented 11 years ago

@bixu — I got it to work with @yourabi's work-around:

I got the build to succeed. This is a short term work around, the real fix needs to happen in "scala-parent"

1) checkout snowflake 2) edit pom.xml to bump maven-finagle-thrift-plugin to 0.0.9 from 0.0.7 (see pull request #13) 3) mvn test (will fail) but create needed directory structure in $HOME/.m2 4) cd $HOME/.m2/repository/org/scala-tools/testing/specs_2.9.2/1.6.9 and download missing specs_2.9.2-1.6.9.jar from typesafe cache: wget http://repo.typesafe.com/typesafe/scala-tools-releases-cache/org/scala-tools/testing/specs_2.9.2/1.6.9/specs_2.9.2-1.6.9.jar 5) mvn -U test

At this point build should succeed.

Have you tried this?

bixu commented 11 years ago

@preillyme, that does indeed work, but I then fail here:

[INFO] compiling thrift files [/root/snowflake/src/main/thrift/Snowflake.thrift, /root/snowflake/src/main/thrift/fb303.thrift, /root/snowflake/src/main/thrift/scribe.thrift] with generators [java, rb]
[INFO] copying /thrift/thrift.null to /var/tmp/thrift8180113951168204754
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.931s
[INFO] Finished at: Fri Jun 28 01:28:12 UTC 2013
[INFO] Final Memory: 5M/85M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.twitter:maven-finagle-thrift-plugin:0.0.9:compile (thrift-sources) on project snowflake: Execution thrift-sources of goal com.twitter:maven-finagle-thrift-plugin:0.0.9:compile failed. NullPointerException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.twitter:maven-finagle-thrift-plugin:0.0.9:compile (thrift-sources) on project snowflake: Execution thrift-sources of goal com.twitter:maven-finagle-thrift-plugin:0.0.9:compile failed.
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution thrift-sources of goal com.twitter:maven-finagle-thrift-plugin:0.0.9:compile failed.
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
        ... 19 more
Caused by: java.lang.NullPointerException
        at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:188)
        at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174)
        at org.codehaus.plexus.util.FileUtils.copyStreamToFile(FileUtils.java:1059)
        at com.twitter.FinagleThriftRunner.getBinary(FinagleThriftRunner.java:53)
        at com.twitter.FinagleThriftRunner.compile(FinagleThriftRunner.java:64)
        at com.twitter.AbstractFinagleThriftMojo.execute(AbstractFinagleThriftMojo.java:186)
        at com.twitter.FinagleThriftCompileMojo.execute(FinagleThriftCompileMojo.java:22)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
        ... 20 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

Is this because I'm on the Illumos (once Solaris) kernel? Or for some other reason? Maven is new to me...

ramayer commented 11 years ago

Not building for me too. Curious if this project's still alive?

hridayeshgupta commented 11 years ago

I am able to build it successfully but I can't get it running. What are the steps to get it running? I could not find anywhere. Tried to run generated jar at target/ but it did not work

satoshi75nakamoto commented 11 years ago

I've asked @jreichhold to take a look at this thread: https://twitter.com/preillyme/status/369502554033356801

hridayeshgupta commented 11 years ago

I have changed version of maven-finagle-thrift-plugin to 0.0.8 in pom.xml then tried to build with mvn test. It failed but some repositories was downloaded at ~/.m2/ then i followed this comment https://github.com/twitter/snowflake/issues/8#issuecomment-11135629 After that i was able to build successfully using mvn test. Also tried mvn install and mvn deploy which was successful. but have no clue on how to run it

jreichhold commented 11 years ago

Can I get someone external to test the branch updated_deps? Looks to fix for me, would love to get external verification

satoshi75nakamoto commented 11 years ago

I'm checking it right now.

satoshi75nakamoto commented 11 years ago

@jreichhold — See below.

$git branch
  master
* updated_deps

$mvn -U test

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.twitter.service.snowflake.SnowflakeConfigSpec
validating development238462076416955672scala
validating development21739671345518817259scala
validating test8779000135063884026scala
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.106 sec
Running com.twitter.service.snowflake.SnowflakeServerSpec
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec
Running com.twitter.service.snowflake.ReporterSpec
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec
Running com.twitter.service.snowflake.IdWorkerSpec
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/preilly/.m2/repository/org/slf4j/slf4j-nop/1.5.8/slf4j-nop-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/preilly/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
generated 1000000 ids in 315 ms, or 3,174,603 ids/second
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.23 sec

Results :

Tests run: 19, Failures: 0, Errors: 0, Skipped: 1

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.177s
[INFO] Finished at: Mon Aug 19 15:33:08 PDT 2013
[INFO] Final Memory: 13M/137M
[INFO] ------------------------------------------------------------------------
hridayeshgupta commented 11 years ago

Build is success now. Can anyone point me to steps on how to run this server? mvn test just runs the test.

satoshi75nakamoto commented 11 years ago

@hridayeshgupta — I run it like this for testing:

$ echo ${JAVA_HOME}
/usr/jdk/jdk1.6.0_31/

$ echo ${JAVA_OPTS}
-Xmx700m -Xms700m -Xmn500m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UseConcMarkSweepGC -verbosegc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseParNewGC -Xloggc:/var/log/snowflake/gc.log -XX:ErrorFile=/var/log//java_error%p.log -server

$ echo ${CLASSPATH}
/home/preilly/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/home/preilly/.m2/repository/org/slf4j/slf4j-api/1.5.8/slf4j-api-1.5.8.jar:/home/preilly/.m2/repository/org/slf4j/slf4j-nop/1.5.8/slf4j-nop-1.5.8.jar:/home/preilly/.m2/repository/thrift/libthrift/0.5.0/libthrift-0.5.0.jar:/home/preilly/.m2/repository/com/twitter/ostrich/8.2.0/ostrich-8.2.0.jar:/home/preilly/.m2/repository/com/twitter/scala-json/3.0.0/scala-json-3.0.0.jar:/home/preilly/.m2/repository/com/twitter/util-core/5.3.0/util-core-5.3.0.jar:/home/preilly/.m2/repository/com/twitter/util-eval/5.3.0/util-eval-5.3.0.jar:/home/preilly/.m2/repository/org/scala-lang/scala-compiler/2.9.2/scala-compiler-2.9.2.jar:/home/preilly/.m2/repository/com/twitter/util-jvm/5.3.0/util-jvm-5.3.0.jar:/home/preilly/.m2/repository/com/twitter/scala-zookeeper-client/3.0.6/scala-zookeeper-client-3.0.6.jar:/home/preilly/.m2/repository/org/apache/zookeeper/zookeeper/3.4.3/zookeeper-3.4.3.jar:/home/preilly/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar:/home/preilly/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar:/home/preilly/.m2/repository/javax/mail/mail/1.4/mail-1.4.jar:/home/preilly/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/home/preilly/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/home/preilly/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar:/home/preilly/.m2/repository/com/twitter/util-logging/5.3.0/util-logging-5.3.0.jar:/home/preilly/.m2/repository/com/twitter/util-thrift/5.3.0/util-thrift-5.3.0.jar:/home/preilly/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.1/jackson-core-asl-1.8.1.jar:/home/preilly/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.1/jackson-mapper-asl-1.8.1.jar:/home/preilly/.m2/repository/com/twitter/util-codec/5.3.0/util-codec-5.3.0.jar:/home/preilly/.m2/repository/org/scala-tools/testing/specs_2.9.3/1.6.9/specs_2.9.3-1.6.9.jar:/home/preilly/.m2/repository/junit/junit/4.5/junit-4.5.jar:/home/preilly/.m2/repository/org/scala-lang/scala-library/2.9.2/scala-library-2.9.2.jar

$ echo ${MAIN_CLASS}
com.twitter.service.snowflake.SnowflakeServer

$ ${JAVA_HOME}/bin/java ${JAVA_OPTS} -cp "/tmp/snowflake/target/*:${CLASSPATH}" ${MAIN_CLASS} -f /tmp/snowflake/config/development2.scala

For a more complete example take a look at: https://github.com/twitter/snowflake/blob/master/src/scripts/snowflake.sh

hridayeshgupta commented 11 years ago

Thanks preillyme I did find out that file and I was able to run the java program, But I could not find how to access it via REST api or some other api whichever is supported. Could not find any doc regarding that Tried opening in browser but there was something else

satoshi75nakamoto commented 11 years ago

@ hridayeshgupta — Take a look at: https://github.com/preillyme/SnowflakeClient it's an example Java client for Snowflake.

AlexCzar commented 11 years ago

So, is Twitter working on the public source of this project? Or the current and future developments are (will be) done "behind the curtain" and interested users should fork snowflake and continue working on it by themselves (with maybe a rebranding—I'm not good at understanding licenses)? I see quite a number of forks and some of them are several commits ahead, yet no pull requests were issued and the one that was is neither accepted nor rejected for 7 months already.

bdd commented 11 years ago

A rewrite a Snowflake with http://github.com/twitter/twitter-server is internally complete but it will take time to release the open source version (it's not just a push-to-github operation). Feel free to fork, rewrite and rename Snowflake as the license permits.