search

pantsbuild / binaries

A temporary home for pants static binaries and scripts
16 stars 37 forks source link

Add build script for Thrift 0.11.0 #83

Closed StephanErb closed 5 years ago

StephanErb commented 6 years ago

Build tested successfully on MacOS Sierra and Centos 6 (pantsbuild/centos6:latest).

As all existing Thrift binaries follow the old scheme of a single script in the root of the repo, I followed the same approach here. Please let me know if this should be changed.

stuhood commented 6 years ago

As all existing Thrift binaries follow the old scheme of a single script in the root of the repo, I followed the same approach here. Please let me know if this should be changed.

@StephanErb : This should be changed. Sorry!

https://github.com/pantsbuild/binaries/commit/80a0051015b708eb78b2ff42479c28aff59c704d is a good example of how to do this.

cosmicexplorer commented 6 years ago

rationale, not what to do yet. what I would like to be done before merging is below

The use of any functions from utils.v1.bash is 100% optional, as is splitting the script into functions -- those are things I did in order to make it easier for me personally to write repeatable scripts.

As mentioned, one thing that we would like to maintain is placing scripts in the directories they extract their contents to -- this is shown in the commit 80a0051 linked above. Another thing that I would really appreciate is to make all the version strings used in the script into command-line arguments to the script, and then provide a build.sh script in the same directory that passes in those version strings. The other utility of a build.sh script is that you can run yum install commands in there instead of just in comments (and I personally find this separation useful).

I don't want to attach too much extra work to this PR, especially since this requires building multiple other tools. For new thrift versions we contribute here in the future, we can incrementally move the build script towards using functions (for example).

here is what I would specifically want to be done

So for this PR I would only want to see the directory placement/symlinking, and then a separation into a build.sh which invokes a template script with each relevant version string (which looks like just requires versions for thrift, bison, byacc, yacc, and flex?). Both of these processes are described in the workflow section of the README.

Please let me know if the above is unclear, if this is a huge hassle to do, or if there's additional work I haven't considered.

cosmicexplorer commented 6 years ago

Hey, just as a quick example of the above, I just made #84 to provide gnu make (for a pants PR). It uses a lot of the functions in utils.v1.bash, but none of that is really necessary at all beyond the directory structure/symlinking. There is definitely a lot of automation that could be done here that we have erred against creating so far for the sake of reproducibility (and also, for the sake of easier upgrading -- hence the creation of the script parameterized by version strings). We're totally open to alternatives, I'm aware this is a lot of structure to go through.

StephanErb commented 6 years ago

Thanks I will take a look. As this is a pretty busy week for me, it will probably take a few days until I can pick up the suggestions.

cosmicexplorer commented 6 years ago

Absolutely! I was only offering it since I happened to have just made the PR — thanks so much for working on this! Please feel no rush to complete. On Mon, Oct 22, 2018 at 22:47 Stephan Erb notifications@github.com wrote:

Thanks I will take a look. As this is a pretty busy week for me, it will probably take a few days until I can pick up the suggestions.

— You are receiving this because your review was requested. Reply to this email directly, view it on GitHub https://github.com/pantsbuild/binaries/pull/83#issuecomment-432101452, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPqT9gYy57hy_CXzZZYJWu02E4CF4s-ks5unq1ngaJpZM4Xj0_I .

StephanErb commented 5 years ago

@cosmicexplorer I have updated the PR trying to follow the conventions of the other PRs. Please let me know if this matches what you had in mind.

StephanErb commented 5 years ago

@cosmicexplorer The bison error is strange. Did maybe the bison build fail silently for you?

StephanErb commented 5 years ago

Is there a way to get this unblocked? Unfortunately I am stuck on an older MacOS version so I cannot reproduce the build failures seen by Danny

stuhood commented 5 years ago

I see the same issue on 10.14, unfortunately. bison aborts with:

Process 39466 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00007fff6f71201e libsystem_kernel.dylib`__abort_with_payload + 10
libsystem_kernel.dylib`__abort_with_payload:
->  0x7fff6f71201e <+10>: jae    0x7fff6f712028            ; <+20>
    0x7fff6f712020 <+12>: movq   %rax, %rdi
    0x7fff6f712023 <+15>: jmp    0x7fff6f6f4e67            ; cerror_nocancel
    0x7fff6f712028 <+20>: retq
Target 0: (bison) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff6f71201e libsystem_kernel.dylib`__abort_with_payload + 10
    frame #1: 0x00007fff6f70d561 libsystem_kernel.dylib`abort_with_payload_wrapper_internal + 82
    frame #2: 0x00007fff6f70d593 libsystem_kernel.dylib`abort_with_payload + 9
    frame #3: 0x00007fff6f61978f libsystem_c.dylib`_os_crash_fmt + 182
    frame #4: 0x00007fff6f64fb8e libsystem_c.dylib`__vfprintf + 16944
    frame #5: 0x00007fff6f67413b libsystem_c.dylib`__v2printf + 475
    frame #6: 0x00007fff6f65956f libsystem_c.dylib`_vsnprintf + 410
    frame #7: 0x00007fff6f6595cc libsystem_c.dylib`vsnprintf_l + 41
    frame #8: 0x00007fff6f64a4d5 libsystem_c.dylib`snprintf + 156
    frame #9: 0x0000000100044de0 bison`vasnprintf(resultbuf="/Users/stuhood/src/pantsbuild-binaries/build-support/bin/thrift/mac/10.12/0.11.0/bison-2.5.1-osx/bison-2.5.1/install_dir/bin/bison���", lengthp=<unavailable>, format=<unavailable>, args=<unavailable>) at vasnprintf.c:0 [opt]
    frame #10: 0x0000000100041293 bison`rpl_fprintf(fp=0x00007fffa165a240, format=<unavailable>) at fprintf.c:45 [opt]
    frame #11: 0x0000000100040eb2 bison`error(status=0, errnum=0, message="%s: missing operand") at error.c:313 [opt]
    frame #12: 0x00000001000072db bison`getargs(argc=1, argv=0x00007ffeefbff480) at getargs.c:0 [opt]
    frame #13: 0x000000010000bdc5 bison`main(argc=1, argv=0x00007ffeefbff480) at main.c:80 [opt]
    frame #14: 0x00007fff6f5bc085 libdyld.dylib`start + 1
stuhood commented 5 years ago

And the same with bison 2.7.1.

EDIT: And as a further datapoint, it looks like the build-thrift-0.10.0.sh script sees the same failure on my machine (OSX 10.14). So it looks like perhaps this is an issue with bison+mojave.

stuhood commented 5 years ago

As a datapoint, bison-3.2 seems to work on Mojave.

StephanErb commented 5 years ago

@stuhood thanks for investigating! I have pushed a change to use bison 3.2 for MacOS. However I have left it at 2.5.1 for the linux build als the new bison fails to compile in the ancient CentOS 6 image.

StephanErb commented 5 years ago

@stuhood done.

Thanks for handling the builds!

stuhood commented 5 years ago

Synced to s3:

upload: bin/thrift/linux/x86_64/0.11.0/build.sh to s3://binaries.pantsbuild.org/bin/thrift/linux/x86_64/0.11.0/build.sh
upload: bin/thrift/linux/x86_64/0.11.0/build-thrift.sh to s3://binaries.pantsbuild.org/bin/thrift/linux/x86_64/0.11.0/build-thrift.sh
upload: bin/thrift/mac/10.11/0.11.0/build-thrift.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.11/0.11.0/build-thrift.sh
upload: bin/thrift/mac/10.11/0.11.0/build.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.11/0.11.0/build.sh
upload: bin/thrift/mac/10.10/0.11.0/build.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.10/0.11.0/build.sh
upload: bin/thrift/mac/10.10/0.11.0/build-thrift.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.10/0.11.0/build-thrift.sh
upload: bin/thrift/mac/10.12/0.11.0/build-thrift.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.12/0.11.0/build-thrift.sh
upload: bin/thrift/mac/10.12/0.11.0/build.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.12/0.11.0/build.sh
upload: bin/thrift/mac/10.13/0.11.0/build-thrift.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.13/0.11.0/build-thrift.sh
upload: bin/thrift/mac/10.13/0.11.0/build.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.13/0.11.0/build.sh
upload: bin/thrift/mac/10.12/0.11.0/thrift to s3://binaries.pantsbuild.org/bin/thrift/mac/10.12/0.11.0/thrift
upload: bin/thrift/mac/10.14/0.11.0/build-thrift.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.14/0.11.0/build-thrift.sh
upload: bin/thrift/mac/10.14/0.11.0/build.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.14/0.11.0/build.sh
upload: bin/thrift/mac/10.11/0.11.0/thrift to s3://binaries.pantsbuild.org/bin/thrift/mac/10.11/0.11.0/thrift
upload: bin/thrift/mac/10.10/0.11.0/thrift to s3://binaries.pantsbuild.org/bin/thrift/mac/10.10/0.11.0/thrift
upload: bin/thrift/mac/10.8/0.11.0/build.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.8/0.11.0/build.sh
upload: bin/thrift/mac/10.8/0.11.0/build-thrift.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.8/0.11.0/build-thrift.sh
upload: bin/thrift/mac/10.9/0.11.0/build-thrift.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.9/0.11.0/build-thrift.sh
upload: bin/thrift/mac/10.14/0.11.0/thrift to s3://binaries.pantsbuild.org/bin/thrift/mac/10.14/0.11.0/thrift
upload: bin/thrift/mac/10.9/0.11.0/build.sh to s3://binaries.pantsbuild.org/bin/thrift/mac/10.9/0.11.0/build.sh
upload: bin/thrift/mac/10.13/0.11.0/thrift to s3://binaries.pantsbuild.org/bin/thrift/mac/10.13/0.11.0/thrift
upload: bin/thrift/linux/x86_64/0.11.0/thrift to s3://binaries.pantsbuild.org/bin/thrift/linux/x86_64/0.11.0/thrift
upload: bin/thrift/mac/10.8/0.11.0/thrift to s3://binaries.pantsbuild.org/bin/thrift/mac/10.8/0.11.0/thrift
upload: bin/thrift/mac/10.9/0.11.0/thrift to s3://binaries.pantsbuild.org/bin/thrift/mac/10.9/0.11.0/thrift