search

NeuronRobotics / nrjavaserial

A Java Serial Port system. This is a fork of the RXTX project that uses in jar loading of the native code.
Other
345 stars 143 forks source link

Refactor native builds #189

Closed MrDOS closed 3 years ago

MrDOS commented 4 years ago

This represents a major stab at resolving #19.

In the years since opening that issue, out-of-box cross-compilers and ~Travis CI~ GitHub Actions have improved. And as distro release cycles have trucked on, the minimum glibc version required to support the vast majority of installed systems has crept upward, while the minimum version targeted by recent compiler releases for the NRJavaSerial codebase has stayed relatively static. Not to mention that you've recently bumped the minimum Java version to 1.8, which implicitly drops a lot of older systems (fairly, I think). Basically none of the carefully-planned insanity I had in mind way back then is necessary to generate a broadly-compatible library now.

I mentioned a couple weeks ago that I was poking at a full JNA “port”. I've not altogether lost hope in that idea, but I think it would be easier to pull off if the current native code was in better shape, and there is a lot of low-hanging fruit to improve upon. But I'm leery of attempting to make too many changes to the existing native code without solid infrastructure for rebuilding it; hence this effort.

As per the commit messages of this branch, this pull request encompasses:

The biggest changes visible to the end user in the refactored makefile are that parallel builds (-j) will work now (although not with full efficiency); there are clean-platform targets so you can be sure you're generating new output; and that the Windows libraries compile the same from Windows and Linux (same targets; no more Wine).

From a developer standpoint, the new makefile is much more declarative (and less like a shell script, as #70 pointed out). The actual build process for a library doesn't change much from platform to platform; compile some objects and link them. What does change is the compiler/linker executable, maybe some flags, and the output path. This makefile reflects that much better, I think. Superficially, the fake targets which the user will want to invoke (e.g., windows, linux64) are all the same, but those in turn all route through the same (dynamic) target to build their output.

Please take it for a spin and tell me what you think. You can see ~some example Travis builds and the exported build artifacts~ an example GitHub Actions build. ~Note how build 19 built new native libraries which got archived, but the natives produced by build 20 were identical to the ones already in the repo, so they weren't archived.~ The artifacts from each build (native libraries + packaged JAR) are archived with the build logs.

I do think it's possible to build the FreeBSD native libraries this way as well (from within a Docker container), ~but it didn't work on first stab~ so I did that too. ~I'm going to keep at that, but I wanted to submit this work as it stands – I think it's a sufficient improvement on what's in the repo currently.~

~If and when you're ready to accept the pull request, before you merge it, I'll e-mail you an access key for that S3 bucket so you can configure the environment variables for the Travis build for this project.~ Unlike Travis, build artifacts are a true first-class feature of GitHub Actions, so no external object store configuration is required.

MrDOS commented 4 years ago

I've now added a Travis job for FreeBSD cross-compilation to this branch, so the Travis job compiles (and, as necessary, archives) native libraries for all target platforms. I'll leave some comments in #173, which I think this now closes, too.

I see now that https://builds.nrjs.org/ isn't loading properly. Funny; it was working (or so I thought) when I opened this PR. I'll look into it.

madhephaestus commented 4 years ago

THis is so cool!

MrDOS commented 4 years ago

I misconfigured the S3 bucket browser I'd tried to use, and didn't encounter the error on my own desktop due to a browser security exception. Oops! https://builds.nrjs.org/ is working now.

I would like to reconfigure Travis to perform a multi-stage build so that jobs ingest their own native libraries (i.e., no manual intervention required to produce a jar with updated natives). Beyond that, @madhephaestus, do you have any objections to merging?

MrDOS commented 4 years ago

That last commit splits the build into two stages: native libraries, and Java. The very first step of the Java job pulls in any native libraries compiled in the first stage, so the produced JAR encompasses new natives (where applicable).

I think I'm about done here. IMO, this fully completes the original intent of #19. The only thing I can think which might be worth adding would be automatic uploading of builds corresponding to tags to Maven Central; but I think that's outside of the scope of this issue.

@madhephaestus, I've just sent you an e-mail with S3 credentials for the builds.nrjs.org bucket. If there's anything else you'd like me to do here, please let me know; otherwise, I'm going to move on to trying to improve upon the portability of port locking (now that I can safely and sanely make changes to the native code...).

MrDOS commented 4 years ago

@wborn, I know you've compiled the native libraries yourself, and maybe @RafalLukawiecki has – not sure if either of you have any thoughts or feedback on this and how it might make things easier/harder for you as downstream consumers.

RafalLukawiecki commented 4 years ago

I am glad to see that you have done the work to automate it. On my end, I have merely recompiled the native libs using my install of FreeBSD. Since I don't use Linux or Docker much, I am not sure I can help you with the crosscompiler, but I would be happy to test anything you have produced. Perhaps @wborn has more thoughts on it.

MrDOS commented 4 years ago

I've been working across a few different branches over the last week, and I've found it really difficult to tell from the browser interface which job/artifacts correspond to which branch. I've changed the path for artifacts so that it includes the branch name: e.g., https://builds.nrjs.org/MrDOS/nrjavaserial/slf4j-logging/46 (instead of just https://builds.nrjs.org/MrDOS/nrjavaserial/46).

Outside of that, it has been working quite well for me.

wborn commented 4 years ago

Thanks for working on this @MrDOS! It all looks very good to me. :+1: I think this approach will be easy to use for downstream users who want to build their own version. With these changes the build will be fully automated and easy to reproduce for everyone. It's also nice that there will only be one (real) Makefile.

MrDOS commented 4 years ago

Given the uncertainty surrounding the new billing model over at Travis, I've converted the build to GitHub Actions. As an added bonus, artifacts work much better now, because they're attached directly to the workflow run. I was okay with the way the archiving worked before, but because it relied on S3, it did mean only those armed with bucket credentials could use it. This way, anyone who forks the repository can also trigger their own cross-platform builds.

@madhephaestus, can I merge this?