search

bytedeco / javacpp

The missing bridge between Java and native C++
Other
4.48k stars 582 forks source link

Maven release #96

Closed Arcnor closed 7 years ago

Arcnor commented 8 years ago

Will it be possible to do a new release of JavaCPP? Last one was on Oct, 2015 and I was hoping to use the latest changes for my project, but I'm inside an environment that doesn't allow to use SNAPSHOT of libraries (so I can't use Sonatype Snapshot).

Also, have you thought about moving to semantic versioning? (http://semver.org/)

Thanks

saudet commented 8 years ago

We should be releasing a new version by next week.

The problem isn't versioning, the problem is the lack of continuous integration. Would you like to contribute in this way? @vb216 has started to work on that already, but only for Linux...

Arcnor commented 8 years ago

Sure, can you point me to what's already been done?

Also, I'm a bit lost... AFAIK JavaCPP doesn't have a native part itself. Or do you mean for tests?

Also, if nothing has been done, do you have any preference as to where run/build? I know CircleCI has OSX, although it's not free I think. Not sure about Windows, though...

saudet commented 8 years ago

The JavaCPP Presets do have native libraries. The whole point of JavaCPP are the presets. If we didn't have this, you might as well use SWIG.

@vb216 is currently setting up Jenkins, but it's proving to be a bit more challenging than first anticipated. If you could try an alternative and report on how easy or not it was to setup, that would be great. It doesn't need to be free. I can get a little bit of a budget for that.

Arcnor commented 8 years ago

I can't say I agree with that part :D. My current usage is for a lib I need to use from the JVM, and for which SWIG I'm sure will have troubles (because it needs parsing the header, etc), so for me JavaCPP is working very well for this purpose.

But then, just to be clear, you need CI for JavaCPP (easy) AND JavaCPP-Presets (not that easy), right?

If you have a few servers (and I'm guessing you do, if you're setting up Jenkins) I prefer & are well versed in Bamboo (disclaimer: I work for Atlassian, but I still think Bamboo is way superior. For an OSS is free as well, but you still need a place to put it), but I think that for what you need, some cloud solution might prove enough except for the part where you want to set up Windows builds, of course... Is JavaCPP cross-compilation friendly (Clang or GCC), or does it really need VisualCPP for Windows? Do you have a preference of Jenkins/Bamboo/random Cloud CI?

Arcnor commented 8 years ago

I've done a bit of quick research, and it seems there is a way of running it for Windows as well in the cloud. So basically, with CircleCI/Travis for Linux and OSX, and Appveyor for Windows, we can cover all platforms (and it's still free, although I know that you said budget can be obtained).

I'll start a fork and will add support for this stack, to see how everything works. Do JavaCPP have tests that will benefit from running on all platforms, or there is no point?

saudet commented 8 years ago

Right, but it's because of the presets that it works. SWIG doesn't do that. They put something together but never actually test it with anything useful.

Yes, we need Visual Studio for CUDA, FlyCapture, and others like that. We also need Mac OS X. But for now if you could get something running say on Windows, that'd be great. I have no preferences w.r.t to the tools. When we get something working, we'll move it somewhere on the cloud, yes, that's the idea. I should be able to get a bit of funding for that.

The presets builds are basically the tests that need to run.

saudet commented 8 years ago

In any case, thanks a lot for your interest and let me know if you encounter any problems! Thanks

Arcnor commented 8 years ago

Branch for JavaCPP here: https://github.com/Arcnor/javacpp/tree/ci

Current builds:

The Windows machines are building but tests are failing (cl can't find a file, or cl itself can't be found, haven't had the time to find out https://ci.appveyor.com/project/Arcnor/javacpp/build/1.0.5/job/o6ekbrslexd4ckej)

Hopefully I'll have a bit more time today or tomorrow to fix the Windows build and then start with JavaCPP presets, although I'd like to move to CircleCI (it has tests integration, while Travis does not, but it will depend on the time I have 😄 )

saudet commented 8 years ago

Nice! Don't feel pressured to do everything in a week though :)

Arcnor commented 8 years ago

I see that you released a new version already, awesome!

I guess we can open a new issue for the CI or just continue here, I don't have a strong opinion on that 😄

JavaCPP is now built in all the platforms I mentioned in my last message without errors, which is nice. If you want you can signup for both services, enable the javacpp repository in each one of them and merge my branch after doing so (because a push is required before they start compiling, so better to do that at the end). I updated the README with build tags pointing to your repo, so after you get a build, the tags should show properly on it.

If you prefer the git commit history to be cleaner and want me to squash commits or things like that, just let me know, although that will take me a while longer 😄

I've also started to compile the presets on my fork as well, same ci branch. For now, lots of errors, as expected 😃 . Linux & OSX for now: https://travis-ci.org/Arcnor/javacpp-presets

saudet commented 8 years ago

Yah, new release, but users don't test stuff until it's released, and then they send a boat load of bug reports: Too late guys. It's released. So that's why we need CI :)

If you want to open another issue, that's fine. There's also a mailing list if you prefer that for discussions: https://groups.google.com/forum/#!forum/javacpp-project Your choice.

So, you're making progress, great! What do your changes look like? What we need built are the presets obviously. The core module of JavaCPP is the easy part...

Arcnor commented 8 years ago

You can check my current changes on my branch: https://github.com/Arcnor/javacpp-presets/tree/ci

Right now, there are some failures for different things: LLVM can't be compiled because it takes more than 50 minutes, and CUDA because a package needs to be installed, for example.

The rest I can fix, I think, I just ran out of time yesterday.

saudet commented 8 years ago

Oh, cool :) Thanks

saudet commented 8 years ago

Hi, any updates with Travis CI? Does it look like it's possible to compile all of the presets with it after all? (BTW, for things like LLVM, it might be possible to build it in chunks..) If not, please let me know. We need to evaluate our options, so it's good to know what works and what doesn't! Thanks for testing all that

BTW, @jjYBdx4IL is another guy that showed interest in getting that stuff up and running, but gave up I think. It's not as easy as it seems to do.

Anyway, if there's anything that needs to be modified in JavaCPP or the presets, but you don't know how, let me know! I'll be happy to fix it up.

Arcnor commented 8 years ago

Hi,

I've been slowly fixing stuff. You can always check my progress on the same link (https://travis-ci.org/Arcnor/javacpp-presets and my repo branch). Last build was on the weekend IIRC.

I've been modifying the scripts a little, mainly removing excessive logging.

Major problems right now are:

All in all, it's looking great IMHO :). Right now we have 22 of 36 successful builds, and some of them (like tensorflow) are "easy" to fix. The others will need some thinking.

saudet commented 8 years ago

Great! Thanks for the update :)

saudet commented 8 years ago

Hey, how is everything going? Is there anything that you are having problems solving? I understand there's a lot of things that we need to get working, but even if we don't get everything working, since they are free services, it gives us at least something :)

CUDA can be installed from repos on Linux as well, should make things easier there. But it will still need a library that only comes with the driver ATM. If we use the "--cudart=static" flag for NVCC it might fix this missing library issue both at build time and at runtime on systems that don't have a GPU: https://github.com/bytedeco/javacpp-presets/issues/219 If you have the time to check this out, it would be great.

BTW, the Linux versions should be built in CentOS. Does Travis CI offer that distro? If not we can use Docker, seems the way to go: https://djw8605.github.io/2016/05/03/building-centos-packages-on-travisci/

Arcnor commented 8 years ago

Hey,

Sorry, I haven't been able to do much on this recently, have been swamped with other work.

Next week I'm planning to continue using/working with JavaCPP, so I'll try to continue on this.

Regarding CUDA, cool that we can do something about it, although that was the last one I was going to try fixing, as it looks the most complex :).

Regarding the distro, right now it uses TravisCI default (Ubuntu). Using Docker is possible, but there are some limitations when doing that, and I need to check if my current approach works with them.

Arcnor commented 8 years ago

Hey, just adding a quick comment to say that I'm still planning to go with what I said, but I didn't get around to get back to my work using JavaCPP because other things took priority. Will try to get back to this ASAP :)

saudet commented 8 years ago

Sounds great, thanks!

Arcnor commented 7 years ago

Hi there, it's been a long time :).

From time to time I want to continue on this, but I still haven't find the time at work to continue on my javacpp related project, which limits my options in that regard :)

In any case, I was thinking that maybe we should just enable CI for the presets that work right now? That way, you'll get less overhead anyway when you need a new version, we don't lose all the work I did and we can fix the ones that doesn't work later.

I'm rerunning the build to see if Travis still works with my changes, and to see which ones are "usable". I also need to merge your latest changes, but one step at a time :).

What do you think?

Arcnor commented 7 years ago

So the working projects on "all" platforms (Linux x86/x64 & OSX x64) are:

Working on Linux x64/OSX x64:

This is all in 1.2, but I think it's a great start. I'll rebase on top of the current master and retry, but this is just so you can get an idea of what we can build right now that might be worth adding to the official repo, and then we fix the rest later.

saudet commented 7 years ago

Yes, it's a good start, thanks! BTW, @vb216 has been working in parallel using Jenkins, some results here: http://bytedeco.org/builds/ This is also still a WIP though. Right now it's running on AWS for Linux and Windows, but there is no support for Mac OS X there...

Given that Travis CI doesn't do Windows, and AWS doesn't do Mac, I'm starting to think of getting a Mac mini at https://www.macstadium.com/ or something, with Linux and Windows running in virtual machines on the same physical machine. It would make it a lot easier to get CI running that way instead of having to deal with security all over the place between different cloud providers. We'd still need something else for Power, but that would be about it. @Arcnor @vb216 Thoughts?

vb216 commented 7 years ago

Well, the big problem for me is time.. It's been hard (/time consuming) enough just getting everything to build in one go - to me the point of a CI env should be to get regular builds out or test to changes, but it feels like I'm usually behind the curve playing catchup.

Platform wise, the jenkins setup is pretty flexible, adding new build nodes is just a case of having them connect via VPN to the main AWS instance. I've just tried moving the arm build from docker/cross-tools to native on a Pi - another case where its been taking more and more time just to get dependencies and libraries working, but alot easier on the native device. Similar approach for Mac at the moment.

So, I guess the key question would be, what's the main end goal, and does re-working the solution architecture move closer towards it. If running it all off of a mac instance there means it could be opened up for more people to resolve build issues, that would be a definite bonus.

saudet commented 7 years ago

@vb216 Yes, the idea is to provide a build system for users. Ideally we would want something like Koji, but you said previously that Jenkins isn't secure enough for that. So what does that leave us?

vb216 commented 7 years ago

When you say, build system for users, do you mean, users can interact with the GUI i.e. kick off jobs? Or just see the progress / output of last builds and have snapshots available from Maven? If its the latter, that's already working in the jenkins->bytedeco website output and the uploaded builds. If its the former, that does add more complexity and management effort. Jenkins itself can be as secure as needed I suspect (as demonstrated with other projects using publicly accessible versions), but, they presumably have a team supporting that effort.

I'm no Jenkins advocate especially, and I suspect that would apply to any CI choice if its going to be open to the public and not be routinely patched, configs updated, intrusion detection logs checked etc. You could maybe have a approach of, lets just go for it, if it gets hacked no problem and impact could be reduced down to maven credentials being compromised (and the usual, servers end up being smtp relays, or join a bot net). Or, if you want public GUI interaction, you could maybe run a public CI in a VM/docker and push the data from the private CI to public to reduce down some of the risks.

I'm definitely no CI expert though, just basing this on lessons learnt the hard way already!

saudet commented 7 years ago

Ideally, the former, that's what a build system is for! And in this case especially being able to get builds for all platforms, regardless of the developer's machine. But sure we can start with the latter and only expose final outputs. Because like you say, the lack of time is one factor. However, for the same reason, I feel it takes more time to maintain multiple machines form multiple cloud vendors, than just one, doesn't it?

vb216 commented 7 years ago

Well, I disagree a little that a build system is primarily for public build requests.. I had a quick look just to sanity check, and its hard to find any Jenkins CI that are setup for the public to start a build. I'd imagine the normal start on that would be either a new commit (giving the benefit of testing that commit, and making it instantly available), a scheduled run, or a developer initiated build.

But, I figure, why not. I've taken a backup, and it's now publicly accessible here: http://javacv.hopto.org:8085/

If you really want anonymous build requests, its one more click to enable that.

The time consuming bit with multiple build machines isn't so much their location or vendor base, its updates/changes to keep them able to build the latest commit either on this repo, or from something upstream that is pulling its latest version. I can't see a way around that, but I can see extra complexity in trying to do it all on one box (with each vm/docker needing 2gb+ to run, that's a sizeable footprint too). Having said that, getting a mac from the place you mentioned earlier could be good, to move more into the cloud. Currently I've got mac and arm platforms here on a separate subnet firewalled off, so it would be one less security headache on my end :)

saudet commented 7 years ago

I'm sorry, I didn't mean that a build system should be available to anonymous users or anything, but that is should be usable by the community. That's what Travis CI is, and that's what Koji is (see https://fedoraproject.org/wiki/Using_the_Koji_build_system ), although that's not what Jenkins is, it's only a tool. It can be part of the solution, and what you are doing right now can actually be used by our community. The workflow is a bit convoluted, but it works! For example, a developer on Windows fixes something with one of the presets, but needs to provide updated Mac binaries for one of their users, so:

  1. They send us a pull request,
  2. We review and merge it,
  3. Your system starts building new SNAPSHOT binaries for all platforms, and
  4. Makes them available on https://oss.sonatype.org/,
  5. Where developers can fetch them and send over to the end users!

So, I do hope you can keep working on that, hopefully with help from others (@Arcnor?). Maintaining the boxes is obviously time consuming. If we had a lot of of members to take care of this it wouldn't be an issue, but the less boxes there are to maintain, the less time anyone has to spend on that, right?

vb216 commented 7 years ago

Hm I'm not sure I follow. I'd argue they're all just tools.. I'm guessing the difference you have in mind is how the build is constructed, with a more dynamic provision of build envs in other CIs, versus Jenkins sending build jobs to existing server instances (?). Conceptually, I prefer the look of Travis and similar, but I think its often the case than when bespoke requirements increase, you need more tailoring. With the Jenkins approach, if you can establish an ssh connection back to the build master, then you can build on that node. That's why adding things like powerpc took only a few days to add in, but I'm struggling to see how that could be achieved with the more cloud based solutions? But, as we've said, the difficulty and time consumption is in having to maintain all these OS instances. But, that has plus sides too - where you've needed specific versions of libs for the build, there's the flexibility to set them up, which might be more challenging if there's only one image type available from another CI provider.

Despite how it might sound, I'm really not advocating one over the other, I'm actually quite interested in how best in practise to tackle this kind of problem. If I had more time, would be quite interesting to working on Travis.

Related to that, and your point about reducing boxes to maintain, I guess the approach there is cross-compiling or dockers, and remove the dependency to build natively powerpc? I could see that blocking alot of CI solutions?

Looking at your workflow, there's not many changes needed for that in the Jenkins setup, just switch it build on new commits (which makes sense, to make latest snapshots available with changes). Meanwhile, any changes to build more out of dockers would help both in the Jenkins solution and possibly for any other CI choice too I think. Looks like I should scrap the native pi arm build and go back to trying to get the docker or cross compile working again..

saudet commented 7 years ago

Ok, I'm getting the terminology wrong then. I'm probably talking about a "build service". Jenkins isn't a build service, but Travis, Koji and https://build.opensuse.org/ too are. So, when people want to use one of those, they just need to create an account and start doing their builds.

In any case, what you are aiming for is great, and is absolutely useful, I'm not questioning any of that at all, but it needs to actually build stuff to be useful. Right now I'm always seeing a lot of red, so I'm trying to figure out if we can't make some things easier, one way or another, to let us concentrate on making the builds actually pass, and have it become useful for the community. That's all, not trying to impose anything here. Sure, we could ignore Power builds, etc for now. That makes sense. Also, since we need to review and merge any changes before starting a build, it doesn't really need to be automated, for now. I'd rather have something working than everything not working. :)

saudet commented 7 years ago

I forgot to mention that I would be setting up that box so that all builds pass. We need a server for that, and would be maintaining that instead of my local machines, so I think we could save a lot of time that way.

vb216 commented 7 years ago

Yeah it has been tricky getting these to build consistently, but most often that seems due to a new release or upgraded dependency causing problems. I think on a static repo, it'd be quite close. Plus getting the last few bits working like each OS build properly sending the right objects to maven took a little longer to iron out.

The mac build gives me the most headaches, so I'm not going to complain about that problem going away!! That build has been stuck on a caffe problem for a while now.

On 20 February 2017 at 08:46, Samuel Audet notifications@github.com wrote:

I forgot to mention that I would be setting up that box so that all builds pass. We need a server for that, and would be maintaining that instead of my local machines, so I think we could save a lot of time that way.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bytedeco/javacpp/issues/96#issuecomment-281019550, or mute the thread https://github.com/notifications/unsubscribe-auth/AMYRJ8Tbaii0RzmrZyLDSr1CtFCOVHWbks5reVLegaJpZM4Idq_Z .

vb216 commented 7 years ago

As another idea.. if we seem a bit blocked with Travis / cloud hosted builds, and if you're exploring building it all out on a Mac, what about using Vagrant? Could help with providing a universal build config for anyone that wants to build locally, and throwaway virtual envs to spin up when project builds needed? Feels a bit like Travis, minus the cloud + perhaps such close git integration?

saudet commented 7 years ago

Vagrant still needs some images for Mac. It's not a technical issue, but a legal one, although some people seem to be able to get away with it: https://github.com/AndrewDryga/vagrant-box-osx#downloads I wonder if we should rely on that... What do you think?

vb216 commented 7 years ago

Ah yes. It's a tricky one, I was prefer to do things in a way that avoids problems down the road, plus would be a shame to put in effort to get it working and Mac to tighten up their license/hardware checking.

I've been looking more into the Travis side though, Arcnor did a lot of groundwork there, and the few build errors still around I largely recognise. There's probably some tuning we could do on some of the builds to get under 50 mins (things like javadoc build could be moved out to a site build rather than snapshots, would save time). I've started trying to replace the linux build with a fresh centos7 docker pull, if I get a little time today that should test the concept.

Quite a fan of having a really clean build env, the Jenkins ones were getting a bit messy with previously installed components and losing the visibility of what was there.

saudet commented 7 years ago

I think we can get builds at least for Linux x86, x86-64, ARM, Android, and Mac on Travis CI. If we could get that, that would be good, yes.

BTW, another reason to go cloud is that Java developers frequently develop under one platform, but need to provide builds for another platform. So, that and Mac legal issues.

vb216 commented 7 years ago

Reasonable progress, centos docker approach largely working for x86_64

Big problem is the 50 min build time limit, llvm and tensorflow are hitting it. Is there anything in default tensorflow build that isn't used and could be disabled? Seems to build around 2250 out of 2500 targets at the 50 min mark.

Only other options I could think of is some prebuild step that supplies a chunk of prebuilt code, but thats not great for making a clean build. Or, it looks like their paid for options have 120 min time limit rather than 50 min (tho I'm not 100% sure what the 120 mins is limiting just yet, I assume individual build steps like the 50 min limit)

Time limit will get harder to meet for other platform targets as may need to install things like cuda, for centos7 nvidia supply a prebuilt docker.

saudet commented 7 years ago

@vb216 thanks for looking into this! looks like the build time limit is going to be a problem: https://github.com/travis-ci/travis-ci/issues/2736 CUDA kernels are what takes most time to compile in TensorFlow, but even if we cut things down now, having to deal with in the long term isn't appealing. @Arcnor would you have ideas?

vb216 commented 7 years ago

FWIW, the 'bootstrap' plan might solve the issue, but its 69usd a month. Seems on the paid for platform, the 120 min time limit applies to the job tasks, and as arcnor split them out, the longest running should complete within that time.

I suspect with builds needing more customisation before the real build starts, we'd be even more north of 50 mins (but probably under 120).

They do offer a trial, for first 100 builds, if the paid for option was ever likely to be a long term option, but otherwise best not to waste effort.

Alternatively, what about circle CI? They seem to have a bit more flexibility for open source projects https://circleci.com/pricing/#faq-section-os-x but I have even less experience of those than of Travis..

vb216 commented 7 years ago

Dropped Travis support an email just to check if there's any flexibility on the 50 mins when its a clear compile time issue for an open source project.

saudet commented 7 years ago

Ok, thanks! I think there is no flexibility though. Even when paying it sounds like 120 minutes is a hard limit we're likely to hit eventually. Anyway, at that price, might as well pay for a Mac instance...

vb216 commented 7 years ago

Good progress I think.. https://travis-ci.org/vb216/javacpp-presets/builds/212507132 as an example of recent build, centos 64 bit docker all built fine, mac build not bad, missing cudnn (not sure how to tackle downloading this into a clean vm, as you need to sign in with developer account to access), and caffe not finding libquadmath.a (rings a bell I think..?)

Windows64 build here https://ci.appveyor.com/project/vb216/javacpp-presets/build/47, similar issue needing cudnn downloaded, libdc1394 needing something, but apart from that looking pretty good.

So, tricky one is cudnn dependency. After that, would need to think what should be snapshot deployed - as these are built out of sequence in parallel, whether a deploy at the end of each step would enable a platform build (the .m2 dir is cached between builds).

saudet commented 7 years ago

Yes, looking pretty good. Thanks! It's not just cuDNN though, there's also FlyCapture and I'm also working on MKL, among others. They wouldn't offer some sort of private area where we could put things required for build, but that wouldn't be available for download?

vb216 commented 7 years ago

I wondered about google drive / dropbox, with a secure download? That way, its not publically downloadable, the access token can be encrypted into the build file, and from a quick read of the cudnn license, it doesn't seem to immediately break any conditions. I assume their main concern is around having control of code if it gets used in someway they're not happy with, so I can't see a locked down build agent coursing much concern (?).

Only other idea is to use their cache, that gets stored on encrypted AWS S3 somewhere, and only accessible by the build tree. But, a) there's a size limit and binaries could eat that up quick, b) drive/dropbox seems more under easy control, and with the access token/password just becomes another download link from the builds perspective

saudet commented 7 years ago

This? https://docs.travis-ci.com/user/environment-variables/#Encrypting-environment-variables Yes that sounds like a great idea!

vb216 commented 7 years ago

Yeah that's the one. Need to set it up for maven deploy credentials anyway so hopefully same technique will solve the downloads issue. Will give it a try

On 19 Mar 2017 11:35 a.m., "Samuel Audet" notifications@github.com wrote:

This? https://docs.travis-ci.com/user/environment-variables/# Encrypting-environment-variables Yes that sounds like a great idea!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bytedeco/javacpp/issues/96#issuecomment-287610364, or mute the thread https://github.com/notifications/unsubscribe-auth/AMYRJ3R3hK_oahjZPgw2w8maC0CpWK5Bks5rnRLlgaJpZM4Idq_Z .

vb216 commented 7 years ago

Encrypted variables works nicely! https://travis-ci.org/vb216/javacpp-presets/builds/213059693 has cuda building fine for mac now. Trying to do the same for flycapture but needs a bit of tidy up to recreate the deb contents in usable way for centos (unless theres an easier way around that!).

Any thoughts on solving this one: lapacke_zsyconv.c: In function 'LAPACKE_zsyconv': lapacke_zsyconv.c:56:1: warning: label 'exit_level_0' defined but not used [-Wunused-label] exit_level_0: ^ make[1]: warning: -jN forced in submake: disabling jobserver mode. gcc-5: error: /usr/local/opt/gcc?/lib/gcc/?/libquadmath.a: No such file or directory

I remember quadmath cropping up before but I thought the brew install gcc-5 fixed it, seems something else missing now? With that cleared out, the mac build is nearly done too.

saudet commented 7 years ago

Nice!

libquadmath.a, that comes with gfortran, have you also installed that?

vb216 commented 7 years ago

seems gfortran stuff wrapped up into gcc brew, but something strange on the paths. added a symlink and works fine.

So, actually pretty close. Windows 64 pretty much all there (https://ci.appveyor.com/project/vb216/javacpp-presets/build/109), centos64 and mac64 also all done apart from tensorflow on mac, throws a genuine error by looks of it. Just adding in flycapture for all of those and then I think we have a clean sweep. Shouldn't be too hard from there adding in the 32 bit versions and arm/ppc.