search

jcuda / jcuda-main

Summarizes the main JCuda libraries
MIT License
98 stars 20 forks source link

Upload to maven central #14

Closed nakul02 closed 7 years ago

nakul02 commented 7 years ago

Hi,

I work on the systemml project, where we use the JCuda project. For now we don't have a straightforward deployment for JCuda (0.75b) through maven and expect the user to have the jcuda jar files in the classpath (and the .so files in the LD_LIBRARY_PATH).

I contributed jcuda8 jars for linux, apple & ppcle64 to the mavenized jcuda project. I did this to try out a simple deployment method in this PR.

In the longterm, we want to have jcuda available through maven. My plan was to upload the jcuda jars and the jcuda-native jars (v0.8.0) to maven central. Does the license allow this and is this ok by you?

For now, people from the nd4j project seem to have jcuda jars for v0.6.0. If you, @jcuda, did this, that would be ideal.

blueberry commented 7 years ago

I am also interested in this, and am willing to help with linux build/testing if help is needed. I think @jcuda mentioned that maven upload is on his TODO list, so I hope his schedule will align with this issue soon :)

jcuda commented 7 years ago

Yes, bringing JCuda into Maven Central is one important goal. One of the main reasons for the restructurings between 0.7.5 and 0.8.0 was exactly done for this: Distributing the natives separately means that the end-user would have to fiddle with the LD_LIBRARY_PATH or java.library.path, which is a nuisance. Now, in version 0.8.0, the natives are contained in the JARs and can be loaded at runtime, and thus, this should be a huge step towards proper Maven deployment - as you also pointed out in the comment of your PR.

In the longterm, we want to have jcuda available through maven. My plan was to upload the jcuda jars and the jcuda-native jars (v0.8.0) to maven central. Does the license allow this and is this ok by you?

Technically and regarding the license, this would be OK. But conceptually, I would strongly prefer to upload them "officially". I already prepared this locally (e.g. Sonatype account etc). But I'd really appreciate help regarding the remaining open issues that I see right now:

Maybe @MysterionRise can give some helpful hints here as well...?

nakul02 commented 7 years ago

@jcuda - I've built the native jars for darwin 64 bit and power8 (ppc64le). The cublas jars are here. Others are in similar locations in the directory structure.

jcuda commented 7 years ago

Sorry, I overlooked that you mentioned "apple" as well.

So right now, I think that one approach could be:

The last step would be quite a lot. I'd create a BATch script for that, unless there is a way to bundle and deploy multiple artifacts that are contained in one big JAR file - does anybody have experience with this?

nakul02 commented 7 years ago

@jcuda - anyway i can help to speed this up?

jcuda commented 7 years ago

Yes, sorry, I had some other appointments this week (glTF webinar, job interviews), but allocated next week for tackling this. I'll start on sunday/monday. If you (or anybody) could tell me whether the approach sketched above could work, or whether there are other/better approaches, that would be great. I'm a bit skeptical about this "jar bundling" step - Maven is veeery picky when something is not absolutely right (that's a good thing, but it can be tricky to get it right). In any case, I'll give it a try, and see how far I get (possibly posting questions or intermediate states here...)

nakul02 commented 7 years ago

I actually don't have any experience uploading to sonatype. Even so, here are my 2 cents: It seems that this method is lesser work than the bundling method, but you'd need access to all the platforms for this method. (Not to mention ease with future releases). But seeing as how that's not possible, maybe sticking to the bundling method is the best.

jcuda commented 7 years ago

Thanks for this link, I've already looked at some resources, but in the end, I guess I'll just have to try them out, and see which one works best (also in view of a then hopefully smooth upload for CUDA 9).

From a first, short (!) look at the link, I'm not sure what you mean by

you'd need access to all the platforms for this method

Does this really refer to the deployment itself? As mentioned above: I cannot build the natives for all libraries anyhow, so would have to resort to the natives that @MysterionRise collected (hoping that the "re-signing" is possible...)

nakul02 commented 7 years ago

I assumed that if you were to follow the directions given here (linked in the original article under point 3), you'd be doing a mvn clean deploy. For doing this, you'd need to be on the platform from which artifacts are being uploaded (think *-natives-0.8.0-linux-x86_64.jar or *-natives-0.8.0-linux-ppc_64.jar, etc). From the previous discussion on this thread, I assumed that you don't have access to a mac or to a power8 and said that just sticking to the bundling method (for all 4 platforms) is a good idea.

jcuda commented 7 years ago

Some questions that I'm currently juggling with:

Now I'm not sure how to bring this information into the respective POMs. It's not sufficient to put them into the parent POM, because this will only be referenced with a relative path. Can one force maven to "inline" the parent POM? (Actually, only parts of the parent POM would have to be added - namely, the ones that the validation complained about)

In any case, I'd like to avoid having specific SCM tags for each artifact (that would be a hassle). Does anybody know whether a generic SCM like this will be valid and sufficient?

<scm>
    <connection>scm:git:git@github.com:jcuda</connection>
    <developerConnection>scm:git:git@github.com:jcuda</developerConnection>
    <url>git@github.com:jcuda</url>
</scm>

In doubt, I would have to create one POM for each artifact (and there are 16 artifacts right now...), each containing the same information, so that these POMs could all be added to the local repo, with 

mvn deploy:deploy-file -DpomFile=pomsForDeployment/jcuda/pom.xml     -Dfile=pomsForDeployment/jcuda//pom.xml     -Durl=file://.
... 
(The same for jcuda-natives, jcublas, jcublas-natives ....)

But that looks a bit odd...

jcuda commented 7 years ago

So it looks like a combination of dedicated POMs (that are only used for deployment) and several empty JARs might work. It seems that additional empty JARs (so called "main JARs") will be required for the *-natives artifacts.

Missing: no main jar artifact found in folder '/org/jcuda/jcublas/0/8/0/org/jcuda/jcurand-natives/0.8.0' ....

I'll try this tomorrow. Also, as seen in the folder: There's still something wrong with the path... /0/8/0 ... I think that all files have to be in the root folder of the resulting bundle.jar....

MysterionRise commented 7 years ago

Sorry, i really missed most of the fun :) What's the current state now? Anything I could help with? For parent pom, you could specify some props and then reuse them in the child pom. Makes sense?

jcuda commented 7 years ago

Now I have created 16 POMs that are only used for preparing the deployment, and that therefore contain the information that is required for Maven Central (developer, name, scm, license). They only differ in the artifact name - there must be a better solution for this...

Beyond that ... tomorrow I'll try adding the remaining (missing, empty) JARs for the *-natives artifacts, throw all this into a flat bundle.jar (i.e. without an internal directory structure), and see whether Sonatype can dissect this properly. If you have any ideas for improvements or simplifications, that would be welcome.

jcuda commented 7 years ago

Another intermediate step: It seems to basically work, but only when uploading each artifact bundle individually (so there are 16 bundles to be created and uploaded...).

But regardless of that, there has to be a different solution for the POMs.

As mentioned above: There now are 16 dedicated POMs for deployment. These contain the developer...scm information, as required by Maven central. They do not (yet) contain dependency information. This has to be added - otherwise, the most important part of maven would not work. But this would make the duplication even more ridiculous: The original POMs and the deployment POMs would then basically be the same, with these differences:

khmarbaise commented 7 years ago

I would suggest to make your parent (../jcuda-common/JCudaParentPOM/pom.xml) a real separate project (git repo) which only consist of the pom file and a README.md (maybe some other files) and put the information you need there: license, project name, url, developer list etc. plus plugin definition (pluginManagement etc. cause you are defining versions in sub projects) into it. Make a release of it . Using the usual mvn release:prepare release:perform way...afterwards you can use this parent in each of your projects and define a single version of this parent which inherits things like description, dev list etc. into your sub projects without the need to define a relativePath in parent definition In each of your sub projects of course you need to define the scm area cause it's using a different git repo so it should have different scm entry for it plus the special things for each project...

jcuda commented 7 years ago

I already considered such a "parent POM project" (and thought I had mentioned it here, but obviously did not), but hoped that this would not be necessary. It would be the third pseudoproject besides jcuda-common and jcuda-main.

In fact, I think it would conceptually make sense to declare jcuda-common as the "Parent POM project", but I'm not sure whether I'm not mixing unrelated things here just for convenience. (I think that it could later make even more sense, when the native code is added, and the jcuda-common-X.Y.Z-sources.jar could contain the source code of jcuda-common - which would otherwise appear nowhere...)

Or, maybe I'm misinterpreting this, because the intended workflow is not yet clear, e.g. for the update to 0.9.0:

?!

In any case, the release hardly can be done with mvn release. The natives have to be built separately on different platforms. Sure, it's technically possible to collect all native libraries and then create all the natives-JARs in one run during the release. But I do this for other projects, and it can really be a pain in the back. Maven is messing around with tags in the SCM, does not properly support passwords, and when something goes wrong, it always happens at the end, and the state has to be cleaned up with git reset --hard XXX+git push -f origin master ... I will not do this for 8 different repos. (In hindsight, it might have been easier to just mess everything into one, large repo, but I'm not so sure about that...)

This is also related to the last point:

In each of your sub projects of course you need to define the scm area

Is this really necessary? Right now, I intended to give every project the same SCM info:

<scm>
  <connection>scm:git:git@github.com:jcuda</connection>
  <developerConnection>scm:git:git@github.com:jcuda</developerConnection>
  <url>git@github.com:jcuda</url>
</scm>

without the actual project name. Of course, this will not work with mvn release, but ... it is not supposed to work with that (in fact, it's only there to make Maven happy...)

jcuda commented 7 years ago

So unless being told otherwise, I assume that the following would be OK:

I hope that I can make some progress here tomorrow and on friday. Any hints and comments would be welcome.

jcuda commented 7 years ago

The next step:

I still have to figure out how to proceed from there. Particularly, I'm looking for a way to create the bundle.jar files conveniently. Doing mvn deploy:deploy-file calls for all the 49 JARs seems odd (even odder than the current infrastructure that copies all these artifacts into the jcuda-main/target directory). I naively tried

mvn -Dmaven.repo.local=C:/JCudaMavenRepo clean install -U

but of course, Maven once more showed me a finger. Sometimes, it feels like it is intended to be complicated.

The natives POM executions:

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>3.0.2</version>
            <executions>

                <!-- Create the JAR containing the native libraries -->
                <execution>
                    <id>create-native-jar</id>
                    <phase>package</phase>
                    <goals>
                        <goal>jar</goal>
                    </goals>
                    <configuration>
                        <classesDirectory>nativeLibraries\${jcuda.os}\${jcuda.arch}</classesDirectory>
                        <classifier>${jcuda.os}-${jcuda.arch}</classifier>
                        <includes>
                            <include>lib/*JCublas*</include>
                            <include>lib/*JCublas2*</include>
                        </includes>
                    </configuration>
                </execution>

                <!-- Create the placeholder main JAR that contains the README.txt -->
                <execution>
                    <id>default-jar</id>
                    <phase>package</phase>
                    <goals>
                        <goal>jar</goal>
                    </goals>
                    <configuration>
                        <classesDirectory>${basedir}/../jcuda-main/resources/readme</classesDirectory>
                    </configuration>
                </execution>

                <!-- Create the JAR containing the native sources -->
                <execution>
                    <id>create-native-sources-jar</id>
                    <phase>package</phase>
                    <goals>
                        <goal>jar</goal>
                    </goals>
                    <configuration>
                        <classesDirectory>${basedir}</classesDirectory>
                        <classifier>sources</classifier>
                        <includes>
                            <include>JCublasJNI/**</include>
                            <include>JCublas2JNI/**</include>
                        </includes>
                    </configuration>
                </execution>

                <!-- Create the placeholder JAR for the javadoc that contains the README.txt -->
                <execution>
                    <id>create-native-javadoc-jar</id>
                    <phase>package</phase>
                    <goals>
                        <goal>jar</goal>
                    </goals>
                    <configuration>
                        <classifier>javadoc</classifier>
                        <classesDirectory>${basedir}/../jcuda-main/resources/readme</classesDirectory>
                    </configuration>
                </execution>

            </executions>
        </plugin>
jcuda commented 7 years ago

It seems like the jcuda-parent project (containing only the parent POM) is in fact necessary. Otherwise, the jcuda-common project could not have a proper POM file that does what it should do. (I thought it should be possible, using pluginManagement, but no avail).

So now there is the jcuda-parent project that all other projects refer to. As already mentioned in https://github.com/jcuda/jcuda-main/issues/14#issuecomment-281485064 , it seems like it is then necessary to first do a

cd jcuda-parent
mvn clean install

before the 

cd jcuda-main
mvn clean install

can be executed. The latter will then place all the JARs and POMs into the jcuda-main/output folder. (These are 100 files...). Locally, I created a .BAT file that then signs all these files with GPG, generating the corresponding .asc files, and finally packs the files together into bundle.jar files.

Eventually, it should be possible to upload these 9 bundle.jar files to Sonatype - I'll try this next week.

Until then, any feedback would be welcome.

jcuda commented 7 years ago

No feedback. OK.

The release has just been created. Iff everything worked well, then the following artifacts should be available in Maven Central soon:

<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcuda</artifactId>
    <version>0.8.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcublas</artifactId>
    <version>0.8.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcufft</artifactId>
    <version>0.8.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcusparse</artifactId>
    <version>0.8.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcusolver</artifactId>
    <version>0.8.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcurand</artifactId>
    <version>0.8.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jnvgraph</artifactId>
    <version>0.8.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcudnn</artifactId>
    <version>0.8.0</version>
</dependency>

(Now, preparing version 0.8.0b, to fix the parts that are not working as expected...)

jcuda commented 7 years ago

I just tested it locally, on Win64, by deleting m2/org/jcuda and the native DLLs in the temp folder, and it seems to work. The dependencies are resolved, the natives are unpacked, and (hoping that it also works for all other platforms), it seems like JCuda finally made its way into Maven Central.

Thanks to all supporters for your comments and hints, and particularly to @MysterionRise for setting the stage with mavenized-jcuda.

If there are any problems, comments, or suggestions for improvements, they can either be added here or in dedicated issues.

(I'll leave this issue open for a while, at least until the website and README are updated)

nakul02 commented 7 years ago

Thanks @jcuda!