Upload built artifacts to relevant repositories

[ulises]

Would it be possible to upload jar files to maven central?

[irmen]

Probably, but I've got no time to research on how to do that. Can you provide the necessary details?

[ulises]

It's a bit long, but the process is explained here: https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide

[irmen]

Not going to do that myself. Pull requests for the appropriate changes gladly accepted.

[ulises]

Fair enough. Will see if I can whip up something together.

[irmen]

Closing this as I'm not going to work on this myself. Pull requests still accepted.

[mausch]

It seems that packages are available here: http://mvnrepository.com/artifact/org.spark-project/pyrolite

[irmen]

Hm. That has not been put there by me. Also, it is quite an old version, trailing behind several releases...

[JoshRosen]

The org.spark-project.pyrolite artifact was published by the Apache Spark project, which depends on Pyrolite. We'd like to upgrade to a newer Pyrolite release, which will require a new artifact to be published. We could do this ourselves under our own namespace, but I'd prefer to help Pyrolite publish its own official Maven releases.

The biggest hassles occur up-front, after which publishing new releases is pretty easy. There's an updated guide at http://central.sonatype.org/pages/ossrh-guide.html, which has a subsection on configuring your Ant build file for compatibility with Maven Central's publishing conventions.

I'd be glad to help with whatever build changes are necessary to get this published.

(/cc @davies @mengxr @pwendell, who have some experience with the Maven Central publishing process).

[irmen]

@JoshRosen I will have a look at the guide you linked, thanks for that (but what does Sonatype have to do with maven central? I'm all new to this)

First though, the issues #19, #20 and #25 need fixing :)

[irmen]

@JoshRosen i've tried some things but could use some assistance now, because I seem to be stuck. I started with the https://github.com/irmen/serpent dependency. I transformed the java part of the project to a maven structure and created a maven pom file based on what instructions I could find. I've created an account at Sonatype. That's all good but I seem to be stuck at two things: I struggle with maven's handling of version numbers (should or should I not use a -SNAPSHOT postfix? why?) and I have troubles uploading the release to what should be sonatype's staging repository. It doesn't show up. Please see my comment in their JIRA issue https://issues.sonatype.org/browse/OSSRH-15083

[mengxr]

@irmen Public release versions should not have -SNAPSHOT suffix, which is also true for staging. You can also try publishing on bintray first, and then use their service to publish in maven central.

[irmen]

bintray looks a great deal more accessible than sonatype, I'll have a look

[mengxr]

Yes, you can then sync releases to maven central: http://blog.bintray.com/2014/02/11/bintray-as-pain-free-gateway-to-maven-central/. (But I haven't tried it.)

[irmen]

Is it really required to have Pyrolite (and Serpent) appear on maven central? Would it be enough to have them appear on bintray?

[mengxr]

bintray repos are "mutable", because the owner can delete it or change the content. That's why we cannot use it in Spark.

[irmen]

There's a little bit of progress, it seems that I finally managed to create a release on sonatype from serpent, the library pyrolite depends on. See https://github.com/irmen/Serpent/issues/6 still waiting for them to set up the sync with maven central i guess

[irmen]

serpent is on maven central now. I can now start to convert pyrolite to follow the same route

[mengxr]

Great! If we can catch the Spark 1.4 release, I will update Spark to use the official artifact.

[irmen]

When's that due?

[mengxr]

The first release candidate went out yesterday. So I would expect the final release in 1~2 weeks.

[irmen]

Pyrolite 4.6 is available on maven central under net.razorvine

Closing this issue, but I'm interested in how things go for the Spark 1.4 release!

[mengxr]

@irmen Did you publish 4.4? It is too late to change the Pyrolite version in Spark 1.4. Especially, Pyrolite 4.6 contains changes to timestamps, which requires more tests before we upgrade.

[irmen]

No, I only just converted the project to a maven structure and added a pom to publish it. I am not sure on how to proceed doing that as well in retrospect for 4.4, any tips?

[mengxr]

If the dependency didn't change since 4.4, the easiest approach would be manually editing the 4.6 pom file and packing it with the 4.4 jar.

[irmen]

I'll see what I can do later this weekend. I still have some troubles to get maven's release plugin to actually do what I want (i.e. creating, signing, and uploading the release version instead of the x.y-SNAPSHOT version of the jars). Not sure what I'm missing, but I had to mess around and trial and error a fair bit to actually get 4.6 release out :confused:

[irmen]

You are aware that the timezone support is also not yet present in Pyrolite 4.4?

I have created a pyrolite4.4-maven branch from which I created a maven release from the 4.4 version. It should be available on central shortly.

[mengxr]

Yes. I think there may exist issues with the timezone support in Pyrolite 4.6. See

https://issues.apache.org/jira/browse/SPARK-6411

But I'm not sure about the root cause. Definitely need more time to test Pyrolite 4.6. So for the Spark 1.4 release, we will use 4.4. We already switched to the official release in https://github.com/apache/spark/pull/6472. Thanks a lot for publishing releases!

[irmen]

Glad you can make use of it. One request, can you explain to me a little bit about how Pyrolite is used in Spark? I'm looking at https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals but can't really see where Pyrolite fits in.