search

criteo / Spark-RSVD

Randomized SVD of large sparse matrices on Spark
Apache License 2.0
77 stars 22 forks source link

Update to spark-3 and scala-2.12 #5

Closed jobar closed 3 years ago

jobar commented 3 years ago

We at Wikimedia Foundation Data Engineering team use this library, and we're migrating to Spark 3 soon. The changes are minimal for the tests to pass :)

vincent-grosbois commented 3 years ago

Hi @jobar , thanks for the PR ! I'm guessing this should probably merged into a new "spark 3" branch though?

jobar commented 3 years ago

A new spark-3 branch would be perfect indeed! Thank you for the fast response :)

jobar commented 3 years ago

And while at it I wonder - Would you release a 1.0_spark3 version of the lib onto Maven? This would make our lives incredibly easier (no need to fork internally). I know it's a burden, so please be frank :)

alois-bissuel commented 3 years ago

Thanks for the PR @jobar! Regarding the branch issue, we publish artifacts only for the master branch (and I would rather want not to meddle with publishing again), thus we will merge it into the master branch. We will simply tag your commit with version 1.1 (no API change...). What do you think?

jobar commented 3 years ago

This would be perfect for us :) Thank you a lot for offering!

alois-bissuel commented 3 years ago

On the publishing side, it seems travis-CI.org is dead, and thus we have to use github CI. I will try to set it up on Monday.

jobar commented 3 years ago

Hello RSVD folks, Would you by any chance have news on the publishing front? Many thanks

alois-bissuel commented 3 years ago

Alas, no news on it there. We are awaiting internally the access to the sonatype password (for publishing). I will keep you updated.

alois-bissuel commented 3 years ago

Hello, I did the necessary work to have the package published on Nexus. Could you confirm it works for you?

jobar commented 3 years ago

Hi, We have used maven-central for the package so far - if your nexus instance is open it could work I assume.

alois-bissuel commented 3 years ago

I find it in https://oss.sonatype.org, but not on https://search.maven.org. Maybe maven-central will be updated soon (I have no idea if there is a link between the two).

jobar commented 3 years ago

Indeed, I don't see it in https://mvnrepository.com/artifact/com.criteo/rsvd I'm gonna wait a bit and see if it gets automagically imported. Otherwise we'll use https://oss.sonatype.org :)

alois-bissuel commented 3 years ago

I'll check internally if this is not the case.

jobar commented 3 years ago

I confirm we have access to the artifact through OSS - Many thanks for the release!