search

apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.77k stars 6.8k forks source link

[RFC] MXNet 2.0 JVM Language development #17783

Open lanking520 opened 4 years ago

lanking520 commented 4 years ago

Since MXNet 2.0 development starts. I would like to initiate a discussion for the future development of JVM languages.

Proposal

  1. Start cleaning on the existing APIs to adapt to 2.0
  2. Start from ground up to rewrite the whole Scala/Java APIs
  3. Start using DJL (djl.ai) as a frontend for MXnet JVM development
  4. Using DJL MXNet JNA as the low level API
  5. Use MXNet JavaCpp as the low level API
  6. (Feel free to add more...)

Statistics

Scala package

scala-mxnet

Clojure package

clojure-mxnet-downloads

@gigasquid @terrytangyuan @zachgk @frankfliu @aaronmarkham

leezu commented 3 years ago

Yes, that's not a problem. However, if we don't have libquadmath.so, libgfortran.so isn't going to load, so is it still useful?

There haven't been any ABI breaks in libquadmath.so. Thus we can simply ask users to install libquadmath.so by themselves and everything will work. Our users will not be able to find an incompatible libquadmath.so

FWIW, it looks to me like libquadmath is LGPL, not GPL: https://github.com/gcc-mirror/gcc/blob/master/libquadmath/COPYING.LIB

You're right, but the consequence is the same.

It doesn't sound to me like they forbid publishing snapshots, just that it shouldn't be documented, which is weird, but whatever. It should be alright to deploy snapshots and keep it a "secret", no?

Yes. Reading the https://infra.apache.org/publishing-maven-artifacts.html again, there is also https://repository.apache.org/snapshots which may be the best location for snapshots?

They say we "should" do this and that, but if none of their services offers support for Maven artifacts, I suppose this means we can use something else, right?

I'm not sure what you mean.

saudet commented 3 years ago

Yes, that's not a problem. However, if we don't have libquadmath.so, libgfortran.so isn't going to load, so is it still useful?

There haven't been any ABI breaks in libquadmath.so. Thus we can simply ask users to install libquadmath.so by themselves and everything will work. Our users will not be able to find an incompatible libquadmath.so

FWIW, it looks to me like libquadmath is LGPL, not GPL: https://github.com/gcc-mirror/gcc/blob/master/libquadmath/COPYING.LIB

You're right, but the consequence is the same.

I suppose it's more business friendly, but it's not a requirement for releasing binaries under Apache, correct? That is, this is a policy specific to the MXNet project?

It doesn't sound to me like they forbid publishing snapshots, just that it shouldn't be documented, which is weird, but whatever. It should be alright to deploy snapshots and keep it a "secret", no?

Yes. Reading the https://infra.apache.org/publishing-maven-artifacts.html again, there is also https://repository.apache.org/snapshots which may be the best location for snapshots?

Yes, I saw that too. There's already a few snapshots from MXNet there, so I assume we can use it freely: http://repository.apache.org/content/groups/snapshots/org/apache/mxnet/ (Oh and look at that, the artifacts are using a generic "-gpu" string as part of their names. I would like to consult with the authors of those artifacts to see what they think of adding versions here before committing to tracking specific versions of CUDA. It's not something I would personally relish to do, until someone actually cares about it at least that is. The only person that ever complained about this for OpenCV is @ericxsun: https://github.com/bytedeco/javacpp-presets/issues/918 )

Now, where can I get an account for that server... Anyway, someone will just need to put their credentials as secrets for GitHub Actions and then we just need to change the URL here for the snapshots: https://github.com/saudet/incubator-mxnet/blob/add-javacpp/java/build.gradle#L143

And that's it. They will appear exactly as they currently are on Sonatype: https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/mxnet/2.0-SNAPSHOT/

They say we "should" do this and that, but if none of their services offers support for Maven artifacts, I suppose this means we can use something else, right?

I'm not sure what you mean.

I was referring to the snapshot repository, which they do offer, so we're good for that, but if we need something else, it would be good to know what the official stance is concerning the use of external services. I suppose anything from GitHub is OK, but other than that, I wonder.

leezu commented 3 years ago

I suppose it's more business friendly, but it's not a requirement for releasing binaries under Apache, correct? That is, this is a policy specific to the MXNet project?

It is a requirement to release the binaries under AL2. You can refer to https://www.apache.org/legal/resolved.html for a list of compatible and incompatible licenses. LGPL is Category-X (not allowed) as it places restrictions on the larger work.

Oh and look at that, the artifacts are using a generic "-gpu" string as part of their names. I would like to consult with the authors of those artifacts to see what they think of adding versions here before committing to tracking specific versions of CUDA. It's not something I would personally relish to do, until someone actually cares about it at least that is.

It's fine with me to re-use the existing pattern if others don't mind.

Now, where can I get an account for that server... Anyway, someone will just need to put their credentials as secrets for GitHub Actions and then we just need to change the URL here for the snapshots:

We can open a ticket with Apache Infra. Would you like to open a PR first?

it would be good to know what the official stance is concerning the use of external services. I suppose anything from GitHub is OK, but other than that, I wonder.

It's fine to use external services as long as the project maintainers (PPMC) control the usage and the published artifacts are compliant with the ASF polices (for example, don't contain LGPL compontents).

saudet commented 3 years ago

Ok, I've finally updated my fork accordingly along with a few additional changes: https://github.com/saudet/incubator-mxnet/commit/09668180fe81da96ed2baee26c373fdcd343cdec If that looks good enough for a pull request, I'll do that right away! @leezu

leezu commented 3 years ago

Thank you. It's fine with me. Once you open the PR, @lanking520 and @gigasquid may be able to review too