Add package to Maven central

[nfx]

As a person, who configures clusters regularly, i'd expect for this package to be published on maven central. There's also a need to publish artifacts for every different version of spark. e.g. 3.0, 3.1, 3.2 and etc.

[choukimasri]

Hey !, thanks for your contributions, @atoakley @nithinpnp, we're also affected by this, any info on if this made it to your roadmap ?

@nfx thanks for bringing up the issue.

[hallihan]

I have added this request to our internal backlog, but the work needed to publish and maintain artifacts on Maven Central is not currently on our roadmap (Microsoft Internal Workitem: ADO#413227). This may be an area where the community could contribute, please check out the LICENSE and CONTRIBUTING.md files for more details.

[nfx]

@hallihan community cannot contribute to this one :)

[choukimasri]

@hallihan thank you !, another option would be to use pypi package format to provide easy distribution, please consider one of the different formats to distribute this library, proper package managment is a deal breaker on every dependency of this sort. a pull request maybe able to address this, but you will need to maintain the upload script somewhere by your team.

[nfx]

@choukimasri pypi package won't work with jar's :)

[choukimasri]

@nfx who cares about jars?, databricks allows fetching pypi packages in cluster configurations, this would be a proper alt to maven too. infact any of these formats is a proper alternative :

{
  "cluster_id": "10201-my-cluster",
  "libraries": [
    {
      "maven": {
        "coordinates": "org.jsoup:jsoup:1.7.2",
        "exclusions": ["slf4j:slf4j"]
      }
    },
    {
      "pypi": {
        "package": "simplejson",
        "repo": "https://my-pypi-mirror.com"
      }
    },
    {
      "cran": {
        "package": "ada",
        "repo": "https://cran.us.r-project.org"
      }
    }
  ]
}

[nfx]

@choukimasri this project is written in java, therefore jars, not pypi :)

[istairbn]

Plus one for this. If it's not "built in" we should definitely have them in an accessible location.