search

tensorflow / ecosystem

Integration of TensorFlow with other open-source frameworks
Apache License 2.0
1.37k stars 392 forks source link

Contribute spark-tensorflow-distributor to the ecosystem #154

Closed sarthfrey closed 4 years ago

sarthfrey commented 4 years ago

This PR aims to act as both a proposal and an initial version for a contribution of the spark-tensorflow-distributor python package. As mentioned in tensorflow#151 the general mandate of this package is to make it easier for users to do distributed training with TensorFlow 2 on their Spark clusters. Currently this package primarily acts as a job launcher for starting TensorFlow servers, configuring GPU and CPU resources for the user based on Spark resource scheduling so that they may easily run their deep learning workloads.

This PR also includes CI with GitHub workflows, which acts at the repository level by default. However, the CI is set up so that the checks will only be triggered by changes to this package's subdirectory in the ecosystem. This behavior is described in .github/workflows/spark-tensorflow-distributor.yml.

I'd also like to publish this package to PyPi and am wondering if there's an ecosystem specific process for that.

Welcoming any and all feedback on this PR :)

cc @guptapriya @mengxr @husseinnagr-db

guptapriya commented 4 years ago

cc @yuefengz can you help review this PR?

ghost commented 4 years ago

@yuefengz ping :)

mengxr commented 4 years ago

@jhseu What is your recommended CI solution? This PR uses github workflow. Is it okay?

mengxr commented 4 years ago

Ping @jhseu and @yuefengz again for CI choices.

jhseu commented 4 years ago

For CI, GitHub workflow is fine for now, I think. We haven't setup an alternative for this repo.

sarthfrey commented 4 years ago

Thanks @yuefengz! I addressed all the feedback. @jhseu are we good to merge?

sarthfrey commented 4 years ago

Made a release here: https://pypi.org/project/spark-tensorflow-distributor/0.0.3/

ghost commented 4 years ago

@guptapriya @yuefengz @jhseu also please let me know who from TensorFlow I should add as an owner on the PyPi project :)