search

apache / singa

a distributed deep learning platform
Apache License 2.0
3.37k stars 1.25k forks source link

Release, versioning and continous integration #585

Open nudles opened 4 years ago

nudles commented 4 years ago
  1. We are following the semantic versioning scheme (i.e., X.Y.Z) , which means every hotfix will bump the version number to X.Y.(Z+1).
  2. To support it, we need continuous integration which tracks the master branch to update the version number and generate the conda packages.
  3. For updates to the dev branch, we also need the continuous integration tool to run the test and generate the nightly builds (conda packages).
  4. Currently, we have Travis (for CPU) and Jenkins (CPU). They need to be updated to support 1, 2 and 3.
dcslin commented 4 years ago

Hi @nudles , is it possible to access the machine running jenkins?

dcslin commented 4 years ago

I have studied how fastai release process works, and semantic versioning and etc.

In my opinion, this topic involves 2 roles:

  1. developers code, commit, and send PR to singa:master or singa:dev. For each PR, CI compiles, builds(conda), tests and returns pass/fail.

  2. supervisors make release, and decide if the release is a patch, minor, major. There should be some automation here, supervisors then can just run command like $ make release-patch or $ make release-minor.

The automation should do a) checkout to latest singa:master (the code merged in latest master should pass CI) b) define new_version <-- semantic versioning c) git tag new_version && git push new_version d) build anaconda package and upload to anaconda cloud

If this approach is ok, then we need to add automation part.

dcslin commented 4 years ago

Hi @nudles , how release is done currently? is this flag "TRAVIS_SECURE_ENV_VARS" (tool/travis/build.sh line 41) used?

if [[ "$TRAVIS_SECURE_ENV_VARS" == "false" ]];
  # install and run unittest
then
  echo "no uploading if ANACONDA_UPLOAD_TOKEN not set"
else
  # turn off debug to hide the token in travis log
  set +x
  # upload the package onto anaconda cloud
  anaconda -t $ANACONDA_UPLOAD_TOKEN upload -u $USER -l main $CONDA_BLD_PATH/$OS/singa-*.tar.bz2 --force
fi
nudles commented 4 years ago

Yes. Currently, we use Travis to build CPU package and Jenkins to build GPU package. If the TRAVIS_SECURE_ENV_VARS is available, the package will be uploaded to the cloud.

dcslin commented 4 years ago

Yes. Currently, we use Travis to build CPU package and Jenkins to build GPU package. If the TRAVIS_SECURE_ENV_VARS is available, the package will be uploaded to the cloud.

Currently, how is the "release" build triggered or how is the Env Var TRAVIS_SECURE_ENV_VARS set? for travis is it done by web UI? https://blog.travis-ci.com/2017-08-24-trigger-custom-build

nudles commented 4 years ago

The travis for apache/singa does not upload the package because the variable is not set. In jenkins, we sync the master branch to nusdbsystem/singa, whose travis will do the building and send the package to the cloud.

dcslin commented 4 years ago

Hi @nudles , could you kindly advise if this PR caters this issue? https://github.com/apache/singa/pull/621 usage: https://github.com/apache/singa/blob/d25539e16b6f3551ad6b1e2a74713025e150d30c/tool/release/README.md

nudles commented 4 years ago
  1. if the building of the conda package fails, we need to delete the newly updated tag? Alternatively, we may need to consider a pre-release version like 3.0.0-alpha0. Then, we need to parse the version ID with -alphaX.
  2. both the CMake and Conda building scripts need to parse the version from the git tag.
  3. The CI needs to monitor both the dev and master branch. Only when the tag of the master branch changes, it uploads the pacakges (and docker images?).
moazreyad commented 4 years ago
  1. if the building of the conda package fails, we need to delete the newly updated tag? Alternatively, we may need to consider a pre-release version like 3.0.0-alpha0. Then, we need to parse the version ID with -alphaX.
  2. both the CMake and Conda building scripts need to parse the version from the git tag.
  3. The CI needs to monitor both the dev and master branch. Only when the tag of the master branch changes, it uploads the pacakges (and docker images?).

All of this can be easily done with Github Actions. The CI in Github can be configured to monitor a given branch. There are many actions available for use that manage the version tags, create docker containers and so on.

dcslin commented 4 years ago

Hi @moazreyad , thank you for the advice, I will study how github actions works.

dcslin commented 4 years ago
  1. if the building of the conda package fails, we need to delete the newly updated tag? Alternatively, we may need to consider a pre-release version like 3.0.0-alpha0. Then, we need to parse the version ID with -alphaX.
  2. both the CMake and Conda building scripts need to parse the version from the git tag.
  3. The CI needs to monitor both the dev and master branch. Only when the tag of the master branch changes, it uploads the pacakges (and docker images?).

Regarding 1, after comparing other framework, like tf.

For torch, similarily:

By right, simply speaking, alpha is first phase of testing internally. beta is testing publicly. rc is almost like stable release unless serious bug is discovered.

nudles commented 4 years ago

Since everything here is public, we do not have an internal testing phase. I suggest skipping alpha and beta versions.

On Fri, Mar 13, 2020 at 11:03 AM Shicong notifications@github.com wrote:

  1. if the building of the conda package fails, we need to delete the newly updated tag? Alternatively, we may need to consider a pre-release version like 3.0.0-alpha0. Then, we need to parse the version ID with -alphaX.
  2. both the CMake and Conda building scripts need to parse the version from the git tag.
  3. The CI needs to monitor both the dev and master branch. Only when the tag of the master branch changes, it uploads the pacakges (and docker images?).

Regarding 1, after comparing other framework, like tf.

  • For major release, they have alpha and beta, then followed by rc, release candidate.
  • And for minor, pre release starts from rc.
  • While patch will be release directly.

For torch, similarily:

  • major starts from alpha, followed by rc
  • minor starts from rc
  • patch is release directly.

By right, simply speaking, alpha is first phase of testing internally. beta is testing publicly. rc is almost like stable release unless serious bug is discovered.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/singa/issues/585#issuecomment-598525352, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA47DRZC7TN6ADQQCSJLYSDRHGO6DANCNFSM4KNSRRIQ .

nudles commented 4 years ago
  1. if the building of the conda package fails, we need to delete the newly updated tag? Alternatively, we may need to consider a pre-release version like 3.0.0-alpha0. Then, we need to parse the version ID with -alphaX.
  2. both the CMake and Conda building scripts need to parse the version from the git tag.
  3. The CI needs to monitor both the dev and master branch. Only when the tag of the master branch changes, it uploads the pacakges (and docker images?).

All of this can be easily done with Github Actions. The CI in Github can be configured to monitor a given branch. There are many actions available for use that manage the version tags, create docker containers and so on.

Is it possible to build the website via Github Action? We need the following steps

  1. when there is a new commit to singa-doc, we build the website to generate the html files
  2. push the html files to singa-site as a new commit