[BigDL 2.0] Traceable CI/CD Jenkins

glorysdj commented 2 years ago

[x] Jenkins Slave from Dockerfile
[x] Jenkins Master from Dockerfile
[x] Jenkins Job to build images
[x] Jenkins Auto deployment with Docker images
[x] Jenkins Conf auto save and backup
[x] Jenkins Job auto save and backup
[x] github repo created
[x] intel-analytics/analytics-zoo-internal#34
[ ] migrate jenkins job
[ ] migrate jenkins slave node (Gondolin server)
[ ] migrate jenkins slave node (ppml/k8s/deploy/mac/clx servers/ray/nano)
[ ] release nodes backup
[ ] jenkins master backup
[ ] Automatically create jenkins jobs

pinggao187 commented 2 years ago

feature request: Jenkin config should also be control by git and go though code review process intel-analytics/analytics-zoo#411 feature request: Testing environment (dockerfile) should also be controled by git intel-analytics/analytics-zoo#412

pinggao187 commented 2 years ago

feature request: Jenkin config should also be control by git and go though code review process intel-analytics/analytics-zoo#411 feature request: Testing environment (dockerfile) should also be controled by git intel-analytics/analytics-zoo#412

request: Update jenkins docker image os to support orca openvino estimator testcase intel-analytics/analytics-zoo#416

pinggao187 commented 2 years ago

Already planned and expected to be completed in Q4. current progress: image dependencies/libs determination intel-analytics/analytics-zoo-internal#34

pinggao187 commented 2 years ago

[x] Jenkins Slave fromDockerfile(11/12(https://github.com/analytics-zoo/cicd/pull/5) add jenkins slave dockerfile intel-analytics/analytics-zoo#2

[x] Create new Jenkins and migrate jenkins jobs(12/03) Jenkins CI/CD: create new Jenkins and migrate jenkins jobs intel-analytics/analytics-zoo#13

[x] Jenkins Master from Dockerfile intel-analytics/analytics-zoo-internal#28

[x] Jenkins Job to build images(12/10)

[x] Jenkins Auto deployment with Docker images(12/10)

[x] Jenkins Conf auto save and backup(12/17)

[x] Jenkins Job auto save and backup(12/17)

[x] github repo created

[x] image dependencies/libs determination #5035

[x] jenkins NB/PR job synchronization(12/03) Jenkins CI/CD: create new Jenkins and migrate jenkins jobs intel-analytics/analytics-zoo#13

[x] chronos test docker image

[ ] migrate jenkins job

[ ] migrate jenkins slave node (Gondolin server)

[ ] migrate jenkins slave node (ppml/k8s/deploy/mac/clx servers/ray/nano)

[ ] release nodes backup

[ ] jenkins master backup

[ ] Automatically create jenkins jobs

pinggao187 commented 2 years ago

There are 7 conda envs in the jenkins test, "py36,py37,py37-horovod-pytorch,py37-horovod-tf,colabtf15,colabtf2,colab-pytorch". We plan to use requirements-XXX.txt as the dev-ops interface. each conda env corresponds to a docker image.

1.python-requirements(datails(https://github.com/analytics-zoo/cicd/pull/6)) (1).requirements-py36py37.txt (2).requirements-py37-horovod-pytorch.txt (3).requirements-py37-horovod-tf.txt (4).Requirements-colabtf1.txt (5).Requirements-colabtf2.txt (6).Requirements-colab-pytorch.txt

2.If you have other requirements besides python, please submit an issue. 3.developer update requirements process E.g: http://10.112.231.51:18888/job/BigDL-requirements-validation/

4.jenkins slave node update process

pinggao187 commented 2 years ago

Another way: create a temporary container: #4

intel-analytics / analytics-zoo

[BigDL 2.0] Traceable CI/CD Jenkins #40