search

jianyunchao / mindlearning

MindSpore learning from scratch.
0 stars 0 forks source link

【镜像】镜像制作，上传与注册 #3

Open jianyunchao opened 1 year ago

jianyunchao commented 1 year ago

场景：对mindspore版本和对应whl包的要求，需要自定义镜像，在modelarts使用 解决：使用docker制作一个自定义的镜像

流程总结

  1. 登录 docker login
  2. 拉取基础镜像 docker pull
  3. 自定义镜像，进入容器操作或者使用dockerfile ...
  4. 对自定义好的镜像提交/打标签 docker commit/tag
  5. 推送到远程仓库 docker push
jianyunchao commented 11 months ago

提前准备好安装的CANN包和对应工程文件，在当前目录下新建install_packages文件夹存放

image

编写Dockfile

# 拉取镜像
FROM swr.cn-central-221.ovaijisuan.com/aicc/mindspore_1_8_0:mindspore_1.8.0-cann_5.1.2-py_3.7-euler_2.8

# 使用华为开源镜像站提供的 pypi 配置
RUN mkdir -p /home/ma-user/.pip/

COPY --chown=ma-user:ma-group pip.conf /home/ma-user/.pip/pip.conf

# 切换root
USER root

# 更新旧包更新
RUN yum update -y

# 当前目录下的文件夹添加到镜像中
ADD ./install_packages /home/ma-user/install_packages

# 更改权限
RUN chmod +x /home/ma-user/install_packages -R

# 安装CANN包
RUN /home/ma-user/install_packages/Ascend-cann-toolkit*.run --full --force --quiet

# 修改wheel包权限
RUN chmod 777 /usr/local/Ascend/ascend-toolkit/latest/lib64/hccl-*-py3-none-any.whl
RUN chmod 777 /usr/local/Ascend/ascend-toolkit/latest/lib64/te-*-py3-none-any.whl

# 安装三方包
USER ma-user
ENV PATH=${ANACONDA_DIR}/envs/${ENV_NAME}/bin:${PATH} \
      PYTHONUNBUFFERED=1

# 更新pip本身
RUN /home/ma-user/anaconda/bin/pip install --upgrade pip
RUN /home/ma-user/anaconda/bin/pip install sympy --force-reinstall --user
RUN /home/ma-user/anaconda/bin/pip install /usr/local/Ascend/ascend-toolkit/latest/lib64/te-*-py3-none-any.whl --force-reinstall --user
RUN /home/ma-user/anaconda/bin/pip install /usr/local/Ascend/ascend-toolkit/latest/lib64/hccl-*-py3-none-any.whl --force-reinstall --user

# 升级cv库
RUN /home/ma-user/anaconda/bin/pip uninstall opencv-python -y
RUN /home/ma-user/anaconda/bin/pip uninstall opencv-contrib-python -y
RUN /home/ma-user/anaconda/bin/pip install -r /home/ma-user/install_packages/videocomposer.txt -i https://pypi.org/simple
RUN /home/ma-user/anaconda/bin/pip install /home/ma-user/install_packages/motion_vector_extractor-1.0.6-cp37-cp37m-linux_aarch64.whl

# ocr缺少依赖拷贝
RUN cp /usr/lib64/libgeos_c.so /home/ma-user/anaconda3/envs/MindSpore/lib/

# 删除基础镜像的mindspore包
RUN /home/ma-user/anaconda/bin/pip uninstall mindspore-ascend -y

# 安装对应版本mindspore
RUN /home/ma-user/anaconda/bin/pip install https://ms-release.obs.cn-north-4.myhuaweicloud.com/2.1.0/MindSpore/unified/aarch64/mindspore-2.1.0-cp37-cp37m-linux_aarch64.whl --trusted-host ms-release.obs.cn-north-4.myhuaweicloud.com -i https://pypi.tuna.tsinghua.edu.cn/simple

# 设置环境变量
# control log level. 0-DEBUG, 1-INFO, 2-WARNING, 3-ERROR, 4-CRITICAL, default level is WARNING.
ENV GLOG_v=2
# Conda environmental options
ENV LOCAL_ASCEND=/usr/local/Ascend
# lib libraries that the run package depends on
ENV LD_LIBRARY_PATH=${LOCAL_ASCEND}/ascend-toolkit/latest/lib64:${LOCAL_ASCEND}/driver/lib64:${LOCAL_ASCEND}/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling:${LD_LIBRARY_PATH}
# Environment variables that must be configured
## TBE operator implementation tool path
ENV TBE_IMPL_PATH=${LOCAL_ASCEND}/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe
## OPP path
ENV ASCEND_OPP_PATH=${LOCAL_ASCEND}/ascend-toolkit/latest/opp
## AICPU path
ENV ASCEND_AICPU_PATH=${ASCEND_OPP_PATH}/..
## TBE operator compilation tool path
ENV PATH=${LOCAL_ASCEND}/ascend-toolkit/latest/compiler/ccec_compiler/bin/:${PATH}
## Python library that TBE implementation depends on
ENV PYTHONPATH=${TBE_IMPL_PATH}:${PYTHONPATH}

# 切换root，删除临时文件
USER root
RUN rm -rf /home/ma-user/install_packages
RUN echo 'source /usr/local/Ascend/ascend-toolkit/set_env.sh' >> ~/.bashrc

# 切换ma-user
USER ma-user

# 设定默认的工作目录，和modelarts的默认工作目录保持一致
WORKDIR /home/ma-user
jianyunchao commented 11 months ago

根据Dockerfile构建镜像

docker build . -t swr.cn-east-324.fjaicc.com/mindspore-lab/videocomposer-ms-py:dev-2.1.0-3.7.10
jianyunchao commented 11 months ago

构建生成好镜像后，使用平台提供的登录命令登录，并推送镜像

docker login -u c***K -p a***8 s***c.com
docker push swr.cn-east-324.fjaicc.com/mindspore-lab/videocomposer-ms-py:dev-2.1.0-3.7.10