4080训练，经常意外中断

hhh151671 commented 1 year ago

bug描述 Describe the Bug

配置信息： GPU: 4080 cuda11.3 cudnn8.9 驱动：530.41.03 paddle：2.4.2

在2080TI上训练PPMatting，可以正常训练；但同样参数换到4080上，经常会因为各种问题导致训练中断： 1.直接卡住，而且显存占用接近14/16G，显存占比却是0 2.

  File "/home/cv/anaconda3/envs/py10/lib/python3.10/site-packages/paddle/optimizer/optimizer.py", line 754, in _add_accumulator
    var_name in self._accumulators_holder
AssertionError: Optimizer set error, batch_norm2d_177.b_0_velocity_1 should in state dict

3.

  File "/home/cv/anaconda3/envs/py10/lib/python3.10/site-packages/paddle/fluid/dygraph/varbase_patch_methods.py", line 297, in backward
    core.eager.run_backward([self], grad_tensor, retain_graph)
OSError: (External) CUDNN error(4), CUDNN_STATUS_INTERNAL_ERROR.
  [Hint: Please search for the error code(4) on website (https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnStatus_t) to get Nvidia's official solution and advice about CUDNN Error.] (at /paddle/paddle/phi/kernels/gpudnn/conv_grad_kernel.cu:583)

4.


--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
No stack trace in paddle, may be caused by external reasons.

----------------------
Error Message Summary:
----------------------
FatalError: `Segmentation fault` is detected by the operating system.
  [TimeInfo: *** Aborted at 1685437112 (unix time) try "date -d @1685437112" if you are using GNU date ***]
  [SignalInfo: *** SIGSEGV (@0x0) received by PID 3633 (TID 0x7fb6c2a49280) from PID 0 ***]

...

其他补充信息 Additional Supplementary Information

No response

zyfncg commented 1 year ago

从2080TI换到4080，除了显卡其他环境配置都是一样的吗？

hhh151671 commented 1 year ago

从2080TI换到4080，除了显卡其他环境配置都是一样的吗？

只有驱动不一样530.41.03，4080是 2080Ti是 465.19.01 ；剩下的cuda，cudnn版本都是一样的，训练的配置文件也是一样的

zyfncg commented 1 year ago

是使用的docker容器吗？

hhh151671 commented 1 year ago

是使用的docker容器吗？

不是，是在ubuntu22.04桌面端跑的

zyfncg commented 1 year ago

好的，可以提供下能复现问题的训练脚本吗？

hhh151671 commented 1 year ago

好的，可以提供下能复现问题的训练脚本吗？

训练脚本：

WORK_DIR=$(cd $(dirname $0); pwd)

cd ..
python label/update_txt_paths.py \
--data_dir ${WORK_DIR}/data

cd /home/cv/workstation/FrameM/PaddleSeg/Matting

config=${WORK_DIR}/ppmatting-hrnet_w48-distinctions.yml
output=${WORK_DIR}/output

python -u tools/train.py \
--config ${config} \
--do_eval  \
--use_vdl  \
--save_interval 500  \
--save_dir ${output}   \
--num_workers 0 \
--resume_model  ${output}/iter_6500/

配置文件：yaml

batch_size: 2
iters: 300000

train_dataset:
  type: MattingDataset
  dataset_root: /home/cv/workstation/segscripts/board_seg/data
  train_file: train.txt
  transforms:
    - type: LoadImages
    - type: Padding
      target_size: [512, 512]
    - type: RandomCrop
      crop_size: [[512, 512],[640, 640], [800, 800]]
    - type: Resize
      target_size: [512, 512]
    # - type: Resize
    #   target_size: [800, 800]
    - type: RandomDistort
    - type: RandomBlur
      prob: 0.1
    - type: RandomHorizontalFlip
    - type: RandomVerticalFlip
    - type: Normalize
  mode: train
  separator: '|'

val_dataset:
  type: MattingDataset
  dataset_root: /home/cv/workstation/segscripts/board_seg/data
  val_file: val.txt
  transforms:
    - type: LoadImages
    - type: LimitShort
      max_short: 1536
    # - type: Resize
    #   target_size: [800, 800]

    - type: ResizeToIntMult
      mult_int: 32
    - type: Normalize
  mode: val
  get_trimap: False
  separator: '|'

model:
  type: PPMatting
  backbone:
    type: HRNet_W48
    pretrained: https://bj.bcebos.com/paddleseg/dygraph/hrnet_w48_ssld.tar.gz
  pretrained: Null

optimizer:
  type: sgd
  momentum: 0.9
  weight_decay: 4.0e-5

lr_scheduler:
  type: PolynomialDecay
  learning_rate: 0.005
  # learning_rate: 0.001
  end_lr: 0
  power: 0.9

zyfncg commented 1 year ago

收到，由于我们这边目前没有4080的显卡，分析可能需要一段时间，如果有结论或者解决方案会在issue里及时同步

hhh151671 commented 1 year ago

收到，由于我们这边目前没有4080的显卡，分析可能需要一段时间，如果有结论或者解决方案会在issue里及时同步

好的，非常感谢！

alita-moore commented 1 year ago

I am experiencing the same issue with the following dockerfile: (我在使用以下的 Dockerfile 时遇到了同样的问题：)

###########
# Cuda Setup
###########

# Use the CUDA 11.0.3 devel image
FROM nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04
ENV DEBIAN_FRONTEND noninteractive \
    TZ=America/Chicago

###########
# System Setup
###########

# Initial setup: installing python3-pip and upgrading pip
RUN sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && apt-get update && apt-get install -y python3-pip && pip3 install --upgrade pip

# Remove any third-party apt sources to avoid issues with expiring keys.
# Install some basic utilities
RUN apt-get upgrade -y && apt-get install -y \
    ca-certificates \
    sudo \
    git \
    git-lfs \
    zip \
    unzip \
    htop \
    bzip2 \
    libx11-6 \
    build-essential \
    libsndfile-dev \
    software-properties-common \
    curl \
    tree \
    iputils-ping \
    vim \
    jq \
    libatomic1 \
    netcat \
    pciutils \
    wget \
    zsh \
    libgl1-mesa-glx \
    && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && apt-get update \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

###########
# VSCode Server Setup
###########

ARG BUILD_DATE
ARG VERSION
ARG CODE_RELEASE
RUN \
  echo "**** install openvscode-server runtime dependencies ****" && \
  sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && apt-get update && \
  echo "**** install openvscode-server ****" && \
  if [ -z ${CODE_RELEASE+x} ]; then \
    CODE_RELEASE=$(curl -sX GET "https://api.github.com/repos/gitpod-io/openvscode-server/releases/latest" \
      | awk '/tag_name/{print $4;exit}' FS='[""]' \
      | sed 's|^openvscode-server-v||'); \
  fi && \
  mkdir -p /app/openvscode-server && \
  curl -o \
    /tmp/openvscode-server.tar.gz -L \
    "https://github.com/gitpod-io/openvscode-server/releases/download/openvscode-server-v${CODE_RELEASE}/openvscode-server-v${CODE_RELEASE}-linux-x64.tar.gz" && \
  tar xf \
    /tmp/openvscode-server.tar.gz -C \
    /app/openvscode-server/ --strip-components=1 && \
  echo "**** clean up ****" && \
  apt-get clean && rm -rf \
    /tmp/* \
    /var/lib/apt/lists/* \
    /var/tmp/*

# Contains information that vscode relies on
COPY root/ /

RUN curl -sL https://deb.nodesource.com/setup_14.x  | bash - && \
    apt-get install -y nodejs && \
    npm install -g configurable-http-proxy

#############
# Setup Environment + User
#############

# Setup Environment
ENV HOME=/home/user \
    CONDA_AUTO_UPDATE_CONDA=false

# Create a user named "user" with absolute power and make it default
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
 && chown -R user:user /app
RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers
USER user

# Setup working directory for the new user
WORKDIR $HOME

RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
    bash ~/miniconda.sh -b -p $HOME/miniconda && \
    rm ~/miniconda.sh 

# Add Miniconda to PATH
ENV PATH="/home/user/miniconda/bin:${PATH}"

# Set permissions (do this last)
RUN chown -R user:user $HOME && chmod -R 777 $HOME

# Switch to the user
USER user

# User the home directory and copy over the files on the Dockerfile level
COPY --chown=user:user . .

# Make the start_server (what starts the vscode server) executable
RUN chmod +x start_server.sh

#############
# Project Specific Configurations
#############

# Create a Conda environment with Python 3.7
RUN conda create --name paddle_env python=3.8 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/ # This is a one line command

# Set environment path
ENV CONDA_PATH="${HOME}/miniconda/envs/paddle_env/bin"
ENV PATH="${CONDA_PATH}:${PATH}"

# Activate the Conda environment
RUN echo "source activate paddle_env" > ~/.bashrc

# Install pip, NumPy, and PyTorch in the Conda environment
RUN conda install -n paddle_env pip numpy cudatoolkit=10.2

RUN $CONDA_PATH/python -m pip install paddlepaddle-gpu -i https://pypi.tuna.tsinghua.edu.cn/simple

RUN $CONDA_PATH/pip install "paddleocr>=2.0.1"

#############

USER root

# User Debian packages
## Security warning : Potential user code executed as root (build time)
RUN --mount=target=/root/packages.txt,source=packages.txt \
    apt-get update && \
    xargs -r -a /root/packages.txt apt-get install -y --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

RUN --mount=target=/root/on_startup.sh,source=on_startup.sh,readwrite \
    bash /root/on_startup.sh

#############
# Final Startup
#############
EXPOSE 7860 3000

# Makes sure that the user has access to the files
USER user
RUN chown -R user:user $HOME && chmod -R 777 $HOME
# Add a new command to the Dockerfile to start the VSCode Server
CMD ["/bin/bash", "-c", "echo 'Starting VSCode Server...' && exec /app/openvscode-server/bin/openvscode-server --host 0.0.0.0 --port 7860 --without-connection-token \"${@}\" --"]

I am running this on T4 on Hugging Face spaces. (我正在Hugging Face spaces上使用T4运行这个。)

Full debug output: (完整调试输出：)

[2023/06/13 01:18:04] ppocr DEBUG: Namespace(alpha=1.0, benchmark=False, beta=1.0, cls_batch_num=6, cls_image_shape='3, 48, 192', cls_model_dir=None, cls_thresh=0.9, cpu_threads=10, crop_res_save_dir='./output', det=True, det_algorithm='DB', det_box_type='quad', det_db_box_thresh=0.6, det_db_score_mode='fast', det_db_thresh=0.3, det_db_unclip_ratio=1.5, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_east_score_thresh=0.8, det_limit_side_len=960, det_limit_type='max', det_model_dir='/home/user/.paddleocr/whl/det/ch/ch_PP-OCRv3_det_infer', det_pse_box_thresh=0.85, det_pse_min_area=16, det_pse_scale=1, det_pse_thresh=0, det_sast_nms_thresh=0.2, det_sast_score_thresh=0.5, draw_img_save_dir='./inference_results', drop_score=0.5, e2e_algorithm='PGNet', e2e_char_dict_path='./ppocr/utils/ic15_dict.txt', e2e_limit_side_len=768, e2e_limit_type='max', e2e_model_dir=None, e2e_pgnet_mode='fast', e2e_pgnet_score_thresh=0.5, e2e_pgnet_valid_set='totaltext', enable_mkldnn=False, fourier_degree=5, gpu_mem=500, help='==SUPPRESS==', image_dir=None, image_orientation=False, ir_optim=True, kie_algorithm='LayoutXLM', label_list=['0', '180'], lang='ch', layout=True, layout_dict_path='/home/user/miniconda/envs/paddle_env/lib/python3.8/site-packages/paddleocr/ppocr/utils/dict/layout_dict/layout_cdla_dict.txt', layout_model_dir='/home/user/.paddleocr/whl/layout/picodet_lcnet_x1_0_fgd_layout_cdla_infer', layout_nms_threshold=0.5, layout_score_threshold=0.5, max_batch_size=10, max_text_length=25, merge_no_span_structure=True, min_subgraph_size=15, mode='structure', ocr=True, ocr_order_method=None, ocr_version='PP-OCRv3', output='./output', page_num=0, precision='fp32', process_id=0, re_model_dir=None, rec=True, rec_algorithm='SVTR_LCNet', rec_batch_num=6, rec_char_dict_path='/home/user/miniconda/envs/paddle_env/lib/python3.8/site-packages/paddleocr/ppocr/utils/ppocr_keys_v1.txt', rec_image_inverse=True, rec_image_shape='3, 48, 320', rec_model_dir='/home/user/.paddleocr/whl/rec/ch/ch_PP-OCRv3_rec_infer', recovery=False, save_crop_res=False, save_log_path='./log_output/', scales=[8, 16, 32], ser_dict_path='../train_data/XFUND/class_list_xfun.txt', ser_model_dir=None, show_log=True, sr_batch_num=1, sr_image_shape='3, 32, 128', sr_model_dir=None, structure_version='PP-StructureV2', table=True, table_algorithm='TableAttn', table_char_dict_path='/home/user/miniconda/envs/paddle_env/lib/python3.8/site-packages/paddleocr/ppocr/utils/dict/table_structure_dict_ch.txt', table_max_len=488, table_model_dir='/home/user/.paddleocr/whl/table/ch_ppstructure_mobile_v2.0_SLANet_infer', total_process_num=1, type='ocr', use_angle_cls=False, use_dilation=False, use_gpu=True, use_mp=False, use_npu=False, use_onnx=False, use_pdf2docx_api=False, use_pdserving=False, use_space_char=True, use_tensorrt=False, use_visual_backbone=True, use_xpu=False, vis_font_path='./doc/fonts/simfang.ttf', warmup=False)

--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
No stack trace in paddle, may be caused by external reasons.

----------------------
Error Message Summary:
----------------------
FatalError: `Segmentation fault` is detected by the operating system.
  [TimeInfo: *** Aborted at 1686619090 (unix time) try "date -d @1686619090" if you are using GNU date ***]
  [SignalInfo: *** SIGSEGV (@0x0) received by PID 199 (TID 0x7ff8799063c0) from PID 0 ***]

alita-moore commented 1 year ago

it seems to work with this, but for some reason it's not using the gpu. (这个好像有效，但是出于某种原因它并没有使用GPU。)

###########
# Cuda Setup
###########

# Use the CUDA 10.1 devel image
FROM nvidia/cuda:11.7.1-cudnn8-devel-ubuntu22.04
ENV DEBIAN_FRONTEND noninteractive \
    TZ=America/Chicago

###########
# System Setup
###########

# Initial setup: installing python3-pip and upgrading pip
RUN sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && apt-get update && apt-get install -y python3-pip && pip3 install --upgrade pip

# Remove any third-party apt sources to avoid issues with expiring keys.
# Install some basic utilities
RUN sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && apt-get update && apt-get upgrade -y && apt-get install -y \
    ca-certificates \
    sudo \
    git \
    git-lfs \
    zip \
    unzip \
    htop \
    bzip2 \
    libx11-6 \
    build-essential \
    libsndfile-dev \
    software-properties-common \
    curl \
    tree \
    iputils-ping \
    vim \
    jq \
    libatomic1 \
    netcat \
    pciutils \
    wget \
    zsh \
    libgl1-mesa-glx \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

###########
# VSCode Server Setup
###########

ARG BUILD_DATE
ARG VERSION
ARG CODE_RELEASE
RUN \
  echo "**** install openvscode-server runtime dependencies ****" && \
  sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && apt-get update && \
  echo "**** install openvscode-server ****" && \
  if [ -z ${CODE_RELEASE+x} ]; then \
    CODE_RELEASE=$(curl -sX GET "https://api.github.com/repos/gitpod-io/openvscode-server/releases/latest" \
      | awk '/tag_name/{print $4;exit}' FS='[""]' \
      | sed 's|^openvscode-server-v||'); \
  fi && \
  mkdir -p /app/openvscode-server && \
  curl -o \
    /tmp/openvscode-server.tar.gz -L \
    "https://github.com/gitpod-io/openvscode-server/releases/download/openvscode-server-v${CODE_RELEASE}/openvscode-server-v${CODE_RELEASE}-linux-x64.tar.gz" && \
  tar xf \
    /tmp/openvscode-server.tar.gz -C \
    /app/openvscode-server/ --strip-components=1 && \
  echo "**** clean up ****" && \
  apt-get clean && rm -rf \
    /tmp/* \
    /var/lib/apt/lists/* \
    /var/tmp/*

# Contains information that vscode relies on
COPY root/ /

RUN curl -sL https://deb.nodesource.com/setup_14.x  | bash - && \
    apt-get install -y nodejs && \
    npm install -g configurable-http-proxy

#############
# Setup Environment + User
#############

# Setup Environment
ENV HOME=/home/user \
    CONDA_AUTO_UPDATE_CONDA=false

# Create a user named "user" with absolute power and make it default
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
 && chown -R user:user /app
RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers
USER user

# Setup working directory for the new user
WORKDIR $HOME

RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
    bash ~/miniconda.sh -b -p $HOME/miniconda && \
    rm ~/miniconda.sh 

# Add Miniconda to PATH
ENV PATH="/home/user/miniconda/bin:${PATH}"

# Set permissions (do this last)
RUN chown -R user:user $HOME && chmod -R 777 $HOME

# Switch to the user
USER user

# User the home directory and copy over the files on the Dockerfile level
COPY --chown=user:user . .

# Make the start_server (what starts the vscode server) executable
RUN chmod +x start_server.sh

#############
# Project Specific Configurations
#############

# Create a Conda environment with Python 3.7
RUN conda create --name paddle_env python=3.7 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/ # This is a one line command

# Set environment path
ENV CONDA_PATH="${HOME}/miniconda/envs/paddle_env/bin"
ENV PATH="${CONDA_PATH}:${PATH}"

# Activate the Conda environment
RUN echo "source activate paddle_env" > ~/.bashrc

# Install pip, NumPy, and PyTorch in the Conda environment
RUN conda install -n paddle_env pip numpy cudatoolkit=10.1

RUN $CONDA_PATH/python -m pip install paddlepaddle paddlepaddle-gpu -i https://pypi.tuna.tsinghua.edu.cn/simple

RUN $CONDA_PATH/pip install "paddleocr>=2.0.1"

#############

USER root

# User Debian packages
## Security warning : Potential user code executed as root (build time)
RUN --mount=target=/root/packages.txt,source=packages.txt \
    sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && sed -i 's http://deb.debian.org http://cdn-aws.deb.debian.org g' /etc/apt/sources.list && sed -i 's http://archive.ubuntu.com http://us-east-1.ec2.archive.ubuntu.com g' /etc/apt/sources.list && sed -i '/security/d' /etc/apt/sources.list && apt-get update && \
    xargs -r -a /root/packages.txt apt-get install -y --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

RUN --mount=target=/root/on_startup.sh,source=on_startup.sh,readwrite \
    bash /root/on_startup.sh

#############
# Final Startup
#############
EXPOSE 7860 3000

# Makes sure that the user has access to the files
USER user
RUN chown -R user:user $HOME && chmod -R 777 $HOME
# Add a new command to the Dockerfile to start the VSCode Server
CMD ["/bin/bash", "-c", "echo 'Starting VSCode Server...' && exec /app/openvscode-server/bin/openvscode-server --host 0.0.0.0 --port 7860 --without-connection-token \"${@}\" --"]

paddle-bot[bot] commented 4 months ago

Since you haven\'t replied for more than a year, we have closed this issue/pr. If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up. 由于您超过一年未回复，我们将关闭这个issue/pr。若问题未解决或有后续问题，请随时重新打开，我们会继续跟进。

PaddlePaddle / Paddle

4080训练，经常意外中断 #54203

bug描述 Describe the Bug

其他补充信息 Additional Supplementary Information