issue passing iso file using source argument using kaniko approach

[HaPlossP]

Just like in #95 we try to create a matlab docker image with some toolboxes on a corporate GitLab instance without privileged rights. Thus we use kaniko. This time, we try to reference the installation media as iso file, as downloaded from MathWorks.com for R2022b. We try to reference the media as indicated by the manual using the --source flag for mpm.

Unfortunately, installation fails with Error: Unable to find the source folder. Specify a valid source folder.

We tried passing the path both with --source=/tmp/matlab.iso as well as --source /tmp/matlab.iso, as documentation is fully not clear about how to provide arguments. (release is passed with = in the example of section "Install Products" but not in the section "Product Installation Options") We ensure that the file matlab.iso is present at /tmp/matlab.iso by issuing command ls -la within the pipeline According to the docu,mentation for R2021b and later releases a ISO image is fine.

.gitlab-ci.yml

stages:
  - build

build:
  stage: build
  when: manual
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - cp /R2022b_Update_7_Linux.iso "${CI_PROJECT_DIR}"
    - cat /etc/gitlab-runner/certs/<redacted>.crt >> /kaniko/ssl/certs/ca-certificates.crt
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$(echo -n $CI_DEPENDENCY_PROXY_SERVER | awk -F[:] '{print $1}')\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor 
      --context "${CI_PROJECT_DIR}"
      --build-arg http_proxy="${http_proxy}"
      --build-arg https_proxy="${https_proxy}"
      --build-arg HTTP_PROXY="${http_proxy}"
      --build-arg HTTPS_PROXY="${https_proxy}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"

Dockerfile

# Copyright 2019 - 2023 The MathWorks, Inc.
# This Dockerfile allows you to build a Docker® image with MATLAB® installed using the MATLAB Package 
# Manager. Use the optional build arguments to customize the version of MATLAB, list of products to 
# install, and the location at which to install MATLAB.

# Here is an example docker build command with the optional build arguments.
# docker build --build-arg MATLAB_RELEASE=r2023b 
#              --build-arg MATLAB_PRODUCT_LIST="MATLAB Deep_Learning_Toolbox Symbolic_Math_Toolbox"
#              --build-arg MATLAB_INSTALL_LOCATION="/opt/matlab/R2023b"
#              --build-arg LICENSE_SERVER=12345@hostname.com 
#              -t my_matlab_image_name .

# To specify which MATLAB release to install in the container, edit the value of the MATLAB_RELEASE argument.
# Use lowercase to specify the release, for example: ARG MATLAB_RELEASE=r2021b
ARG MATLAB_RELEASE=r2022b

# Specify the list of products to install into MATLAB.
ARG MATLAB_PRODUCT_LIST="MATLAB"

# We will use an input file with a list of products instead
ARG MPM_INPUT_FILE="mpm-input-files/R2022b/mpm_input_r2022b.txt"

# Specify MATLAB Install Location.
# Has to match the location in the input file
ARG MATLAB_INSTALL_LOCATION="/opt/matlab/${MATLAB_RELEASE}"

ARG MATLAB_ISO="/R2022b_Update_7_Linux.iso"

# Specify license server information using the format: port@hostname 
# ARG LICENSE_SERVER
ARG LICENSE_FILE="license/20231207_license.lic"

# When you start the build stage, this Dockerfile by default uses the Ubuntu-based matlab-deps image.
# To check the available matlab-deps images, see: https://hub.docker.com/r/mathworks/matlab-deps
FROM mathworks/matlab-deps:${MATLAB_RELEASE}

# Declare build arguments to use at the current build stage.
ARG MATLAB_RELEASE
ARG MATLAB_PRODUCT_LIST
ARG MATLAB_INSTALL_LOCATION
#ARG LICENSE_SERVER
ARG MPM_INPUT_FILE
ARG MATLAB_ISO

# Install mpm dependencies.
RUN export DEBIAN_FRONTEND=noninteractive \
    && apt-get update \
    && apt-get install --no-install-recommends --yes \
    wget \
    unzip \
    ca-certificates \
    && apt-get clean \
    && apt-get autoremove \
    && rm -rf /var/lib/apt/lists/*

# Add "matlab" user and grant sudo permission.
RUN adduser --shell /bin/bash --disabled-password --gecos "" matlab \
    && echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab \
    && chmod 0440 /etc/sudoers.d/matlab

# Set user and work directory.
USER matlab
WORKDIR /home/matlab

COPY /R2022b_Update_7_Linux.iso /tmp/matlab.iso

RUN ls -la /tmp/

RUN env

# Run mpm to install MATLAB in the target location and delete the mpm installation afterwards.
# If mpm fails to install successfully, then print the logfile in the terminal, otherwise clean up.
# Pass in $HOME variable to install support packages into the user's HOME folder.

#COPY ${MPM_INPUT_FILE} .
#RUN wget -q https://www.mathworks.com/mpm/glnxa64/mpm \ 
#    && chmod +x mpm \
#    && sudo HOME=${HOME} ./mpm install \
#    --inputfile=${MPM_INPUT_FILE} \
#    || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) \
#    && sudo rm -f mpm /tmp/mathworks_root.log \
#    && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab

RUN wget -q https://www.mathworks.com/mpm/glnxa64/mpm \ 
    && chmod +x mpm \
    && sudo HOME=${HOME} ./mpm install \
    --source=/tmp/matlab.iso \
    --destination=${MATLAB_INSTALL_LOCATION} \
    --products ${MATLAB_PRODUCT_LIST} \
    || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) \
    && sudo rm -f mpm /tmp/mathworks_root.log \
    && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab

# Note: Uncomment one of the following two ways to configure the license server.

# Option 1. Specify the host and port of the machine that serves the network licenses
# if you want to store the license information in an environment variable. This
# is the preferred option. You can either use a build variable, like this: 
# --build-arg LICENSE_SERVER=27000@MyServerName or you can specify the license server 
# directly using: ENV MLM_LICENSE_FILE=27000@flexlm-server-name
#ENV MLM_LICENSE_FILE=$LICENSE_SERVER

# Option 2. Alternatively, you can put a license file into the container.
# Enter the details of the license server in this file and uncomment the following line.
COPY ${LICENSE_FILE} ${MATLAB_INSTALL_LOCATION}/licenses/

# The following environment variables allow MathWorks to understand how this MathWorks 
# product (MATLAB Dockerfile) is being used. This information helps us make MATLAB even better. 
# Your content, and information about the content within your files, is not shared with MathWorks. 
# To opt out of this service, delete the environment variables defined in the following line. 
# To learn more, see the Help Make MATLAB Even Better section in the accompanying README: 
# https://github.com/mathworks-ref-arch/matlab-dockerfile#help-make-matlab-even-better
# ENV MW_DDUX_FORCE_ENABLE=true MW_CONTEXT_TAGS=MATLAB:DOCKERFILE:V1

ENTRYPOINT ["matlab"]
CMD [""]

full log of gitlab runner

Running with gitlab-runner 16.6.1 (f5da3c5a)
  on <redacted>
Preparing the "docker" executor
00:02
Using Docker executor with image gcr.io/kaniko-project/executor:debug ...
Pulling docker image gcr.io/kaniko-project/executor:debug ...
Using docker image sha256:b2f0e64d2a66329ae9b6eebd73d688625d6943328205d8471755cea6edbc9536 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:0818d281f155b9bb1b1f0704e55939b9c440b0155cbbf65b567ca19e19e20223 ...
Preparing environment
00:00
Updating CA certificates...
Running on runner-jrgxsfjp3-project-263-concurrent-0 via c7c7f1401705...
Getting source from Git repository
00:01
$ git config --global http.proxy $HTTP_PROXY; git config --global https.proxy $HTTPS_PROXY;
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/<redacted>/docker-matlab/.git/
Checking out 3e1a6c08 as detached HEAD (ref is <redacted>)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
08:28
Using docker image sha256:b2f0e64d2a66329ae9b6eebd73d688625d6943328205d8471755cea6edbc9536 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:0818d281f155b9bb1b1f0704e55939b9c440b0155cbbf65b567ca19e19e20223 ...
$ cp /R2022b_Update_7_Linux.iso "${CI_PROJECT_DIR}"
$ cat /etc/gitlab-runner/certs/<redacted>.crt >> /kaniko/ssl/certs/ca-certificates.crt
$ mkdir -p /kaniko/.docker
$ echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$(echo -n $CI_DEPENDENCY_PROXY_SERVER | awk -F[:] '{print $1}')\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
$ /kaniko/executor --context "${CI_PROJECT_DIR}" --build-arg http_proxy="${http_proxy}" --build-arg https_proxy="${https_proxy}" --build-arg HTTP_PROXY="${http_proxy}" --build-arg HTTPS_PROXY="${https_proxy}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
INFO[0000] Retrieving image manifest mathworks/matlab-deps:r2022b 
INFO[0000] Retrieving image mathworks/matlab-deps:r2022b from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest mathworks/matlab-deps:r2022b 
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'mathworks/matlab-deps:r2022b' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd RUN export DEBIAN_FRONTEND=noninteractive     && apt-get update     && apt-get install --no-install-recommends --yes     wget     unzip     ca-certificates     && apt-get clean     && apt-get autoremove     && rm -rf /var/lib/apt/lists/* requires it. 
INFO[0014] ARG MATLAB_RELEASE                           
INFO[0014] ARG MATLAB_PRODUCT_LIST                      
INFO[0014] ARG MATLAB_INSTALL_LOCATION                  
INFO[0014] ARG MPM_INPUT_FILE                           
INFO[0014] ARG MATLAB_ISO                               
INFO[0014] RUN export DEBIAN_FRONTEND=noninteractive     && apt-get update     && apt-get install --no-install-recommends --yes     wget     unzip     ca-certificates     && apt-get clean     && apt-get autoremove     && rm -rf /var/lib/apt/lists/* 
INFO[0014] Initializing snapshotter ...                 
INFO[0014] Taking snapshot of full filesystem...        
INFO[0019] Cmd: /bin/sh                                 
INFO[0019] Args: [-c export DEBIAN_FRONTEND=noninteractive     && apt-get update     && apt-get install --no-install-recommends --yes     wget     unzip     ca-certificates     && apt-get clean     && apt-get autoremove     && rm -rf /var/lib/apt/lists/*] 
INFO[0019] Running: [/bin/sh -c export DEBIAN_FRONTEND=noninteractive     && apt-get update     && apt-get install --no-install-recommends --yes     wget     unzip     ca-certificates     && apt-get clean     && apt-get autoremove     && rm -rf /var/lib/apt/lists/*] 
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3367 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3454 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [29.7 kB]
Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1192 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3952 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [32.4 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3549 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1487 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [55.2 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [28.6 kB]
Fetched 30.6 MB in 4s (7312 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (20230311ubuntu0.20.04.1).
unzip is already the newest version (6.0-25ubuntu1.2).
wget is already the newest version (1.20.3-1ubuntu2).
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
INFO[0028] Taking snapshot of full filesystem...        
INFO[0029] RUN adduser --shell /bin/bash --disabled-password --gecos "" matlab     && echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab     && chmod 0440 /etc/sudoers.d/matlab 
INFO[0029] Cmd: /bin/sh                                 
INFO[0029] Args: [-c adduser --shell /bin/bash --disabled-password --gecos "" matlab     && echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab     && chmod 0440 /etc/sudoers.d/matlab] 
INFO[0029] Running: [/bin/sh -c adduser --shell /bin/bash --disabled-password --gecos "" matlab     && echo "matlab ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/matlab     && chmod 0440 /etc/sudoers.d/matlab] 
Adding user `matlab' ...
Adding new group `matlab' (1000) ...
Adding new user `matlab' (1000) with group `matlab' ...
Creating home directory `/home/matlab' ...
Copying files from `/etc/skel' ...
INFO[0029] Taking snapshot of full filesystem...        
INFO[0030] USER matlab                                  
INFO[0030] Cmd: USER                                    
INFO[0030] WORKDIR /home/matlab                         
INFO[0030] Cmd: workdir                                 
INFO[0030] Changed working directory to /home/matlab    
INFO[0030] No files changed in this command, skipping snapshotting. 
INFO[0030] COPY /R2022b_Update_7_Linux.iso /tmp/matlab.iso 
INFO[0074] Taking snapshot of files...                  
INFO[0330] RUN ls -la /tmp/                             
INFO[0330] Cmd: /bin/sh                                 
INFO[0330] Args: [-c ls -la /tmp/]                      
INFO[0330] Util.Lookup returned: &{Uid:1000 Gid:1000 Username:matlab Name: HomeDir:/home/matlab} 
INFO[0330] Performing slow lookup of group ids for matlab 
INFO[0330] Running: [/bin/sh -c ls -la /tmp/]           
total 19399164
drwxrwxrwt 2 root root          53 Mar 18 13:19 .
drwxr-xr-x 1 root root        4096 Mar 18 13:19 ..
-rw-r--r-- 1 root root         544 Mar 18 00:11 base-dependencies.txt
-rw-r--r-- 1 root root 19864733696 Mar 18 13:20 matlab.iso
INFO[0330] Taking snapshot of full filesystem...        
INFO[0389] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0389] RUN env                                      
INFO[0389] Cmd: /bin/sh                                 
INFO[0389] Args: [-c env]                               
INFO[0389] Util.Lookup returned: &{Uid:1000 Gid:1000 Username:matlab Name: HomeDir:/home/matlab} 
INFO[0389] Performing slow lookup of group ids for matlab 
INFO[0389] Running: [/bin/sh -c env]                    
HTTPS_PROXY=http://<redacted>:8080
MPM_INPUT_FILE=mpm-input-files/R2022b/mpm_input_r2022b.txt
MATLAB_ISO=/R2022b_Update_7_Linux.iso
HOME=/home/matlab
MATLAB_RELEASE=r2022b
https_proxy=http://<redacted>:8080
http_proxy=http://<redacted>:8080
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MATLAB_PRODUCT_LIST=MATLAB
DEBIAN_FRONTEND=noninteractive
MATLAB_INSTALL_LOCATION=/opt/matlab/r2022b
PWD=/home/matlab
HTTP_PROXY=http://<redacted>:8080
TZ=Etc/UTC
INFO[0389] Taking snapshot of full filesystem...        
INFO[0435] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0435] RUN wget -q https://www.mathworks.com/mpm/glnxa64/mpm     && chmod +x mpm     && sudo HOME=${HOME} ./mpm install     --source=/tmp/matlab.iso     --destination=${MATLAB_INSTALL_LOCATION}     --products ${MATLAB_PRODUCT_LIST}     || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false)     && sudo rm -f mpm /tmp/mathworks_root.log     && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab 
INFO[0435] Cmd: /bin/sh                                 
INFO[0435] Args: [-c wget -q https://www.mathworks.com/mpm/glnxa64/mpm     && chmod +x mpm     && sudo HOME=${HOME} ./mpm install     --source=/tmp/matlab.iso     --destination=${MATLAB_INSTALL_LOCATION}     --products ${MATLAB_PRODUCT_LIST}     || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false)     && sudo rm -f mpm /tmp/mathworks_root.log     && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab] 
INFO[0435] Util.Lookup returned: &{Uid:1000 Gid:1000 Username:matlab Name: HomeDir:/home/matlab} 
INFO[0435] Performing slow lookup of group ids for matlab 
INFO[0435] Running: [/bin/sh -c wget -q https://www.mathworks.com/mpm/glnxa64/mpm     && chmod +x mpm     && sudo HOME=${HOME} ./mpm install     --source=/tmp/matlab.iso     --destination=${MATLAB_INSTALL_LOCATION}     --products ${MATLAB_PRODUCT_LIST}     || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false)     && sudo rm -f mpm /tmp/mathworks_root.log     && sudo ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab] 
Error: Unable to find the source folder. Specify a valid source folder.
MPM Installation Failure. See below for more information:
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

[mw-skardile]

Hi HaPlossP,

Sorry for the late reply, and thanks for your question.

For the --source flag, mpm expects the *.iso file to be already mounted. Please use the full path to where /tmp/matlab.iso is mounted as your --source. Let me know if that solves the problem.

Also, we appreciate your feedback on the documentation. We'll work on making the instructions clearer.

[HaPlossP]

Dear @mw-skardile , thank you for providing a suggestion. We were desperate and fell back trying to issue the mpm installation command directly from the linux machine that serves as the GitLab runner. The iso R2022b_Update_7_Linux.iso that we used must have been corrupted to some degree?! While we were able to both mount / extract it, mpm would fail with above error. We then downloaded the most recent iso of R2022b R2022b_Update_8_Linux.iso directly on the machine and to our surprise, suddenly things started working. First, we instructed mpm like this: ./mpm install --destination=matlab_dest --source=/var/lib/docker/mnt-data/R2022b_Update_8_Linux.iso --products MATLAB which got quitted by Error: Unable to find the source folder. Specify a valid source folder

Then, we issued the following commands to mount the iso to a directory from pwd = /var/lib/docker/mnt-data/matlab-test:

# create mount directory
mkdir matlab-mnt
# mount the iso
mount -o loop /var/lib/docker/mnt-data/R2022b_Update_8_Linux.iso /var/lib/docker/mnt-data/matlab-test/matlab-mnt
# verify mount succeeded by checkinig for /dev/loop in output of
df -h
# ... and also listing the content of the mountpoint
ls /var/lib/docker/mnt-data/matlab-test/matlab-mnt
# finally issuing the *successful* mpm installation command with
./mpm install --destination=matlab_dest --source=/var/lib/docker/mnt-data/matlab-test/matlab-mnt --products MATLAB

Consequently, we can sum up that installation was not successful with the initial iso used. Installation with pointing directly to the new iso was also not successful. Successful installation happened, when pointing to mounted content of the iso.

I feel this is not what the documentation of the --source flag suggests, so I'm not yet closing this issue.

[mw-skardile]

Hi @HaPlossP , I am glad you were able to install products using the --source flag after mounting the ISO. We'll work on updating the documentation based on your feedback.