[BUG] Successful builds are not available for package run

[Johnz86]

Describe the bug Executing monai-deploy package ends with success. Produced docker image is not available locally and for next monai-deploy run. Invisible failed builds continuously consume disk space and they are not locally available.

Steps/Code to reproduce bug Consider following long run of command, which does not contain any error in output:

(monai.io) PS C:\GIT\monai.io> monai-deploy apps/simple_imaging_app -t simple_app:latest -l DEBUG
...
[2022-04-27 22:38:01,466] [DEBUG] (app_packager) -  => => sha256:becc2dd24a066a0e0851bcff6eed40963e58590cd 1.01GB / 1.83GB  748.0s

[2022-04-27 22:38:01,466] [DEBUG] (app_packager) -  => => sha256:83ffbc495ca2a28e17acf9c82fe289efa2a03 626.68MB / 626.68MB  640.2s

[2022-04-27 22:38:01,467] [DEBUG] (app_packager) -  => => sha256:d0521bbd1a062f8965f8b5436461c688f1962617b 4.88MB / 4.88MB  642.8s

[2022-04-27 22:38:01,467] [DEBUG] (app_packager) -  => => sha256:6f143002aec07b3aacc87c2d1beabce82bf72f018 1.14kB / 1.14kB  643.1s

[2022-04-27 22:38:01,467] [DEBUG] (app_packager) -  => => sha256:5026903fc514d85b6c21e4e1fcc7575d332a398 29.47MB / 29.47MB  659.3s

[2022-04-27 22:38:01,468] [DEBUG] (app_packager) -  => => sha256:d9663d4d62ddde031aa570b682fb85f46da892935f814 0B / 3.50kB  748.0s

[2022-04-27 22:38:01,468] [DEBUG] (app_packager) -  => [internal] load build context                                          0.9s

[2022-04-27 22:38:01,468] [DEBUG] (app_packager) -  => => transferring context: 535.54kB                                      0.9s

[2022-04-27 22:38:01,469] [DEBUG] (app_packager) - ------

[2022-04-27 22:38:01,469] [DEBUG] (app_packager) -  > [ 1/15] FROM nvcr.io/nvidia/pytorch:21.07-py3@sha256:ac932f866068b75009b156030ff9c348f1d78343227c01cbdadf59baeab624eb:

[2022-04-27 22:38:01,469] [DEBUG] (app_packager) - ------

[2022-04-27 22:38:01,474] [DEBUG] (app_packager) -
[2022-04-27 22:38:01,475] [DEBUG] (app_packager) -
[2022-04-27 22:38:01,476] [DEBUG] (app_packager) -
[2022-04-27 22:38:01,476] [DEBUG] (app_packager) -
Done
(monai.io) PS C:\GIT\monai.io> echo $LASTEXITCODE
0

Following command fails next.

(monai.io) PS C:\GIT\monai.io> monai-deploy run simple_app:latest input output
Checking dependencies...
--> Verifying if "docker" is installed...

--> Verifying if "simple_app:latest" is available...

Checking for MAP "simple_app:latest" locally
"simple_app:latest" not found locally.

Trying to pull from registry...
Error response from daemon: pull access denied for simple_app, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: Unable to fetch required image.
Execution Aborted

Expected behavior Failed builds should be visible. Failed builds should not occupy disk space. If the build is succesfull, then following command

monai-deploy run simple_app:latest input output

Should not fail on not finding image locally.

Environment details (please complete the following information)

(monai.io) PS C:\GIT\codecon\monai.io> python -c 'import monai; monai.config.print_debug_info()'
================================
Printing MONAI config...
================================
MONAI version: 0.8.1
Numpy version: 1.22.3
Pytorch version: 1.9.0+cu111
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: 71ff399a3ea07aef667b23653620a290364095b1

Optional dependencies:
Pytorch Ignite version: 0.4.8
Nibabel version: 3.2.2
scikit-image version: 0.19.2
Pillow version: 9.1.0
Tensorboard version: 2.8.0
gdown version: 4.4.0
TorchVision version: 0.10.0+cu111
tqdm version: 4.64.0
lmdb version: 1.3.0
pandas version: NOT INSTALLED or UNKNOWN VERSION.
einops version: 0.3.2
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================
Printing system config...
================================
System: Windows
Win32 version: ('10', '10.0.18363', 'SP0', 'Multiprocessor Free')
Win32 edition: Enterprise
Platform: Windows-10-10.0.18363-SP0
Processor: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
Machine: AMD64
Python version: 3.9.5
Process name: python.exe
Command: ['C:\\Python39\\python.exe', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='C:\\WINDOWS\\System32\\en-US\\KernelBase.dll.mui', fd=-1), popenfile(path='C:\\WINDOWS\\System32\\en-US\\kernel32.dll.mui', fd=-1)]
Num physical CPUs: 6
Num logical CPUs: 12
Num usable CPUs: 12
CPU usage (%): [11.5, 13.5, 8.9, 1.3, 9.0, 0.4, 4.5, 1.8, 5.8, 2.2, 3.6, 73.1]
CPU freq. (MHz): 2592
Load avg. in last 1, 5, 15 mins (%): [0.0, 0.0, 0.0]
Disk usage (%): 93.9
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 31.8
Available memory (GB): 15.3
Used memory (GB): 16.6

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 11.1
cuDNN enabled: True
cuDNN version: 8005
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37']
GPU 0 Name: Quadro P2000
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 6
GPU 0 Total memory (GB): 4.0
GPU 0 CUDA capability (maj.min): 6.1

Additional context Latest docker is installed. This is locally sucesfull:

(monai.io) PS C:\GIT\codecon\monai.io> python apps/simple_imaging_app/app.py -i apps/simple_imaging_app/brain_mr_input.jpg -o output
Going to initiate execution of operator SobelOperator
Executing operator SobelOperator (Process ID: 31232, Operator ID: 09cc67fb-ec29-4346-82b5-b5fd5ddf565d)
Done performing execution of operator SobelOperator

Going to initiate execution of operator MedianOperator
Executing operator MedianOperator (Process ID: 31232, Operator ID: 4165b9f5-5a2d-4a07-8975-17d148a4c187)
Executing base operator...
Done performing execution of operator MedianOperator

Going to initiate execution of operator GaussianOperator
Executing operator GaussianOperator (Process ID: 31232, Operator ID: 37e43a43-b519-4696-9750-ff40cf167f62)
Images with dimensions (M, N, 3) are interpreted as 2D+RGB by default. Use `multichannel=False` to interpret as 3D image with last dimension of length 3.
[2022-04-27 22:49:45,312] [WARNING] (imageio) - Lossy conversion from float64 to uint8. Range [0, 1]. Convert image to uint8 prior to saving to suppress this warning.
Done performing execution of operator GaussianOperator

[gigony]

Hi @Johnz86 , Thank you for using MONAI Deploy App sdk and raising an issue!

Given the information you provided, it was hard for me to reproduce it. If you don't mind, could you please attach log file in here? I am particularly interested in which commands were executed when creating docker image

Step 1/29 : FROM nvcr.io/nvidia/pytorch:21.07-py3
...
Step 5/29 : LABEL sdk_version="0.3.0+5.gc64296d"

Seeing the following message (having only 15 steps and using nvcr.io/nvidia/pytorch:21.07 instead of nvcr.io/nvidia/pytorch:21.07-py3) doesn't look correct.

[ 1/15] FROM nvcr.io/nvidia/pytorch:21.07

And I guess executing the following command is supposed to give error because there is no package command.

monai-deploy apps/simple_imaging_app -t simple_app:latest -l DEBUG

# monai-deploy package apps/simple_imaging_app -t simple_app:latest -l DEBUG

Could you please try upgrade the package version

pip install --upgrade monai-deploy-app-sdk

and following the instruction below works?

https://docs.monai.io/projects/monai-deploy-app-sdk/en/0.3.0/getting_started/tutorials/01_simple_app.html#executing-from-shell

FYI, we didn't test on the Windows Operating system. It seems that you are running on Windows (not Windows Subsystem for Linux[WSL]). MONAI Deploy App SDK is officially supporting Ubuntu OS and Windows users can install CUDA on WSL and work inside WSL to use MONAI Deploy App SDK.

[MMelQin]

Links to documentation provided.