Automate pipeline application building lambda and api gateway deployment

[mcduffie]

Describe use in Automated Build Documentation

[GodwinShen]

@mcduffie is this one of the work tickets for the main task https://github.com/unity-sds/unity-project-management/issues/89? If so I will add this ticket to the main task Work Tickets list.

[GodwinShen]

@mcduffie how is this task coming along?

[mcduffie]

@edwinsarkissian Making progress. Remaining work is to add authentication check and triggers to Lambda deployment. Also need to add api-gateway connection to Lambda.

[GodwinShen]

@mcduffie ping for status.

[mcduffie]

@edwinsarkissian Added Lambda and api-gateway deployment. Remaining work is enabling authentication and testing with it on. But when testing currently is running into this error in requests lib causing problems with testing: https://github.com/psf/requests/issues/6707

[GodwinShen]

@mcduffie ping for status.

[edwinsarkissian]

The issue related to Python "requests" and "docker" packages have been resolved. I go beyond the first failing point; however, there is a new problem with no obvious reason, which has prevented moving forward with full automation. The full automation is 95% done, but I cannot test it due to the second problem. Now, apparently the docker image does not get registered in the local (In the EC2 Instance) docker server. This had been working for me for months, and it stopped working without any update on my side.

[mike-gangl]

Because this is a "shared" service, we can proceed with the near fully automated solution into test/prod.

[mike-gangl]

The issue related to Python "requests" and "docker" packages have been resolved. I go beyond the first failing point; however, there is a new problem with no obvious reason, which has prevented moving forward with full automation. The full automation is 95% done, but I cannot test it due to the second problem. Now, apparently the docker image does not get registered in the local (In the EC2 Instance) docker server. This had been working for me for months, and it stopped working without any update on my side.

This is the docker container that is built by app-pack-gen? And it's not being registered within the 'local' docker upon build? and so that prevents pushing? i'm just trying to understand the issue. Is tehre an error? is it silent? My first assumption would be disk space if this has been running for a long while...

[mike-gangl]

i ssh'ed into the machine, and tried running the build_ogc_app as the gitlab-runner user. Ended up not getting very far in the build:

step 27/50 : RUN TIMEFORMAT='time: %3R' bash -c 'time /tmp/install-base-env.bash' && rm -rf /tmp/install-base-env.bash /tmp/env
 ---> Running in 5ce55d452dfb
+ dirname /tmp/install-base-env.bash
+ cd /tmp
+ export MAMBA_VERSION=1.5.7
+ MAMBA_VERSION=1.5.7
+ export CONDA_VERSION=24.3.0
+ CONDA_VERSION=24.3.0
+ URL=https://anaconda.org/conda-forge/micromamba/1.5.7/download/linux-64/micromamba-1.5.7-0.tar.bz2
+ unset HOME
+ mkdir -p /srv/conda
+ export MICROMAMBA_EXE=/usr/local/bin/micromamba
+ MICROMAMBA_EXE=/usr/local/bin/micromamba
+ tar -xvj bin/micromamba
+ wget -qO- https://anaconda.org/conda-forge/micromamba/1.5.7/download/linux-64/micromamba-1.5.7-0.tar.bz2
bin/micromamba
time: 2.560
+ mv bin/micromamba /usr/local/bin/micromamba
+ chmod 0755 /usr/local/bin/micromamba
++ /usr/local/bin/micromamba shell hook -p /srv/conda -s posix
+ eval '
# Copyright (C) 2012 Anaconda, Inc
# SPDX-License-Identifier: BSD-3-Clause

__mamba_exe() (
    "/usr/local/bin/micromamba" "${@}"
)

__mamba_hashr() {
    if [ -n "${ZSH_VERSION:+x}" ]; then
        \rehash
    elif [ -n "${POSH_VERSION:+x}" ]; then
        :  # pass
    else
        \hash -r
    fi
}

__mamba_xctivate() {
    \local ask_conda
    ask_conda="$(PS1="${PS1:-}" __mamba_exe shell "${@}" --shell bash)" || \return
    \eval "${ask_conda}"
    __mamba_hashr
}

micromamba() {
    \local cmd="${1-__missing__}"
    case "${cmd}" in
        activate|reactivate|deactivate)
            __mamba_xctivate "${@}"
            ;;
        install|update|upgrade|remove|uninstall)
            __mamba_exe "${@}" || \return
            __mamba_xctivate reactivate
            ;;
        self-update)
            __mamba_exe "${@}" || \return

            # remove leftover backup file on Windows
            if [ -f "/usr/local/bin/micromamba.bkup" ]; then
                rm -f "/usr/local/bin/micromamba.bkup"
            fi
            ;;
        *)
            __mamba_exe "${@}"
            ;;
    esac
}

if [ -z "${CONDA_SHLVL+x}" ]; then
    \export CONDA_SHLVL=0
    # In dev-mode MAMBA_EXE is python.exe and on Windows
    # it is in a different relative location to condabin.
    if [ -n "${_CE_CONDA+x}" ] && [ -n "${WINDIR+x}" ]; then
        PATH="${MAMBA_ROOT_PREFIX}/condabin:${PATH}"
    else
        PATH="${MAMBA_ROOT_PREFIX}/condabin:${PATH}"
    fi
    \export PATH

    # We'\''re not allowing PS1 to be unbound. It must at least be set.
    # However, we'\''re not exporting it, which can cause problems when starting a second shell
    # via a first shell (i.e. starting zsh from bash).
    if [ -z "${PS1+x}" ]; then
        PS1=
    fi
fi'
++ '[' -z '' ']'
++ export CONDA_SHLVL=0
++ CONDA_SHLVL=0
++ '[' -n '' ']'
++ PATH=/srv/conda/condabin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ '[' -z '' ']'
++ PS1=
+ micromamba activate
+ local cmd=activate
+ case "${cmd}" in
+ __mamba_xctivate activate
+ local ask_conda
++ PS1=
++ __mamba_exe shell activate --shell bash
++ /usr/local/bin/micromamba shell activate --shell bash
+ ask_conda='PS1='\''(base) '\''
export PATH='\''/srv/conda/bin:/srv/conda/condabin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'\''
export CONDA_PREFIX='\''/srv/conda'\''
export CONDA_SHLVL='\''1'\''
export CONDA_DEFAULT_ENV='\''base'\''
export CONDA_PROMPT_MODIFIER='\''(base) '\'''
+ eval 'PS1='\''(base) '\''
export PATH='\''/srv/conda/bin:/srv/conda/condabin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'\''
export CONDA_PREFIX='\''/srv/conda'\''
export CONDA_SHLVL='\''1'\''
export CONDA_DEFAULT_ENV='\''base'\''
export CONDA_PROMPT_MODIFIER='\''(base) '\'''
++ PS1='(base) '
++ export PATH=/srv/conda/bin:/srv/conda/condabin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/srv/conda/bin:/srv/conda/condabin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export CONDA_PREFIX=/srv/conda
++ CONDA_PREFIX=/srv/conda
++ export CONDA_SHLVL=1
++ CONDA_SHLVL=1
++ export CONDA_DEFAULT_ENV=base
++ CONDA_DEFAULT_ENV=base
++ export 'CONDA_PROMPT_MODIFIER=(base) '
++ CONDA_PROMPT_MODIFIER='(base) '
+ __mamba_hashr
+ '[' -n '' ']'
+ '[' -n '' ']'
+ hash -r
+ export PATH=/tmp/bin:/srv/conda/bin:/srv/conda/condabin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/tmp/bin:/srv/conda/bin:/srv/conda/condabin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ cat
+ micromamba install conda=24.3.0 mamba=1.5.7 -y
+ local cmd=install
+ case "${cmd}" in
+ __mamba_exe install conda=24.3.0 mamba=1.5.7 -y
+ /usr/local/bin/micromamba install conda=24.3.0 mamba=1.5.7 -y

and it ends there. no real useful warning or error.

[mike-gangl]

I think we saw this end up being a memory error. So we decided to increase the instance size of these for now. We will really, really need to figure out what we do with concurrent builds and many users requesting builds a single instance is not going to work, and i'm not sure if gitlab runners do all the queueing work or not (i think they do...)

[mcduffie]

Currently encountering an error where push_docker fails despite docker login succeeding.

[mcduffie]

Remaining part is to connect Lambda authorizer to API-Gateway path

[mcduffie]

Fully automated, needs to finish pushing in changes after testing changes.