search

1718-io / propositions-relatives-au-ric

Quelques propositions concernant les organisations qui considèrent l'instauration du RIC comme leur premier objectif.
1 stars 0 forks source link

Automated deployment : Heroku #2

Closed Jean-Baptiste-Lasselle closed 3 years ago

Jean-Baptiste-Lasselle commented 3 years ago

Heroku Deployment

Okay,:

Ok, now one thing to finish issues 1 and 2 of https://github.com/pegasus-io/git-hush-hush to completely automate the process :

Jean-Baptiste-Lasselle commented 3 years ago

dernières vérification à faire pour le déploiement Heroku, en comparant au codesource de la commande https://circleci.com/developer/orbs/orb/circleci/heroku#commands-deploy-via-git

Jean-Baptiste-Lasselle commented 3 years ago

Go app automatically detected by Heroku

Chié, au lieu de faire mon heroku.yml, la palteforme essaie automatiquement de directment fairele build golang grrrrr :

Jean-Baptiste-Lasselle commented 3 years ago

Mais c'est dingue, ça, chez Heroku

Purée mais chez moi le Docker build fonctionne parfaitement, et chez eux, avec EXACTEMENT le même Dockerfile, ils le font foirer (apparrement parce qu'il n'y a pas le .git/ dans l'image ...? ) : 

remote: drwx------    6 root     root          62 Dec 26 05:40 static
remote: drwx------    4 root     root          38 Dec 26 05:40 themes
remote: Removing intermediate container 874f1ef45d1e
remote:  ---> 31cf3b98c2ad
remote: Step 21/48 : RUN export PATH=$PATH:/usr/local/go/bin && cd /pokus.io/hugo/src/ && hugo -b "${HUGO_BASE_URL}"
remote:  ---> Running in dcb9c5ae1ce9
remote: Start building sites … 
remote: ERROR 2020/12/26 05:46:51 Failed to read Git log: fatal: not a git repository (or any of the parent directories): .git
remote: Total in 153 ms
remote: Error: Error building site: logged 1 error(s)
remote: The command '/bin/sh -c export PATH=$PATH:/usr/local/go/bin && cd /pokus.io/hugo/src/ && hugo -b "${HUGO_BASE_URL}"' returned a non-zero code: 255
remote: 
remote:  !
remote:  ! ## Warning - The same version of this code has already been built: 3e4a496a7cd1ee603108c2acd453f07f24b94be7
remote:  !
remote:  ! We have detected that you have triggered a build from source code with version 3e4a496a7cd1ee603108c2acd453f07f24b94be7
remote:  ! at least twice. One common cause of this behavior is attempting to deploy code from a different branch.
remote:  !
remote:  ! If you are developing on a branch and deploying via git you must run:
remote:  !
remote:  !     git push heroku <branchname>:main
remote:  !
remote:  ! This article goes into details on the behavior:
remote:  !   https://devcenter.heroku.com/articles/duplicate-build-version
remote: 
remote: Verifying deploy...
remote: 
remote: !   Push rejected to ric-carl.
remote: 
To https://git.heroku.com/ric-carl.git
 ! [remote rejected] master -> main (pre-receive hook declined)
error: failed to push some refs to 'https://heroku:db292ab0-182b-4f56-90b0-6a4e898c0b99@git.heroku.com/ric-carl.git'
jibl@poste-devops-jbl-16gbram:~/propositions-relatives-au-ric$

Oh ils me font chier chez Heroku

Jean-Baptiste-Lasselle commented 3 years ago

Identified the cause of the issue


Okay, so now that I collected all those informations, here are the questions I want an answer to : 
* [ ] **_QUESTION 1_** Is the Docker image build still successful, on my machine, with that one small modification, the added `COPY .git /pokus.io/hugo/src/` command ? **_ANSWER IS : YES BUILD SUCCESFUL_**
* [ ] **_QUESTION 2_** Is the hugo build still successful, on my machine, on the excat same version which is deployed to Heroku, verison which includes that one small modification, the added `COPY .git /pokus.io/hugo/src/` command ? (should obviously be yes, since the `heroku.Dockerfile` is not used in the `hugo` build) **_ANSWER IS : YES BUILD SUCCESFUL_**
* [ ] **_QUESTION 3_** Does the `hugo` build fails, if I remove the `.git/` folder ? 
* [ ] **_QUESTION 4_** Does the Docker image build fails, if I both : 
  * remove the `.git/` folder
  * and remove the the `COPY .git /pokus.io/hugo/src/` Dockerfile command, from the `heroku.Dockerfile` command ? 

What test will I run to answer all of those questions ? 

* [ ] **_QUESTION 1_** : 

```bash
export DESIRED_VERSION=master
docker system prune -f --all && docker system prune -f --volumes
rm -fr ~/propositions-relatives-au-ric-test
git clone git@github.com:1718-io/propositions-relatives-au-ric.git ~/propositions-relatives-au-ric-test
cd ~/propositions-relatives-au-ric-test
git checkout "${DESIRED_VERSION}"

source .heroku.env
export QUAY_OCI_IMAGE_TAG=0.0.1-heroku
docker-compose build hugo_heroku

export DESIRED_VERSION=master

rm -fr ~/propositions-relatives-au-ric-test
git clone git@github.com:1718-io/propositions-relatives-au-ric.git ~/propositions-relatives-au-ric-test
cd ~/propositions-relatives-au-ric-test
git checkout "${DESIRED_VERSION}"

export HUGO_BASE_URL=https://ric-carl.herokuapp.com/
export PATH=$PATH:/usr/local/go/bin 
cd /pokus.io/hugo/src/
hugo -b "${HUGO_BASE_URL}"

export DESIRED_VERSION=master

rm -fr ~/propositions-relatives-au-ric-test
git clone git@github.com:1718-io/propositions-relatives-au-ric.git ~/propositions-relatives-au-ric-test
cd ~/propositions-relatives-au-ric-test
git checkout "${DESIRED_VERSION}"

rm -fr ./.git/

export HUGO_BASE_URL=https://ric-carl.herokuapp.com/
export PATH=$PATH:/usr/local/go/bin 
cd /pokus.io/hugo/src/
hugo -b "${HUGO_BASE_URL}"

export DESIRED_VERSION=master

docker system prune -f --all && docker system prune -f --volumes
rm -fr ~/propositions-relatives-au-ric-test

git clone git@github.com:1718-io/propositions-relatives-au-ric.git ~/propositions-relatives-au-ric-test
cd ~/propositions-relatives-au-ric-test
git checkout "${DESIRED_VERSION}"
echo "# ------------------------------------------------------------------------------------------------------------------"
echo "Here below the exact git commit at the time of this test, so"
echo "you can reproduce this test anytime, from this git repo : "
echo "# ------------------------------------------------------------------------------------------------------------------"
git rev-parse HEAD
echo "# ------------------------------------------------------------------------------------------------------------------"

rm -fr ./.git/
# remove the Dockerfile command 
sed -i "s~COPY .git /pokus.io/hugo/src/~# COPY .git /pokus.io/hugo/src/~g" ./heroku.Dockerfile
cat ./heroku.Dockerfile | grep "COPY .git /pokus.io/hugo/src/"

source .heroku.env
export QUAY_OCI_IMAGE_TAG=0.0.1-heroku
docker-compose build hugo_heroku

Tests environment

On the machine where i run those tests, here are the versions of golang, hugo, docker, docker-compose, that are installed : 

jbl@pc-alienware-jbl:~/.hugo.extended/v0.78.2$ docker version
Client: Docker Engine - Community
 Version:           19.03.14
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        5eb3275d40
 Built:             Tue Dec  1 19:20:17 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.14
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       5eb3275d40
  Built:            Tue Dec  1 19:18:46 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
jbl@pc-alienware-jbl:~/.hugo.extended/v0.78.2$ docker-compose version
docker-compose version 1.25.0, build unknown
docker-py version: 4.1.0
CPython version: 3.8.6
OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020
jbl@pc-alienware-jbl:~/.hugo.extended/v0.78.2$ go version
go version go1.15.6 linux/amd64
jbl@pc-alienware-jbl:~/.hugo.extended/v0.78.2$ hugo version
Hugo Static Site Generator v0.78.2-959724F0 linux/amd64 BuildDate: 2020-11-13T10:08:14Z
jbl@pc-alienware-jbl:~/.hugo.extended/v0.78.2$ hugo env
Hugo Static Site Generator v0.78.2-959724F0 linux/amd64 BuildDate: 2020-11-13T10:08:14Z
GOOS="linux"
GOARCH="amd64"
GOVERSION="go1.15.1"

Tests Results

Okay, so now that I collected all those informations, here are the questions I want an answer to : 

git clone git@github.com:1718-io/propositions-relatives-au-ric.git ~/propositions-relatives-au-ric-test
cd ~/propositions-relatives-au-ric-test
git checkout "${DESIRED_VERSION}"

rm -fr ./.git/

export HUGO_BASE_URL=https://ric-carl.herokuapp.com/
export PATH=$PATH:/usr/local/go/bin 
cd /pokus.io/hugo/src/
hugo -b "${HUGO_BASE_URL}"
Cloning into '/home/jbl/propositions-relatives-au-ric-test'...
remote: Enumerating objects: 31, done.
remote: Counting objects: 100% (31/31), done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 670 (delta 8), reused 24 (delta 4), pack-reused 639
Receiving objects: 100% (670/670), 7.37 MiB | 2.08 MiB/s, done.
Resolving deltas: 100% (226/226), done.
Branch 'master' set up to track remote branch 'master' from 'origin'.
Switched to a new branch 'master'
-bash: cd: /pokus.io/hugo/src/: No such file or directory
Start building sites … 
ERROR 2020/12/27 14:01:11 Failed to read Git log: fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Total in 77 ms
Error: Error building site: logged 1 error(s)

jbl@pc-alienware-jbl:~$ export DESIRED_VERSION=master

docker system prune -f --all && docker system prune -f --volumes
rm -fr ~/propositions-relatives-au-ric-test

git clone git@github.com:1718-io/propositions-relatives-au-ric.git ~/propositions-relatives-au-ric-test
cd ~/propositions-relatives-au-ric-test
git checkout "${DESIRED_VERSION}"
echo "# ------------------------------------------------------------------------------------------------------------------"
echo "Here below the exact git commit at the time of this test, so"
echo "you can reproduce this test anytime, from this git repo : "
echo "# ------------------------------------------------------------------------------------------------------------------"
git rev-parse HEAD
echo "# ------------------------------------------------------------------------------------------------------------------"

rm -fr ./.git/
# remove the Dockerfile command 
sed -i "s~COPY .git /pokus.io/hugo/src/~# COPY .git /pokus.io/hugo/src/~g" ./heroku.Dockerfile
cat ./heroku.Dockerfile | grep "COPY .git /pokus.io/hugo/src/"

source .heroku.env
export QUAY_OCI_IMAGE_TAG=0.0.1-heroku
docker-compose build hugo_heroku
Total reclaimed space: 0B
Total reclaimed space: 0B
Cloning into '/home/jbl/propositions-relatives-au-ric-test'...
remote: Enumerating objects: 31, done.
remote: Counting objects: 100% (31/31), done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 670 (delta 8), reused 24 (delta 4), pack-reused 639
Receiving objects: 100% (670/670), 7.37 MiB | 6.23 MiB/s, done.
Resolving deltas: 100% (226/226), done.
Branch 'master' set up to track remote branch 'master' from 'origin'.
Switched to a new branch 'master'
# ------------------------------------------------------------------------------------------------------------------
Here below the exact git commit at the time of this test, so
you can reproduce this test anytime, from this git repo : 
# ------------------------------------------------------------------------------------------------------------------
f6c7c1d4e5d1ff34bf4052e33ee7aae08f9d283f
# ------------------------------------------------------------------------------------------------------------------
# COPY .git /pokus.io/hugo/src/
WARNING: The GIT_COMMIT_ID variable is not set. Defaulting to a blank string.
Building hugo_heroku
Step 1/48 : ARG ALPINE_OCI_IMAGE_TAG=${ALPINE_OCI_IMAGE_TAG}
Step 2/48 : ARG GOLANG_VERSION=${GOLANG_VERSION:-1.15.6}
Step 3/48 : ARG HTTPD_OCI_IMAGE_TAG=${HTTPD_OCI_IMAGE_TAG}
Step 4/48 : FROM golang:$GOLANG_VERSION-alpine$ALPINE_OCI_IMAGE_TAG AS hugo_build_base
1.15.6-alpine: Pulling from library/golang
801bfaa63ef2: Pull complete
ee0a1ba97153: Pull complete
1db7f31c0ee6: Pull complete
ecebeec079cf: Pull complete
63b48972323a: Pull complete
Digest: sha256:49b4eac11640066bc72c74b70202478b7d431c7d8918e0973d6e4aeb8b3129d2
Status: Downloaded newer image for golang:1.15.6-alpine
 ---> 1463476d8605
Step 5/48 : ARG ALPINE_OCI_IMAGE_TAG=${ALPINE_OCI_IMAGE_TAG:-'latest'}
 ---> Running in 47889e2641ae
Removing intermediate container 47889e2641ae
 ---> baa5bbc46aaf
Step 6/48 : ARG GOLANG_VERSION=${GOLANG_VERSION}
 ---> Running in c4cfb40b3ba6
Removing intermediate container c4cfb40b3ba6
 ---> af73eda1be93
Step 7/48 : ARG HUGO_VERSION=${HUGO_VERSION}
 ---> Running in 364e574d4dd9
Removing intermediate container 364e574d4dd9
 ---> 584b1990992a
Step 8/48 : ARG HUGO_BASE_URL=${HUGO_BASE_URL}
 ---> Running in faaace1e7382
Removing intermediate container faaace1e7382
 ---> ee2dec3d79b5
Step 9/48 : RUN echo "GOLANG_VERSION=[${GOLANG_VERSION}] and HUGO_VERSION=[${HUGO_VERSION}] and HUGO_BASE_URL=[${HUGO_BASE_URL}]"
 ---> Running in 946968c21873
GOLANG_VERSION=[1.15.6] and HUGO_VERSION=[0.78.2] and HUGO_BASE_URL=[https://ric-carl.herokuapp.com/]
Removing intermediate container 946968c21873
 ---> f866517afee7
Step 10/48 : USER root
 ---> Running in 4de644e69369
Removing intermediate container 4de644e69369
 ---> 97c185fe5233

(HERE I skipped a lot of stdout )

GOVERSION="go1.15.6"
Removing intermediate container 4204bd69aae2
 ---> 51fa0198b17e

Step 17/48 : FROM hugo_build_base AS hugo_build
 ---> 51fa0198b17e
Step 18/48 : RUN mkdir -p /pokus.io/hugo/src/
 ---> Running in 4c09ffc937c2
Removing intermediate container 4c09ffc937c2
 ---> 0bb34885e101
Step 19/48 : COPY . /pokus.io/hugo/src/
 ---> fa23cec02aac
Step 20/48 : RUN ls -allh /pokus.io/hugo/src/
 ---> Running in 7d325cbd3ea8
total 140K   
drwxr-xr-x    1 root     root        4.0K Dec 27 19:53 .
drwxr-xr-x    1 root     root        4.0K Dec 27 19:53 ..
drwxr-xr-x    2 root     root        4.0K Dec 27 19:48 .circleci
-rw-r--r--    1 root     root         110 Dec 27 19:48 .env
-rw-r--r--    1 root     root          32 Dec 27 19:48 .gitignore
-rw-r--r--    1 root     root         112 Dec 27 19:48 .heroku.env
-rw-r--r--    1 root     root        4.2K Dec 27 19:48 GITFLOW.md
-rw-r--r--    1 root     root        4.3K Dec 27 19:48 README.md
drwxr-xr-x    4 root     root        4.0K Dec 27 19:48 assets
-rw-r--r--    1 root     root        1.7K Dec 27 19:48 config.toml
drwxr-xr-x    3 root     root        4.0K Dec 27 19:48 content
-rw-r--r--    1 root     root        1.2K Dec 27 19:48 docker-compose.yml
drwxr-xr-x   10 root     root        4.0K Dec 27 19:48 docs
-rw-r--r--    1 root     root         118 Dec 27 19:48 go.mod
-rw-r--r--    1 root     root        9.2K Dec 27 19:48 go.sum
-rw-r--r--    1 root     root        5.3K Dec 27 19:48 heroku.Dockerfile
-rw-r--r--    1 root     root         715 Dec 27 19:48 heroku.alpine.hugo-extended.setup.sh
-rw-r--r--    1 root     root        1.7K Dec 27 19:48 heroku.apache.start.sh
-rw-r--r--    1 root     root         271 Dec 27 19:48 heroku.container.healthcheck.sh
-rw-r--r--    1 root     root         248 Dec 27 19:48 heroku.yml
-rw-r--r--    1 root     root       20.0K Dec 27 19:48 httpd.conf
drwxr-xr-x    2 root     root        4.0K Dec 27 19:48 i18n
drwxr-xr-x    4 root     root        4.0K Dec 27 19:48 pipeline
drwxr-xr-x    3 root     root        4.0K Dec 27 19:48 resources
drwxr-xr-x    6 root     root        4.0K Dec 27 19:48 static
drwxr-xr-x    4 root     root        4.0K Dec 27 19:48 themes
Removing intermediate container 7d325cbd3ea8
 ---> 231f8589a029
Step 21/48 : RUN export PATH=$PATH:/usr/local/go/bin && cd /pokus.io/hugo/src/ && hugo -b "${HUGO_BASE_URL}"
 ---> Running in 80ac9295c2ab
Start building sites … 
ERROR 2020/12/27 19:53:34 Failed to read Git log: fatal: not a git repository (or any of the parent directories): .git
Total in 85 ms
Error: Error building site: logged 1 error(s)
ERROR: Service 'hugo_heroku' failed to build: The command '/bin/sh -c export PATH=$PATH:/usr/local/go/bin && cd /pokus.io/hugo/src/ && hugo -b "${HUGO_BASE_URL}"' returned a non-zero code: 255

Note, as seen in the stdout of the last test, that you may reproduce all those 4 tests, from the commit in this git repo identified qith the hash f6c7c1d4e5d1ff34bf4052e33ee7aae08f9d283f

Conclusion

I believe now, That I have made clear, that the Heroku platform does deletes the .git/ folder from the docker build context , when you deploy using a heroku.yml and a heroku stack set to container ( heroku stack:set container --app ${HEROKU_APP_ID} )

Which is why, I have here a clear case, were the build of a software, is impossible with Heroku platform, using the container stack, and the heroku.yml

Since there is zero way you can freely address any issue to the Heroku team, well now the best option I have to solve this issue is this :

Ok, let's got for that new option

Jean-Baptiste-Lasselle commented 3 years ago

Deploying to Heroku by docker pushing an existing image

export DESIRED_VERSION=0.0.2
docker system prune -f --all && docker system prune -f --volumes
rm -fr ~/propositions-relatives-au-ric-test

git clone git@github.com:1718-io/propositions-relatives-au-ric.git ~/propositions-relatives-au-ric-test
cd ~/propositions-relatives-au-ric-test
git checkout "${DESIRED_VERSION}"
source .heroku.env
export QUAY_OCI_IMAGE_TAG=0.0.1-heroku
docker-compose build hugo_heroku

export HEROKU_APP_ID=ric-carl
# the process type, is the thing that you can find in the `heroku.yml` (for me it's `web`, and
# there are others, which I don't care about like for backends I guess, stuff like that)
export HEROKU_PROCESS_TYPE=web
export SECRETHUB_ORG=ric1718
export SECRETHUB_REPO=cicd
export HEROKU_API_KEY=$(secrethub read "${SECRETHUB_ORG}/${SECRETHUB_REPO}/carlbot/heroku/api-token")

docker login --username=_ --password=${HEROKU_API_KEY} registry.heroku.com

docker tag quay.io/ric1718/une_proposition:${QUAY_OCI_IMAGE_TAG} registry.heroku.com/${HEROKU_APP_ID}/${HEROKU_PROCESS_TYPE}
docker push registry.heroku.com/${HEROKU_APP_ID}/${HEROKU_PROCESS_TYPE}

# After the image is pushed, you have one more thing to do, to complete the deployment. 
# This thing is invoking one Heroku API Endpoint, to "release" the pushed container. Like this :  

docker system prune -f --all && docker system prune -f --volumes

export DOCKER_IMAGE_GUN=registry.heroku.com/${HEROKU_APP_ID}/${HEROKU_PROCESS_TYPE}
docker pull ${DOCKER_IMAGE_GUN}
export DOCKER_IMAGE_ID=$(docker inspect ${DOCKER_IMAGE_GUN} --format={{.Id}})

# --- 
# If using the [--netrc] ption, the auth. credentials have to be in the [~/.netrc] file.
# 
# curl --netrc -X PATCH https://api.heroku.com/apps/$APP_ID_OR_NAME/formation \
#   -d '{
#   "updates": [
#     {
#       "type": "${HEROKU_PROCESS_TYPE}",
#       "docker_image": "$WEB_DOCKER_IMAGE_ID"
#     }
#   ]
# }' \
#   -H "Content-Type: application/json" \
#   -H "Accept: application/vnd.heroku+json; version=3.docker-releases"
# ---

curl -iv -H "Authorization: Bearer ${HEROKU_API_KEY}" -X PATCH https://api.heroku.com/apps/$HEROKU_APP_ID/formation \
  -d "{
  \"updates\": [
    {
      \"type\": \"${HEROKU_PROCESS_TYPE}\",
      \"docker_image\": \"${DOCKER_IMAGE_ID}\"
    }
  ]
}" \
  -H "Content-Type: application/json" \
  -H "Accept: application/vnd.heroku+json; version=3.docker-releases"

References :

Jean-Baptiste-Lasselle commented 3 years ago

awesome, so much faster deployment method than triggering a build inside the Heroku platform :)