Github Actions for Deployment

[Perevalov]

Context

We need to ensure proper CI/CD for this project. Therefore we'll use Github Actions for that to (1) build a docker image, (2) push it to docker hub, (3) deploy it on our server. Below you'll find a reference GH actions file:

name: Docker Image CI

env:
  DOCKER_IMAGE_NAME: dbpedia/dbpedia-chatbot-backend
  DOCKER_CONTAINER_NAME: dbpedia-chatbot-backend

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3
      - name: Get current version
        run: >
          echo "X_APP_VERSION=$(cat < main.py | grep -Po '(?<=__version__ = \")([^\"]*)')" >> $GITHUB_ENV
      - name: Build the Docker image for deployment
        run: >
          docker build --tag "$DOCKER_IMAGE_NAME:latest" .
      - name: Tag image with specific version
        run: >
          docker tag "$DOCKER_IMAGE_NAME:latest" "$DOCKER_IMAGE_NAME:$X_APP_VERSION"
      - name: Docker Login
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DBP_DOCKERHUB_CREDENTIAL_USERNAME }}
          password: ${{ secrets.DBP_DOCKERHUB_CREDENTIAL_TOKEN_PUSHIMAGES }}
      - name: Push docker image with latest version
        run: docker push "$DOCKER_IMAGE_NAME:latest"
      - name: Push docker image with specific version
        run: docker push "$DOCKER_IMAGE_NAME:$X_APP_VERSION"
  deploy:
    runs-on: ubuntu-latest
    needs: build

    steps:
      - name: SSH into server
        uses: appleboy/ssh-action@master
        with:
          host: ${{ secrets.SERVER_HOST }}
          username: ${{ secrets.SERVER_USERNAME }}
          password: ${{ secrets.SERVER_PASSWORD }}
          port: ${{ secrets.SERVER_PORT }}
          envs: DOCKER_IMAGE_NAME,DOCKER_CONTAINER_NAME
          script: |
            docker stop "$DOCKER_CONTAINER_NAME" || true && docker rm "$DOCKER_CONTAINER_NAME" || true
            echo "Pulling image: $DOCKER_IMAGE_NAME" 
            docker pull $DOCKER_IMAGE_NAME:latest
            echo "Starting container: $DOCKER_CONTAINER_NAME" 
            docker run --restart=always -d -p 40197:8080 --name "$DOCKER_CONTAINER_NAME" "$DOCKER_IMAGE_NAME:latest"

This file is to be used for this project. Note, that it uses Github Secrets e.g. secrets.SERVER_HOST which was already set by @Perevalov.

Definition of Done

• [x] The reference GH actions .yml was analyzed and understood
• [x] The .yml was adapted (if needed) and pushed to the repository
• [x] The functionality was checked on the main branch (ideally, you'll see it deployed at http://141.57.8.18:40194)

[muskan-k]

@Perevalov for building the docker image, there is currently no Dockerfiles for the entire application except the two backed services for which we created the Dockerfiles. In the workflow, I will be building the images for these two services only correct? in that case there will be two builds.

I have built the webhook service only for now. It fails during deploy with the error: 2023/07/04 14:55:12 dial tcp ***:***: connect: connection refused

[Perevalov]

for building the docker image, there is currently no Dockerfiles for the entire application except the two backed services for which we created the Dockerfiles. In the workflow, I will be building the images for these two services only correct? in that case there will be two builds.

Yes, this is correct: 2 Dockerfiles -- webhook and rdf vis

[Perevalov]

I have built the webhook service only for now. It fails during deploy with the error: 2023/07/04 14:55:12 dial tcp ***:***: connect: connection refused

@muskan-k Change port to 40194

[muskan-k]

still says connection refused. @Perevalov I've updated the workflow to build and deploy both the services. build is successful. gets stuck at the last step due to connection refused.

[Perevalov]

still says connection refused. @Perevalov I've updated the workflow to build and deploy both the services. build is successful. gets stuck at the last step due to connection refused.

It seems that something wrong was with the secrets. Now, if you see at the action's logs, it actually connects to the server and tries to run the defined scripts. You can proceed with your work.

Remark: webhook and rdfviz have to be different images not one image with different tags. So, please give them meaningful names and change the yml respectively (avoid using slashes in image names). Here you can check how your images are appearing after a push: https://hub.docker.com/u/dbpedia

[muskan-k]

@Perevalov I need to run two images now. Since I'll be using the port 40194 for the first service, the second service cannot use the same port again. Do we have another port or is there another fix to run two different containers on same port?

[muskan-k]

@Perevalov the deployment is successful for both webhook and rdfviz. Webhook URL: http://141.57.8.18:40194 Tested /health endpoint RDF Viz URL: http://141.57.8.18:40193 Tested /visualize/urn:graph:24a1132b-6c7d-4770-8f30-e87cd43f8cf8 endpoint The workflow ran successfully as well. I've updated the yml file.

You can also try from your end and close this issue if everything works fine.

[Perevalov]

@muskan-k Take a look at Dockerhub e.g. rdfviz: https://hub.docker.com/r/dbpedia/dbpedia-chatbot-backend-rdfviz/tags. There is an inappropriate tag:

X_APP_VERSION in the .yml file should be used as a variable i.e. $X_APP_VERSION

[muskan-k]

If I try to tag the image with the value of X_APP_VERSION, the build fails because that variable for some reason does not have any value. The version is not being stored to the variable. @Perevalov

[Perevalov]

If I try to tag the image with the value of X_APP_VERSION, the build fails because that variable for some reason does not have any value. The version is not being stored to the variable. @Perevalov

You may try to test the command (https://github.com/dbpedia/dbpedia-chatbot-backend/blob/dd29ee14b33c23a571b9384d7d98dbb3b3380506/.github/workflows/deploy.yml#L20C33-L20C42) locally in your terminal. This line is just trying to find something like __version__ = "0.0.1" in file main.py, so the value of X_APP_VERSION is 0.0.1 (I previously provided it to you from one of my github actions), so you probably need to change it because there is no main.py file in the repository.

[muskan-k]

@Perevalov I've resolved the app version issue. The image should be tagged with 0.0.1 now. Could you please verify and let me know? The docker hub is not accessible to me for some reason. It says "Something went wrong" when I open the URL.

[muskan-k]

@Perevalov Also, the URL with port http://141.57.8.18:40194/ (for webhook service) does not work. Is this normal and is this how it is supposed to be? Because this URL was used for creating the proxy. The URL for rdf viz works fine though. I thought since they're both been deployed, it would work. Let me know if that's how it is supposed to be?

[Perevalov]

@Perevalov I've resolved the app version issue. The image should be tagged with 0.0.1 now. Could you please verify and let me know? The docker hub is not accessible to me for some reason. It says "Something went wrong" when I open the URL.

Yes, both docker images are now being tagged with the correct versions:

https://hub.docker.com/r/dbpedia/dbpedia-chatbot-backend-rdfviz/tags

https://hub.docker.com/r/dbpedia/dbpedia-chatbot-backend-webhook/tags

[Perevalov]

@Perevalov Also, the URL with port http://141.57.8.18:40194/ (for webhook service) does not work. Is this normal and is this how it is supposed to be? Because this URL was used for creating the proxy. The URL for rdf viz works fine though. I thought since they're both been deployed, it would work. Let me know if that's how it is supposed to be?

No, it's not intended. I just saw in the log of this container on the server the following:

Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 454, in main
    h11_max_incomplete_event_size=h11_max_incomplete_event_size,
  File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 576, in run
    server.run()
  File "/usr/local/lib/python3.7/site-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/uvicorn/server.py", line 67, in serve
    config.load()
  File "/usr/local/lib/python3.7/site-packages/uvicorn/config.py", line 479, in load
    self.loaded_app = import_from_string(self.app)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/importer.py", line 24, in import_from_string
    raise exc from None
  File "/usr/local/lib/python3.7/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./main.py", line 2, in <module>
    from fulfillment.model import handle_request
  File "./fulfillment/model.py", line 1, in <module>
    from helpers import intents
  File "./helpers/intents.py", line 6, in <module>
    import config
ModuleNotFoundError: No module named 'config'

Please, try to run and test the container locally to reproduce this error. Probably, some file structure or naming was changed and not tested?

[muskan-k]

@Perevalov resolved the error. The URL works fine now, the webhook service is up and handles webhook requests correctly. Also tested POST requests.

[muskan-k]

Closing this issue as the task is completed and tested successfully.