[ELE-208] Add gitlab guide to documentation

[clad]

As a new user of elementary, I would like some documentation on adding elementary to gitlab ci. If nothing else, a sample gitlab-ci.yml would help gitlab users. I think the safe assumption would be that the user already has a working gitlab ci/cd setup with dbt, and wants to add elementary to it.

_ELE-208

[elongl]

Hi @clad! By a little help from the GitLab docs, I was able to get a good starting point for what .gitlab-ci.yml should look like, however, I did not test it myself at the moment.

Here's the file (make sure to replace <ADAPTER>):

stages:
  - monitor

elementary:
  stage: monitor
  script:
    - apt-get update -qy && apt-get install -y python-dev python-pip # This might not be required.
    - mkdir -p ~/.dbt && echo "$PROFILES_YML" > ~/.dbt/profiles.yml
    - pip install elementary-data[<ADAPTER>]
    - edr monitor --slack-webhook $SLACK_WEBHOOK

You'll need to create two variables in Settings > CI/CD > Variables in your GitLab project:

• PROFILES_YML - The content of your profiles.yml file.
• SLACK_WEBHOOK - Your Slack webhook.

Please let me know what you think. If it works as expected we'll add it to the docs. Thanks a lot.

[clad]

@elongl This looks good to me, thanks!

[elongl]

@clad Just to make sure I understand, did you try it and it worked properly?

[esenilsson]

@elongl Hi, may I ask what image is used for the stage? ubuntu or your docker image

[clad]

@elongl - This sample works, minor edits needed for ubuntu image - not sure if that's the default. If image is alpine, then it would be apk and different package names.

image: ubuntu:latest

stages:
  - monitor

elementary:
  stage: monitor
  script:
    - apt-get update -qy && apt-get install -y python3-dev python3-pip
    - mkdir -p ~/.dbt && echo "$PROFILES_YML" > ~/.dbt/profiles.yml
    - pip install elementary-data[<ADAPTER>]
    - edr monitor --slack-webhook $SLACK_WEBHOOK

[elongl]

@esenilsson @clad That's actually a great point! I think it'd be much better to utilize the official Docker image.

Could you try the following .gitlab-ci.yml and share how it turned out?

image:
  name: ghcr.io/elementary-data/elementary:latest
  entrypoint: [""]

stages:
  - monitor

elementary:
  stage: monitor
  script:
    - mkdir -p ~/.dbt && echo "$PROFILES_YML" > ~/.dbt/profiles.yml
    - edr monitor --slack-webhook "$SLACK_WEBHOOK"

Thanks a lot.

[clad]

@elongl

No errors other than my missing slack webhook - I cannot test the slack integration.

[elongl]

@clad Have you tested it with edr report? I'm asking because I want to see if the secrets work as expected. It could be that PROFILES_YML is still empty.

[clad]

@elongl

Yes, I have gotten to edr report w/o the slack piece as I do not have admin access to slack. Here's the gitlab pipeline output -

Using docker image sha256:ed114d240f0722c9a7ae1f80dad994917ed43ec51d074d387b8dc8a2a5978800 for ghcr.io/elementary-data/elementary:latest with digest ghcr.io/elementary-data/elementary@sha256:dff97adae953a58313979ebc0d75d9a689479acd1fe2a044a6e97d79820b242d ...
$ mkdir -p ~/.dbt && echo "$PROFILES_YML" > ~/.dbt/profiles.yml
$ edr monitor --slack-webhook "$SLACK_WEBHOOK"
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   
Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary
[20](https://gitlab.tangocard.com/engineering/data/dbt-fivetran/-/jobs/668680#L20)[23](https://gitlab.tangocard.com/engineering/data/dbt-fivetran/-/jobs/668680#L23)-01-[24](https://gitlab.tangocard.com/engineering/data/dbt-fivetran/-/jobs/668680#L24) 13:36:03 — INFO — Running with edr=0.6.7
Traceback (most recent call last):
  File "/usr/local/bin/edr", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/elementary/cli/cli.py", line 54, in invoke
    return super().invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1635, in invoke
    rv = super().invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line [26](https://gitlab.tangocard.com/engineering/data/dbt-fivetran/-/jobs/668680#L26), in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/elementary/monitor/cli.py", line 265, in monitor
    config.validate_monitor()
  File "/usr/local/lib/python3.10/site-packages/elementary/config/config.py", line 177, in validate_monitor
    raise InvalidArgumentsError(
elementary.exceptions.exceptions.InvalidArgumentsError: Either a Slack token and a channel or a Slack webhook is required.
Cleaning up project directory and file based variables

[elongl]

@clad @esenilsson Created the guide here! Thanks a lot for you help.