[Staging] New tasks in Kubernetes world

[szabozoltan69]

Issue

Due to the new Kubernetes system there are some new tasks:

• [ ] We should find an "advised workflow", what to do with develop and terraform branches, how to keep them parallel, when to merge. (CircleCI could be switched off or should be healed to avoid the "[IFRCGo/go-api] Run failed: Chartpress Build, Publish & Deploy - terraform" messages.)
• [x] We should avoid displaying green status of deployment in pipeline while the deployment is not finished.
• [ ] An env variable change (in pipeline's list) and a manually started deployment should result a changed env var in the vm.
• [ ] update_index should be done, when new elastic_search vm is created
• [x] 2 notifications arrive for the same new event (or appeal). Are the jobs run parallel twice somehow? If I comment out every rows in the ...-api crontab, only 1 notification arrives. Should we remove the whole crontab list?
• [x] Every new deployment creates a notification sending (or 2 ^ ). Why is it?
• [x] https://github.com/IFRCGo/go-api/blob/develop/api/management/commands/ingest_appeals.py#L90 - settings.APPEALS_USER should exist.
• [x] File upload seems to be not working. E.g. add a little image to the snippets in https://goadmin-stage.ifrc.org/en/api/event/4636/ - results a Server Error 500
• [x] The old blob storage files should be transported to the new storage (or should be used the old, off-kubernetes one)
• [ ] A basic knowhow is needed for the Loki/Graphana/Prometheus log checking.
• [x] https://github.com/IFRCGo/go-api/issues/1785 is related to this. FRONTEND_URL env var fixed.
• [ ] Recently env variables are kept in 8 (!) locations: docker-compose.yml, deploy/docker-compose.yml, deploy/terraform/resources/helm-ifrcgo.tf, deploy/terraform/resources/variables.tf, deploy/terraform/main.tf, deploy/terraform/variables.tf, deploy/helm/ifrcgo-helm/values.yaml, deploy/helm/ifrcgo-helm/templates/config/secret.yaml or deploy/helm/ifrcgo-helm/templates/config/configmap.yaml | It's a bit confusing. Could we reduce them? | A quick workaround: via a script, which inserts all needed 8 positions the variable (and value, where needed), when the need of a new variable introducing occurs.
• [ ] In the Azure container registry (ifrcgoacr, ifrcgo-api) a plenty of 2GB containers are collected. Should we clean up sometimes manually?
• [x] The red "!" in triangle leads to a nonexistent page, due to the missing /admin/ from the URL:
• [x] Let's change https://github.com/IFRCGo/go-api/blob/develop/main/settings.py#L404-L413 for the sake of k8s – in order to not appear in the log the below rows. APPLICATION_INSIGHTS_INSTRUMENTATION_KEY is OK in the k8s env.

  [2023-05-29 06:16:03,056] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,165] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,273] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,274] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,278] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,283] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,286] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,293] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,296] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,300] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,327] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,336] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,350] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,474] __init__: WARNING - Queue is full. Dropping telemetry.
  [2023-05-29 06:16:03,481] __init__: WARNING - Queue is full. Dropping telemetry.

[szabozoltan69]

@batpad @geohacker

[szabozoltan69]

Some logs about saving a "General Document":

    raise _wrap_exception(ex, AzureException)
azure.common.AzureException: Invalid URL 'http:///subscriptions/39308fb0-.../resourceGroups/ifrct...02rg/providers/Microsoft.Storage/storageAccounts/ifrctgo.blob.core.windows.net/api/documents/test1/a.jpg': No host supplied

[geohacker]

advised workflow", what to do with develop and terraform branches, how to keep them parallel, when to merge.

@szabozoltan69 we should delete the terraform branch once it's merged to develop. Which I think we can do right before going to production

In the Azure container registry (ifrcgoacr, ifrcgo-api) a plenty of 2GB containers are collected. Should we clean up sometimes manually?

We might have to do this manually. https://learn.microsoft.com/en-gb/azure/container-registry/container-registry-auto-purge

@szabozoltan69 @batpad I believe we are ready to switch to the k8s based stack for production too. Is there anything else blocking this move?

[szabozoltan69]

Yes (about blocking), it would be fine to have a sharp endpoint of the deployment (when the sign changes to green), because recently it changes to green, and "at some point it turns out that is is ready". (We should avoid displaying green status of deployment in pipeline while the deployment is not finished.) I suppose it's not a big deal, but makes things clear.

And also another thing: In https://goadmin-stage.ifrc.org/api/v2/brief the git-last-tag has no relation to the real value. I suppose the pipeline git value disturb this. We should find a solution to be able to check the deployed branch "git tag" hash (here in /brief). https://github.com/IFRCGo/go-api/blob/develop/api/views.py#L504-L505

[geohacker]

it would be fine to have a sharp endpoint of the deployment (when the sign changes to green), because recently it changes to green, and "at some point it turns out that is is ready".

I'm not sure if I understand this correctly. Are you referring to the azure pipeline behaviour?

And also another thing: In https://goadmin-stage.ifrc.org/api/v2/brief the git-last-tag has no relation to the real value.

@szabozoltan69 It's because for staging deploys we don't create git tags, but we can if needed. But the "git_last_commit": "ec1f00cd" would line up correctly with the commit that triggered the deploy https://github.com/IFRCGo/go-api/pull/1725/commits/ec1f00cd6fb1c666c46f08f847c7a8d411766cd5

[szabozoltan69]

@geohacker Yes, I refer to azure pipeline behaviour (in my first comment). @batpad mentioned somewhere a way how to avoid that pipeline gets green and the deployment is still not fully finished.

The git last commit is ok, but the last git tag (which is like 1.1.482) is not. If possible, it would be fine to be able to check that also. If it is too difficult, I can let it go.

[szabozoltan69]

Also it is a question in me, that is it really unavoidable that a new env variable should appear in 8 locations? Can't we reduce these somehow?

[szabozoltan69]

Also: a basic knowhow is needed for the Loki/Graphana/Prometheus log checking. Without this it makes no sense to do the go-live cutover.

[geohacker]

@szabozoltan69 @batpad I've added a section in the docs about how to access and view logs in Grafana and using kubectl https://github.com/IFRCGo/go-api/tree/terraform#see-logs-from-kubernetes

The git last commit is ok, but the last git tag (which is like 1.1.482) is not.

Creating tags per staging release seems unnecessary. If we want to do that for production that's ok. But that has nothing to do with the azure pipeline.

that is it really unavoidable that a new env variable should appear in 8 locations

@szabozoltan69 unfortunately, we can't avoid this right now. Though are there 8 locations? The value is set in azure pipeline secrets, then read from github actions, then used in Docker Compose to the container env, then in variables.tf and main.tf (that's 4-5 in total, right?) We can look into optimising this later on but there are no easy ways. It's better to be more verbose about variables than it being magically populated.

[szabozoltan69]

Ok, I can accept these. Otherwise it's really 8: docker-compose.yml, deploy/docker-compose.yml, deploy/terraform/resources/helm-ifrcgo.tf, deploy/terraform/resources/variables.tf, deploy/terraform/main.tf, deploy/terraform/variables.tf, deploy/helm/ifrcgo-helm/values.yaml, deploy/helm/ifrcgo-helm/templates/config/secret.yaml or deploy/helm/ifrcgo-helm/templates/config/configmap.yaml – though one of them is not for pipeline but for local development (but unavoidable).

[szabozoltan69]

About Graphana my question (howto) was not only to know the technical details how to reach it in the browser, but mostly what are the really effective queries to check relevant events, details about the system. So: "If you want to run custom queries, use the Explore tab." – but what custom queries should be relevant in our case? I suppose there are not 10000 interesting queries, but maybe 2-3 types.

[geohacker]

@szabozoltan69 You can see some examples in the Grafana docs here https://grafana.com/docs/loki/latest/logql/query_examples/ — but ideally you should just go to Explore and use the options there to frame a query if needed. It's all to look through logs and search for specific keywords. I'm pretty sure you'll be able to figure it out!

[szabozoltan69]

Refreshment of SSL certificates should be done on a non-manual way. @sunu