search

appsmithorg / appsmith

Platform to build admin panels, internal tools, and dashboards. Integrates with 25+ databases and any API.
https://www.appsmith.com
Apache License 2.0
34.26k stars 3.71k forks source link

[Epic] Improved Deployments #4430

Closed Nikhil-Nandagopal closed 2 years ago

Nikhil-Nandagopal commented 3 years ago

Objective

Simplify the following modes of deployment

Success Metrics

List of all metrics you are tracking and the desired goal. Goal Metric
Improved Installation Funnel 95%
Better Telemetry Capture Rate 50%

Requirements

Requirement User Story Importance Notes
Docker The user should be able to deploy appsmith with a single docker command on any OS. The container is rebuilt whenever a change to the frontend or backend occurs High
Docker The user should be able to bring up appsmith with a docker-compose file/run command High
K8s The user should be able to deploy on Kubernetes with a helm chart. We should have a public chart repo for appsmith with the default release High
AWS The user should be able to deploy on AWS with a single click using the AWS stack High
Heroku The user should be able to deploy on Heroku with a single click after setting up mongo with no additional configuration required High
UI Config The user can configure all environment variables via a UI. High
SSL The user should be able to install SSL via a single CLI command for all modes of deployment Medium
Logs The user should be able to view the logs of the container with a simple docker logs command Medium
Export The user should be able to easily take a dump of the instance and restore it Medium

Inspiration

Metabase

User Story

Users self-hosting appsmith should be welcomed first instead of being thrown onto a login screen. They should proceed to fill in the details of the administrator of the instance.

The user can now update the instance settings from the administrative section. This section allows users to configure their appsmith instance

Developer Handoff Document in Figma

Link to the developer Handoff Document:

sharat87 commented 3 years ago

Can we add monitoring to this? The self-hosted setup could come with an initial monitoring system as well (optionally), that will monitor the server, frontend, mongo etc. in one single place.

Debsourabh commented 3 years ago

WIP Figma File

https://www.figma.com/file/lqtUfzJO50RpEhxKttYeOg/Local-Dev?node-id=1%3A70

Nikhil-Nandagopal commented 3 years ago

New Comment by a user

embedded mongo db password in docker-compose.yml poor choice. Should go in an environment file that can be given stricter permissions

sharat87 commented 3 years ago

Opened #6345 for issue from above comment.

sharat87 commented 3 years ago

Admin Configuration Fields

Display Name Type Internal Name Detail
Instance Name Text APPSMITH_INSTANCE_NAME Name of your Appsmith instance. Usually your company name, or your team's name if it will be used solely by your team.
MongoDB URI Text APPSMITH_MONGODB_URI The URI to a MongoDB server, that will be used as the database.
Redis URI Text APPSMITH_REDIS_URL The URI to a Redis server, that will be used for storing sessions and other ephemeral information.
Enable Email Boolean APPSMITH_MAIL_ENABLED If true, email is enabled, and needs to be configured, using the other fields here. If false, Appsmith server won't be able to send any emails.
Email: From Address Email Address APPSMITH_MAIL_FROM The email address to use as the from address when sending emails.
Email: Reply To Email Address APPSMITH_REPLY_TO The email address to use for the Reply-To header for emails sent by Appsmith.
Email: SMTP Host Text APPSMITH_MAIL_HOST The host address of the SMTP server to use.
Email: SMTP Port Number APPSMITH_MAIL_PORT The port of the SMTP server to use.
Email: SMTP Username Text APPSMITH_MAIL_USERNAME Username to use for authenticating with the SMTP server, if authentication is enabled.
Email: SMTP Password Text APPSMITH_MAIL_PASSWORD Password to use for authenticating with SMTP server.
Email: Enable TLS for SMTP Boolean APPSMITH_MAIL_SMTP_TLS_ENABLED Whether to use TLS when authenticating with SMTP server, if authentication is enabled.
Signup: Disable Boolean APPSMITH_SIGNUP_DISABLED If true, signing up on this Appsmith instance will not be allowed.
Signup: Allowed Email Domains Boolean APPSMITH_SIGNUP_ALLOWED_DOMAINS When set to a comma separated list of domains, only emails with these domains will be allowed to signup.
Admin Email(s) Zero or more email addresses APPSMITH_ADMIN_EMAILS These email address(es) will be notified of instance-level changes and alerts. They will be allowed to signup, irrespective of above two settings.
Google ReCaptcha Site Key Text APPSMITH_RECAPTCHA_SITE_KEY Provide ReCaptcha Site key here, if you wish to use it for the signup page.
Google ReCaptcha Secret Text APPSMITH_RECAPTCHA_SECRET_KEY Provide ReCaptcha Secret key here, if you wish to use it for the signup page.
Google Maps API Key Text APPSMITH_GOOGLE_MAPS_API_KEY API Key for Google Maps. Providing this will enable the Maps widget to work.
Anonymous tracking Boolean APPSMITH_DISABLE_TELEMETRY Help Appsmith improve by sharing anonymous usage information.
heroic commented 3 years ago

Currently when deploying using k8s, there is no health probe. Sometimes connection to mongo or redis may end, but since no health route exists, there is no way for the pod to be restarted.

sharat87 commented 3 years ago

@Nikhil-Nandagopal, regarding the item Generate Docker Compose File, instead of generating just a docker-compose.yml file and having that download, should we give a zip with the docker-compose.yml file and the docker.env file, with the current env variables? The docker-compose.yml file would have a reference to this docker.env file, so I'm guessing just the yml file wouldn't be of much use.

Additionally, if the idea of providing this is to be able to export the whole environment in one click, to reproduce elsewhere, shouldn't it include the database dumps as well?

sharat87 commented 3 years ago

Currently when deploying using k8s, there is no health probe. Sometimes connection to mongo or redis may end, but since no health route exists, there is no way for the pod to be restarted.

Hey @heroic, you mean a health endpoint on the backend server, right? One option is to use /api/v1/users/me, and confirm it responds with 200 status code, and perhaps that the JSON body contains anonymousUser.

Let me know if I misunderstood your question.

Nikhil-Nandagopal commented 3 years ago

@sharat87 the zip sounds good. The idea for this was quick replication for the DB admin. I think with git sync the need for a DB dump will go away. Only the users will need to be re-added which should be fine once we create domain wide sign ups & invite links

heroic commented 3 years ago

Currently when deploying using k8s, there is no health probe. Sometimes connection to mongo or redis may end, but since no health route exists, there is no way for the pod to be restarted.

Hey @heroic, you mean a health endpoint on the backend server, right? One option is to use /api/v1/users/me, and confirm it responds with 200 status code, and perhaps that the JSON body contains anonymousUser.

Let me know if I misunderstood your question.

Yes. I ended up using /applications

sharat87 commented 3 years ago

@sbalaji1192, here's the variable names for some of the configs:

DiptoChakrabarty commented 3 years ago

Currently when deploying using k8s, there is no health probe. Sometimes connection to mongo or redis may end, but since no health route exists, there is no way for the pod to be restarted.

Hey @heroic, you mean a health endpoint on the backend server, right? One option is to use /api/v1/users/me, and confirm it responds with 200 status code, and perhaps that the JSON body contains anonymousUser.

Let me know if I misunderstood your question.

Hey ,can I send a PR to add this liveness probe in the backend deployment file. My plan is to setup a http request probe to the /api/v1/users/me path with a period of 5 seconds.

mohanarpit commented 3 years ago

Currently when deploying using k8s, there is no health probe. Sometimes connection to mongo or redis may end, but since no health route exists, there is no way for the pod to be restarted.

Hey @heroic, you mean a health endpoint on the backend server, right? One option is to use /api/v1/users/me, and confirm it responds with 200 status code, and perhaps that the JSON body contains anonymousUser. Let me know if I misunderstood your question.

Hey ,can I send a PR to add this liveness probe in the backend deployment file. My plan is to setup a http request probe to the /api/v1/users/me path with a period of 5 seconds.

@DiptoChakrabarty Sure. Please go ahead.

Nikhil-Nandagopal commented 3 years ago

@sharat87 are the same events being fired for when telemetry is opted in / out?

magick93 commented 3 years ago

For k8s deployments it should be possible to deploy without the need for clusterscope.

The installation should be entirely confined to a single namespace.

For example, I believe it should be possible to functionally achieve the below but without the need for a ClusterRoleBinding.

{{- if and .Values.autoupdate.enabled (ne .Values.autoupdate.scheduler "") }}
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: imago
  namespace: {{ include "appsmith.namespace" . }}
spec:
  schedule: {{ .Values.autoupdate.scheduler | quote }}
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            k8s-app: imago
        spec:
          restartPolicy: Never
          serviceAccount: imago
          serviceAccountName: imago
          containers:
            - name: imago
              image: philpep/imago
              imagePullPolicy: Always
              args: ["--update"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: imago
  namespace: {{ include "appsmith.namespace" . }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: imago
rules:
  - apiGroups:
      - ""
      - apps
    resources:
    - pods
    - replicasets
    - statefulsets
    verbs:
    - list
  - apiGroups:
      - ""
      - batch
    resources:
    - cronjobs
    verbs:
    - get
    - list
    - update
  - apiGroups:
      - ""
      - apps
    resources:
      - daemonsets
      - deployments
      - statefulsets
    verbs:
      - get
      - list
      - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: imago
roleRef:
  kind: ClusterRole
  name: imago
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: imago
    namespace: {{ include "appsmith.namespace" . }}
{{- end }}

Thanks

jalberto commented 2 years ago

Hi, in the latest blog post Helm support has been announced, but I cannot find the docs for it

sharat87 commented 2 years ago

Hey @jalberto, you can checkout https://github.com/appsmithorg/appsmith/tree/release/deploy/helm where we have some documentation for this. We are working on including this in our official docs.appsmith.com soon. Thanks!

sharat87 commented 2 years ago

Action Items: @sharat87 -- Review and merge helm charts, infra compliance changes. @sub1983 -- Work with GeekUP for IaC changes.

hiteshjoshi commented 2 years ago

Closing this.

Nikhil-Nandagopal commented 2 years ago

@hiteshjoshi I still see 10 open issues in the epic. What are we doing about those?

hiteshjoshi commented 2 years ago

@Nikhil-Nandagopal They will still be open, the problem is DevOps is being tracked on Notion + EE repo due to a lot of restrictions. We will close them but the priority have changed a bit for some of the open issues.