search

appsmithorg / appsmith

Platform to build admin panels, internal tools, and dashboards. Integrates with 25+ databases and any API.
https://www.appsmith.com
Apache License 2.0
33.92k stars 3.66k forks source link

[Epic] Git Sync & Version Control #3199

Closed Nikhil-Nandagopal closed 2 years ago

Nikhil-Nandagopal commented 3 years ago

Problem statement

Today we get a lot of requests around migrating applications between instances as well as moving applications across organizations. There is also a need to have some kind of version control that will make it easy to restore the state of applications. This should also make it easy for other developers in a team to do code reviews of changes to applications. Changes to an application by a developer should not overwrite the changes other developers make.

Related issues

Solution

The Story

Discovery

A developer comes to appsmith and wants to try it out. She runs a docker command and gets it running on her local system #4430. A browser window opens and she is welcomed. She mentions that she is developing locally and the mode of the app changes to offline mode. She goes through our onboarding. She is prompted to connect to her data source. She is able to easily connect to her local API / DB without NGrok. She is in an offline mode and is not able to share/invite users to her application.

Sharing

She builds her first application with a lot of excitement and wants to share it so everyone gets access to it. This prompts her to connect a Git Repository and select a directory to commit her app & docker-compose file to. The readme of the repository is also updated with

  1. The steps of starting the application locally
  2. The documentation links
  3. The discord support links
  4. The name / image of the app

https://github.com/Nikhil-Nandagopal/codesandbox

After the git repo is connected, the user is asked to connect to deploy a cloud version and is prompted to either select appsmith cloud or their own private cloud. Selecting appsmith cloud will ask the user to create / sign in to an account. Once this is successful, the user has to select an organisation and the app gets created in that org.

If the user select private cloud, they are prompted to deploy appsmith using the steps below

They must then enter the URL of their appsmith login page and follow the steps to login / select the org. She accesses her appsmith cloud and is prompted to create the datasources that she created locally. She then invites all her users to use the app. She should be able to change the cloud environment she is pointing to.

Iteration

Her users get invites to the application and begin trying it out. They leave her feedback using the commenting feature which she receives via email. She opens the application on her local and is able to see all the comments. She creates a branch of the application and makes changes to it. She configures new datasources on this branch that point to her staging environment. She is able to make changes to the application on this branch and test them out. She then hits deploy which commits the changes to the new git branch and she is given the link of the branch. She now raises a pull request from this branch to master and pings a colleague to review/test the changes. The colleague goes over the PR, uses a visual diff tool, tests it, and approves it. The PR is merged to master and the new changes of the application are deployed to all users. The actions do not change the datasource they were pointing to on merge.

Evangelism

As more developers review her PR, they are able to see a link of the cloud instance of the tool in the PR to review & test. They try out the tool and are automatically able to get access to it by authenticating with a common Google domain or setting it up locally.

They begin to learn more about the tool and try to create an app. All the underlying deployment & environment infrastructure is already set up for them. New developers in the team are easily able to check out the app locally and request access to take their changes online.

Every new app developed locally must be committed to a git repo and select an organization in the cloud to be a part of.

Git Syncing

App branches may go out of sync or have merge conflicts and we should intimate developers of this and allow them to manually update their branches as well as inform them of conflicts and links of where to resolve them. Users can authentication with a git repo specifying their username/password or their public key. Users should enter the git path to connect to a repo. Every commit should autogenerate a message. Users can manually specify a message for certain commits. (Enhancement) Allow users to authenticate with Github to select their git repo and credentials.

Summary

  1. Developers can easily bring appsmith up locally using a docker-compose file. #4430
  2. There is a version of appsmith in their cloud which business users use.
  3. All local instances can switch between online/offline versions. The offline version is limited without auth, users, orgs and contains apps from all the repos that the user has access to.
  4. Developers can create a version of an app from anywhere (cloud / local) & commit it to a git branch in the global git repo.
  5. The cloud backend is reading/updating each application from a git branch.
  6. Comments and feedback can be given on individual versions of the application on the git branch

CE Constraints

  1. Each commit comes with a hash generated by the instance. This hash contains the hash of the version and a secret of the instance to prevent rehydration on any system that is not the instance that created it. This prompts larger companies that need staging and production on separate sandboxes to pay for us.
  2. Environments will not be available and all datasources are tracked against the branch (except secrets)

Flows

  1. Deploying Appsmith Locally #4430
  2. Connect to their local data sources
  3. Switch between offline & live modes
  4. Deploy their app to a cloud instance
  5. Connect their app to a git repo
  6. Commit changes
  7. Create branches for their apps & Sync apps across branches
  8. Deploying changes to branches

Details

The Git Repository must have a clean human-readable structure. Pages -- Page1 -- Widgets ---- Widget1 ---- Widget2 ---- Widget3 -- Actions ---- Action1 ---- Action2 Datasources -- Datasource1

Each widget / action / data source should have a clean JSON of it's properties The git repository must have a readme with the information of the application and appsmith.

Figma

https://www.figma.com/file/TdPXDzYZvb11PYUmVAU4r1/Local-Dev-Flows?node-id=0%3A1&viewport=683%2C-1019%2C0.2663029432296753

https://www.figma.com/file/dHbJ2KNXa9597W6BzsFIuE/Local-Dev-Git-Sync?node-id=1%3A594

Test env

http://ec2-3-108-64-217.ap-south-1.compute.amazonaws.com

Docs (wip)

https://docs.google.com/document/d/1765hZEen2sip_LwXF5C9_hfTLE7FZaM9X4zAo_gCHrc/edit?usp=sharing

marcius commented 3 years ago

This would be an awesome feature!

sharat87 commented 3 years ago

Also slightly related: #3424.

srijanshetty commented 3 years ago

This would be a great addition!

ioloc commented 3 years ago

This is for sure, my most anticipated feature :)

ec-raphael commented 3 years ago

This is something we'd love as well. We see AppSmith as a nice companion to tools like Hasura. Open source, but with a SaaS option for cases where we can't/don't want to deal with hosting.

The open source bit is important because it lets us run the full suite of tools on a developer machine (as opposed to something like Retool). With Hasura, we love the declarative configuration, which you can export/import, and allows a developer to create a configuration locally and apply it in staging/prod. Environment variables are used for things like api, endpoints, secrets, etc.

The reason this could be so magical with a tool like AppSmith is it would allow a developer who has intimate knowledge of the system they are building to quickly put together admin tools as a part of that system. With platforms like Retool, etc. the admin side feels pretty disconnected from dev. It feels like an "addon" rather than part of a system.

For us, this is the potential differentiator with platforms like Retool. For now, the differentiator is more control, which is honestly far less appealing. Maybe cost? Depends on the setup.

Nikhil-Nandagopal commented 3 years ago

@ec-raphael that's a really neat take on this feature. We'd love to hear more from you on your thoughts about this as a product differentiator. Would you be up for a quick chat? This is my calendly, I'd love for just 30mins of your time to hear your ideas https://calendly.com/appsmith-nikhil/appsmith-meeting

ericwooley commented 3 years ago

I'm in the same situation as @ec-raphael, currently, we are using Hasura with retool to quickly build out some proof of concepts.

The only reason we are using retool is that it has better graphql support. I was very tempted to use app smith anyways, because I love the direction, but asking other devs to manually translate grapqhl in regular post requests was too much. Even with that, if there were git versioning, I think we would have gone with app smith.

I even considered using appsmith with a mongo docker instance, that had it's db volume mounted inside our git repo, so we could just commit that, but it seemed a little too janky. I don't think I want to see what a merge conflict would look like.

While your considering this, I just wanted to point out that hasura uses yaml, and nx.dev uses json to keep track generated things. Having dealt with merge conflicts in each, yaml is soooooooo much easier to handle conflicts in. Not having all the opening and closing brackets (which don't technically conflict), is really helpful.

If git integration does happen, I would use app smith for sooooo so many projects, that I use CRA, or nextjs with vercel for now. Especially if you can create a vercel like workflow for deployments.

Great project! Keep up the awesome work!

ericwooley commented 3 years ago

OH, I got carried away and forgot the original reason I commented.

Since you are using mongo now, it might not be that hard to implement this feature using https://github.com/louischatriot/nedb

Nikhil-Nandagopal commented 3 years ago

HI @ericwooley thanks for your input! YAML does seem like a better option for resolving conflicts. We're now releasing a quick import/export feature which you can write a job on top of to commit to git. We'll work on improving this to directly sync to GitHub as well and include YAML instead of JSON as an option.

I'm interested to know what you feel is lacking in our GraphQL Support. Would you be up for a quick chat regarding this? This is my Calendly https://calendly.com/appsmith-nikhil/appsmith-meeting

ereio commented 3 years ago

The lack of version control is a major blocker in my experience for getting teams approval for appsmith or similar. It would make a pivotal difference if appsmith had this ability! :)

Nikhil-Nandagopal commented 3 years ago

Hi @ereio 👋 We're picking this up soon! You can expect to see something by end of July :)

rishabhrathod01 commented 3 years ago

Showing Difference for JSON can be something similar to the UI in DiffyCurl

ereio commented 3 years ago

No rush, just wanted to check in and see where this is in priority. Still hoping to work on the feature soon?

Nikhil-Nandagopal commented 3 years ago

@ereio development is already well underway! You can expect an early preview at the end of the month 😁

polwayne commented 3 years ago

Any update when this will be released? :-)

Nikhil-Nandagopal commented 3 years ago

Hey @polwayne 👋 it's scheduled for release late this month 🙂

roitalpaz commented 2 years ago

Wow, amazing feature!!

Any schedule for release? I saw it was supposed to be released in the previous month :))) is possible to clone an early release and help with testing?

Nikhil-Nandagopal commented 2 years ago

Hello @roitalpaz 👋 This has been delayed due to unforeseen technical challenges. We're still hard at work on it and we're optimistic about a beta preview in another 2 weeks :) You can look at the early release in our test env here http://ec2-3-108-64-217.ap-south-1.compute.amazonaws.com

RakshaKShetty commented 2 years ago

Adding the Testplan Link here for reference https://docs.google.com/spreadsheets/d/1bLZb7lDy8CySKa0oByFUS7ICw3BWfnMKZxqqwr8dxHE/edit#gid=0\

rishabhsaxena commented 2 years ago

Pending decisions/inputs needed:

More details can be found here: https://www.notion.so/appsmith/Gitsync-and-version-control-0c0ef0365495400db85e7e209c3d6c79#a6ee85d90c0740028198696683776a59

rishabhsaxena commented 2 years ago

Blockers for going to prod with the beta version:

rishabhsaxena commented 2 years ago

Next set of action items:

rishabhsaxena commented 2 years ago

Posting the updates from the last catchup, for ref:

BE:

FE

Design

rishabhsaxena commented 2 years ago

Design changes:

Action items:

AnaghHegde commented 2 years ago
hiteshjoshi commented 2 years ago
abhvsn commented 2 years ago
AnaghHegde commented 2 years ago
abhvsn commented 2 years ago

Next set of action items:

abhvsn commented 2 years ago

Meeting notes

Good to have:

Requires discussion:

abhvsn commented 2 years ago

Next set of action

abhvsn commented 2 years ago

Next set of action

abhvsn commented 2 years ago

Next set of action items:

hiteshjoshi commented 2 years ago

Closing this.