Epic: Remove need to commit to git (Gitpod "Inner Loop")

mickmister commented 2 years ago

Improve the speed that users can iterate on their configuration, particularly their .gitpod.yml in a way that doesn't require them to leave gitpod and/or start other workspaces. Doc or research progress.

There are some main configuration areas to improve on:

gitpod.yml - inclusive of the task commands
Image: Dockerfile updates / prebuilds
IDE integrations
Dotfiles iterations
Logs for feedback

Also useful to think on different dimensions:

Does it lint?
Does it build?
Does it run? (integrate)

In scope

Nice to have

Out of scope - Improve outer loop

Closes

Related issues:

Original issue description

## Context When I'm building a workspace, I oftentimes want to iterate on the .gitpod.yml file and test its behavior. My workflow is: - make a change to .gitpod.yml - commit and push change - open gitpod through the commit's/branch's GitHub url - now I have a workspace created from my edited .gitpod.yml file ### Problem with this approach Two things that are inconvenient about my current workflow: - I'm going to github as part of the workflow. I'd rather stay in gitpod while I'm iterating - I end up with an extra workspace that I need to delete ## Possible Solution Add a command to the gp CLI, which takes the current .gitpod.yml (and dockerfile) and starts a fresh workspace with an explicit prebuild stage (so it runs through the automation). The CLI would output the build logs (docker build, prebuild, tasks) to standard out and at the end (ready state) print a link so one can optionally go and open the IDE. The workspace stops when I terminate the process. We can then have IDE commands to run this. I think a code lens in the .gitpod.yml would be even better. Also, a notification when the config has changed could be useful for users to discover this.

svenefftinge commented 2 years ago

@mickmister this has been discussed more on a solution side in https://github.com/gitpod-io/gitpod/issues/4861 and https://github.com/gitpod-io/gitpod/issues/6818. I like that you framed it from the problem/user perspective, so would like to use this as the driving epic. That means we are updating the initial description as we discover this.

mickmister commented 2 years ago

@svenefftinge It would be interesting if the workflow could match the style of docker image development, how you can iterate on a dockerfile as it caches the execution of each step. It's more difficult in the .gitpod.yml case, because there are multiple scripts running at once, and probably a million other things I'm not aware of that makes it difficult to cache the steps.

I haven't tried the approach of using a dockerfile to setup the Gitpod image. In what's currently available, that may provide a faster workflow of building the workspace, as you can iterate on the Dockerfile and test with the resulting Docker image, without needing to recreate the workspace on each iteration.

We may be able to access the build context from the Docker build process using build arguments:

docker build --build-arg repository=https://github.com/gitpod-io --build-arg commit=8c3e9a08d96f61d74fad72d67029eefe61a53dba

Then the project maintainer can develop the Dockerfile with access to the repo name etc. Then the Dockerfile would clone the repo and install dependencies etc. It looks like we have access to /workspace during Docker build time here, so this seems possible from a cursory investigation.

loujaybee commented 2 years ago

Moving to Q3 as the currently proposed solution for this would be dependent upon https://github.com/gitpod-io/gitpod/issues/7185, which in turn is dependent on API changes introduced in https://github.com/gitpod-io/gitpod/issues/7900 and https://github.com/gitpod-io/gitpod/issues/8512. However, if required, some simpler (non-dependent) solutions could be found for immediate pain relief here rather than doing this via a Gitpod CLI / using the Gitpod API's.

loujaybee commented 2 years ago

Note: There is also the related use-case of iterating/feedback loop on updating dotfiles configuration (in addition to .gitpod.yml, which might well be solved with the same solution) but there might be some way to improve the feedback loop for dotfiles iteration in addition, so wanted to add the dotfile use-case / consideration here, also.

raphaeltm commented 2 years ago

Just adding some feedback from another perspective.

We have a custom Docker image for our workspaces where we pre-install various CLIs that we use on a bunch of projects. If I update the Docker image, to get the latest CLIs, but don't want to get rid of the files in my current workspace (which might include docker volumes with data that I still need), don't have a good path forward. To get the latest image, I need to create a new workspace. It would be really useful (in my use case) to be able to rebuild the current workspace, preserving all the files in /workspace. i.e. re-run the prebuild steps, and then remount /workspace.

We've only just started testing GitPod in the last few weeks, so I might be missing a few things about how it works, but I think this ties in well with this issue.

richardfrankza commented 2 years ago

Loving the product overall, but as a newbie to gitpod customising my workspace image - it has been painfully slow to iterate and get it right. I look forward to this feature.

jmls commented 2 years ago

yeah, needing this asap ;)

csweichel commented 2 years ago

run-gp almost provides that - we'd just need to disable the IDE and port forwarding, and add a good means for getting feedback from the tasks.

Built a prototype using run-gp: https://github.com/gitpod-io/run-gp/compare/cw/lint

loujaybee commented 2 years ago

We have begun technical investigations here and created some draft plans around this. However, I'm going to move this from Q3 currently as it's unlikely to complete in this quarter, with most of our efforts going towards the JetBrains integration.

However, we will continue to gather more information, and you comments, ideas and feedback on this issue are still much appreciated and will get incorporated into any final solution(s).

loujaybee commented 1 year ago

Temporarily removing @andreafalzetti as owner, will need to figure out who picks this up again in Q4.

loujaybee commented 1 year ago

Documentation raised w/ announcement: https://github.com/gitpod-io/website/pull/3290

loujaybee commented 1 year ago

Hey folks, a beta version of the gp validate command is now shipped, here's the announcement:

https://www.gitpod.io/changelog/validate-gitpod-yaml-without-commit

You can find some more information of how to use the feature...

Here, in the context of configuring a workspace: https://www.gitpod.io/docs/configure/workspaces And also here for the CLI command: https://www.gitpod.io/docs/references/gitpod-cli#validate

Feedback is tagged up against the gp validate label [1], feel free to add more issues with any other feedback.

I say we consider this issue closed 🙏

GitMensch commented 1 year ago

Sorry for posting to an existing issue, but as this is linked from within the IDE to provide feedback...

First: I definitely like that, no more "create extra branch, temporarily switch default branch to that, create/adjust .gitpod.yml, restart workspace" any more! ... but:

gp validate currently always yields in the workspace being force-closed with a message:

Oh, no! Something went wrong! container workspace completed; containers of a workspace pod are not supposed to do that

after "Building the workspace image..."

(tested both with the default image and with image: gitpod/workspace-c)

akosyakov commented 1 year ago

@GitMensch It sounds like a regression. Could you create a new issue and share with which repo you can reproduce it please? I'm trying to reproduce with https://github.com/akosyakov/parcel-demo but I cannot reproduce with VS Code Browser. I tried with both workspace-full and gitpod/workspace-c.

gitpod-io / gitpod

Epic: Remove need to commit to git (Gitpod "Inner Loop") #7671