search

kestra-io / plugin-git

Apache License 2.0
3 stars 4 forks source link

Improve the Git diff output in the logs in the `io.kestra.plugin.git.Sync` task #48

Closed kriko closed 8 months ago

kriko commented 9 months ago

Expected Behavior

When running git.Sync against a repository that houses flows for child namespaces as well, then flow naming should be namespace.child.grandchild.flow.yml, instead of just flow.yml - this would be consistent with git.Push behaviour.

When none of the files have been changed, then they should not be marked as created. Look at the actual behaviour example.

This has briefly been discussed in Slack.

Actual Behaviour

Running a git.Sync task against a repository storing flows of a namespace (and sub-namespaces) when nothing has changes produces the following output:

Start cloning from 'https://gitlab/namespace.git'
+ /_flows/run.yml
+ /_flows/run.yml
+ /_flows/run.yml
~ /_flows/batch.yml
Dry run is disabled, performing following actions (- for deletions, + for creations, ~ for update or no modification):
~ /README.md

The Git repository has a directory /_flows which houses all the flows in the repository, all flows are placed in the root of that folder (no folder nesting).

The repositorys flows folder also contains flows for sub-namespaces, the files are named accordingly:

In reality, none of the files were changed, updated, added or deleted. batch.yml seems to be correctly displayed for "update or no moficication"

Steps To Reproduce

Create a Git repository that has /_flows folder. Create several flows with the same name, but under different child namespaces. Run Git sync against that repository. Repeat the run.

Environment Information

Example flow

No response

anna-geller commented 9 months ago

agree that the dryRun output can be improved as it's confusing that flows are first listed and only then dry run for namespace files is shown:

image

Regarding the request to force the files to be prefixed with namespace, I'm afraid we will not be able to do that for the following reasons:

  1. Difficult for beginners - it may be difficult to understand that naming, especially for nested namespaces; for now we deploy all flows from _flows directory to the same namespace as the task incl. all child namespaces, and if a different namespace is specified it will be overridden
  2. Namespace-level permissions -- the flow located in a namespace engineering cannot deploy to any other namespace than engineering and child namespaces for privacy/security/permission reasons

If you have complex dependencies between flows/scripts and namespaces and you want to use the Git plugin, I'd recommend making a choice to use one of these (but not both): git Push or git Sync. Git Push for kestra as a single source of truth where you edit from the UI and you don't worry about Git, you just let kestra handle it. Or use git Sync with Git as a single source of truth and have one Git branch or repo per parent namespace.

For now, with git Sync we only can sync Git repo to a specific namespace. So if you have e.g. engineering namespace and a similar git setup as here, this will work nicely: https://github.com/anna-geller/kestra-git-sync

anna-geller commented 9 months ago

hope this docs addition can help clarify also a bit https://github.com/kestra-io/plugin-git/commit/ed1c61f7dec611719e5ec9c088d2471f3b6e8241

anna-geller commented 9 months ago

and this issue may help to better manage child namespaces https://github.com/kestra-io/plugin-git/issues/49

It seems that having a separate directory per namespace may help to structure flows in the repo a bit better in Git Sync deployment patterns

anna-geller commented 8 months ago

the issue will be addressed in the new task PushFlows 🎉 you can follow the progress and updates here https://github.com/kestra-io/plugin-git/issues/56