search

kestra-io / plugin-git

Apache License 2.0
3 stars 4 forks source link

`git.Sync`- Flows arranged in sub directories aren't be parsed #45

Closed Ben8t closed 9 months ago

Ben8t commented 9 months ago

Expected Behavior

Having everything under _flows being parsed. Also the possibility to arrange flows in nested namespace in subfolders

Actual Behaviour

When having the following repository structure:

_flows/
  namespace_a/
    git-sync.yml
    test.yml

With test.yml:

id: test
namespace: namespace_a

tasks:

  - id: hello
    type: io.kestra.core.tasks.log.Log
    message: Kestra team wishes you a great day! 👋

git-sync.yml as:

id: git-sync
namespace: integration
tasks:

  - id: git-sync
    type: io.kestra.plugin.git.Sync
    url: https://github.com/Ben8t/kestra_prod
    branch: test-git-ops
    username: Ben8t
    password: "{{ secret('GITHUB') }}"

triggers:
  - id: every_minute
    type: io.kestra.core.models.triggers.types.Schedule
    cron: "*/1 * * * *"

The git-sync flow raises this error:

image

Having nested directories in _flows doesn't seem to work; it's not parsed + the logs is not very informative.

Steps To Reproduce

No response

Environment Information

Example flow

No response

anna-geller commented 9 months ago

FWIW this is an expected behavior so I'll add documentation to clarify that and close the issue

flows are not Namespace Files, they must follow the structure of being added as not-nested files in the _flows directory. We need to educate users that flows are stored in the database, not as Namespace Files and they need to be treated differently

A large part of the problem comes from the VS Code editor that exposes flows as a directory 😢 we should have never done that. By seeing flows in a directory, users think that flows are arbitrary files rather than orchestration metadata structured in YAML config and sent to the API

anna-geller commented 9 months ago

https://github.com/kestra-io/plugin-git/commit/ed1c61f7dec611719e5ec9c088d2471f3b6e8241

anna-geller commented 9 months ago

It seems that having a separate directory per namespace may help to structure flows in the repo a bit better in Git Sync deployment patterns + better manage child namespaces https://github.com/kestra-io/plugin-git/issues/49