triage: create gh action for poetry

[JaeAeich]

Maybe it woulld be better to create a poetry action and get it on gh marketplace so we can reuse, this would give use version on the action as they develop and help us main the action better.

[uniqueg]

Sure, why not :)

[JaeAeich]

@uniqueg I have been think about this and I think I have a better good solution. Why don't we centralise all the CI on an organisation level. We already have couple of CI we should use in all the repos like checking the PR title, precommit checks like EOF fixing etc etc then comes language specific CI for linting for python, node, rust etc etc. Give more or less the deps and workflow of these repos are similar, I propose creating a monorepo for CIs so that we can easily modify CI changes and standardize tooling across our repos. Consider something like this:

.
├── .github
│   ├── workflows
│   │   ├── yaml-linting.yaml
├── common
│   ├── pr-eval
│   ├── pre-commit
│   ├── funnel-setup
├── node
│   ├── linting
│   ├── formatting
│   ├── code-testing
│   ├── deployment-npm
├── rust
│   ├── linting
│   ├── formatting
│   ├── code-testing
│   ├── deployment-crate
├── python
│   ├── poetry
│   │   ├── setup
│   ├── linting
│   ├── formatting
│   ├── code-testing
│   ├── deployment-pypi
├── helm
│   ├── linting
│   ├── formatting
│   ├── deployment-artifacthub
├── docker
│   ├── linting
│   ├── formatting
│   ├── deployment-dockerhub
└── README.md

I have tested this way and it seems to work seemlessly, there are tradeoff definetly but given most of our repos look identical and I think this could be something to look into. Thought???

[uniqueg]

Big +1 on this. It's also what I had in mind when I saw your Cookiecutter GH Actions workflows. The setup looks great!

A few questions/suggestions:

• Shouldn't we have YAML (and markdown) linting workflows as well, in common?
• An OpenAPI checks workflow would probably also be good, also in common, or in a separate folder. Maybe also compliance testing.
• We should also look more closely at automatic version updating (but might be on a per-language basis).
• What tradeoffs do you see?
• More related to the Cookiecuter, but now that I think of it: It would also be good to be able to set up GitHub repo settings automatically (global settings, branch rules, team permissions, secrets, labels etc.), for any new repos.

[JaeAeich]

• Well yes the above was just an initial thought of proj str, not a complete list, we will need yaml-linting as probably the only CI in this proj but all other non code stuff (yaml-linting included) I have been preferring to do via pre-commit config, I find it easier. Hence all those non code checks like md linting, checking accidental push of a big file etc etc are taken care of by pre-commit.yaml for an exhaustive list checkout the pre-commit config in cookicutter.
• Yes thats the idea, I was thinking of creating a composite CI such as the one for setting up poetry or funnel to do these kind of checks. But this would be on need per basis so I assume could be a part of miscellaneous CI workflows.
• Yes, but unfortunately I haven't yet found an ideal workflow that fits our usecase, and the more I look for them I see a huge tradeoff in DX, for example to have an automated release, we need to have commit logs, which means there needs each commit to be passed through a commitlinter but since we squash commit anyways, I am not sure how that would plan out. I'll need to put more thoughts on this one.
• • Pros
  • This would closely couple all our repo, which is good as in the process of debugging will become easy and all of the repo will have similar if not identical DX which would make it easier for newcommers to jump back and forth.
  • Having a good makefile should also make all our projects language agnostic, ie as long as you have oyur env set up, you can run make i and that should install all the deps be it a node, python or a rust project, as this command will under the hood run the specific npm, poetry or cargo command either way.
  • Cons
  • Each project has to run in harmony and there will not be alot of wiggle room.
  • Any updates /chages in the CI will impact all the project that rely on it.
  • Single point of failure, if the CI misbehaves, all the broject will too.
  • Finally this is not a tradeoff but I believe the biggest hurdle if we want to see this through. For this to work all our project need to comply to the standards we will be settign here, ie all python projects need to run on poetry stack, the linting and formatting should be just right. You get teh idea, basically there will be huge changes in each project just to get this through.
• Check this out maybe this is what you are looking for.

[JaeAeich]

Not related to this, but I think one more thing that we should def look into is removing config files and using symlinks for those in our other repo. For example, assuming we wan tto have .markdownlint.yaml, pre-commit-config.yaml, ISSUE-TEMPLATE etc etc, we shouldn;t generate these, since these files are language agnostic, they should be in all the repos regardless and we cant/shouldn't rely on cruft for that in other languages. So maybe centralise these files in cookicutter or someother place and only use symlinks from there?

It will def de-clutter the repo but Im not really sure how much we would wan that, hence not opening another issue if you think this is something to consider as well, we can open an issue for this too.

[uniqueg]

All sound good, so just commenting on the listed CONS:

Each project has to run in harmony and there will not be alot of wiggle room.

Nobody forces us to use the Cookiecutter or any of the CI workflows if there are good reasons not to. So we still got all the wiggle room we need, if necessary.

Any updates /chages in the CI will impact all the project that rely on it.

Awesome! So we can push new/improved policies to all repos at the same time :) Seriously though, if we fail to foresee unwanted consequences of a workflow upgrade on any one repo, it's likely still easier to make changes in that repo, e.g., by pinning the version of an action or overriding the workflow. And if multiple or all repos are affected, we can roll back the change in one central place and fix it (or account for different needs).

Single point of failure, if the CI misbehaves, all the broject will too.

I think in most cases this is more than offset by "single source of truth" - because we need to fix things only in one place.

Finally this is not a tradeoff but I believe the biggest hurdle if we want to see this through. For this to work all our project need to comply to the standards we will be settign here, ie all python projects need to run on poetry stack, the linting and formatting should be just right. You get teh idea, basically there will be huge changes in each project just to get this through.

I disagree that this is only useful if all projects comply. Rather, we will roll this out incrementally, repo by repo. And if we feel that it doesn't make sense for any given repo, we just skip it. We would still reap the benefits for all other repos.

[uniqueg]

Not related to this, but I think one more thing that we should def look into is removing config files and using symlinks for those in our other repo. For example, assuming we wan tto have .markdownlint.yaml, pre-commit-config.yaml, ISSUE-TEMPLATE etc etc, we shouldn;t generate these, since these files are language agnostic, they should be in all the repos regardless and we cant/shouldn't rely on cruft for that in other languages. So maybe centralise these files in cookicutter or someother place and only use symlinks from there?

It will def de-clutter the repo but Im not really sure how much we would wan that, hence not opening another issue if you think this is something to consider as well, we can open an issue for this too.

I am not sure about this. Generally it's better to be explicit than implicit. Also, using symlinks might have unexpected consequences, e.g., the Community Standards may not pick up on symlinks and that may affect our projects' "SEO" ratings. Instead, static files that are the same across all repos, regardless of language, might be better placed in a template repo that could be used in the Cookiecutter.

[JaeAeich]

Rather, we will roll this out incrementally, repo by repo.

Yeah, I just meant that doing that will be alot of work but will def pay in dividends :rocket: .