Generate all CML repositories from a single repository

iesahin commented 2 years ago

This one is to add a generate.bash script that creates multiple CML example repositories from a single seed repository.

Closes #100

iesahin commented 2 years ago

This is mostly done. It may have duplicate files that are better to be removed but I didn't check them before your approval.

generate.bash

Each $repo directory in github is considered a repository to be placed in https://github.com/iterative/$repo
Each directory in $repo directory, (e.g. github/repo/experiment) is considered a branch in this repository
The script clones iterative/example_cml first, copies the files in github/$repo/$branch/ on top of this as a new branch
It creates a push script to push all generated repositories at once.

I didn't add gitlab and bitbucket yet, need to make sure that this is the right approach first.

A new $hub is (gitlab, bitbucket) is added by adding a directory to example-cml/$hub
Each $hub can have its own URL scheme for git remote.
A new repository to github is added by example-cml/github/$repo
A new branch to a repository is added by example-cml/github/$repo/$branch
If you put a modify-branch.bash script to example-cml/github/$repo/$branch directory, it's run before checking in the files. This file is deleted before commit. So if there are modifications that are better be done in a script rather than copying files, you can put them here.

There was a branch/with/slashes in one of the repositories. The above structure is not compatible with branch names with slashes, so I converted it to a branch-with-dashes.

I didn't use any dvc specific code, only the push script runs dvc push and I didn't test it yet. (The push script actually pushes all the repositories and please make sure the repositories are ok before using it.)

I didn't add support for tags, but if you need them I can add another level of directories for tags.

This doesn't use any modifications to the committer name and commit date, yet. These can be modified per-hub basis.

Let me know what you think and would like to see here. Thanks.

@casperdcl @iterative/cml @shcheklein

iesahin commented 2 years ago

I know you're busy with the release but maybe you have time for a quick review to this one: @casperdcl

iesahin commented 2 years ago

ping! @casperdcl :)

iesahin commented 2 years ago

ping :)

0x2b3bfa0 commented 2 years ago

🙏🏼 Thank you very much, @iesahin! This is an awesome stride in the right direction, and we should review it rather sooner than later.

@casperdcl, like the rest of the team, is a bit busy with releases, but we should track this issue somewhere so it doesn't get lost in the meanwhile.

0x2b3bfa0 commented 2 years ago

Please ping @iterative/cml and @casperdcl as much as you want if you don't hear back from us.

iesahin commented 2 years ago

When do you think you could have more time to review this? @iterative/cml

0x2b3bfa0 commented 2 years ago

I guess that the answer is going to be “after the release” 🙃

shcheklein commented 2 years ago

@iesahin I think it's almost impossible to review. Let's split into some baby steps please. One project at a time. 100-500 line PRs if possible ...?

0x2b3bfa0 commented 2 years ago

example-cml/github/cml-runner-base-case/expeiment/README.md

s/$expe$$iment$/\1r\2/g

casperdcl commented 2 years ago

I think the approach leads to a lot of code (14k lines) which could be avoided...

Many of the branches and even repos are based on each other, I feel applying patches is a better approach?
If patch files are too clunky, maybe we can apply git commits instead. Keep things in a single generator repo and a script will push individual branches to different target repos. Whenever something needs to be updated in the root commit, all branches need to be rebased.

iterative / example-repos-dev

Generate all CML repositories from a single repository #101