Avnu / OpenAvnu

OpenAvnu - an Avnu sponsored repository for Time Sensitive Network (TSN and AVB) technology
470 stars 289 forks source link

Github Restructuring #800

Open ghindman opened 6 years ago

ghindman commented 6 years ago

All,

We have some pending product requirements that will necessitate getting some OpenAvnu components into common distros. Towards that end I would like to reopen the discussion on restructuring the OpenAvnu repo to make the various components more consumable by external distributions.

We would like to start with breaking the following folders out into top-level repos under the AVnu github project: kmod/igb lib/igb lib/libavtp daemons/gptp

If github admins can create the top-level repos I will commit to moving the code over and untangling any build/test dependencies so that everyone still builds properly.

From a timing/coordination standpoint, if we do a clone then no exiting project dependencies will be broken until everything is fixed up, the only downside would be changes would have to be tracked between the master and the clone until the master is retired. Alternatively we could either freeze those project until completed, or push all of the changes at once, but the latter would require considerable coordination.

Thoughts/concerns?

andrew-elder commented 6 years ago

@ghindman - we have talked about splitting the repo in the past (sometime in the last 12 months), but at the time we didn't feel the extra maintenance, testing and co-ordination of APs (documentation) was worth the final outcome. It sounds like you are in position where you are motivated and think the re-org will provide definite payoffs?

ghindman commented 6 years ago

@andrew-elder yes, near term we have plans to get the above components integrated into the Clear Linux project https://clearlinux.org and other OSVs like Canonical also have a growing interest in TSN/AVB. The repo structure as it is now makes it difficult for distros to consume the redistributable components, and forces all components into a monolithic release/versioning cycle.

We're willing to do the lift on breaking out the components noted here, and we leave it to other component owners (avtp_pipeline, daemons, etc.) to request the same as it makes sense for them.

andrew-elder commented 6 years ago

@ghindman - is the plan to set up submodules in the main repo?

ghindman commented 6 years ago

By main repo you mean the top level AVnu project? We postponed investigating a new structure until we had agreement with the community. If there is consensus then we can come back with a proposal. Libavtp will be the easy one since I don't believe there are yet dependents for it.

andrew-elder commented 6 years ago

@ghindman - so far you and I are the only ones to comment. If we don't get any push back (or other comments) in 48 hours, I'd suggest you go ahead and come back with a proposal.

ghindman commented 6 years ago

Sounds reasonable. It will likely be a few weeks - we have some preliminary tasks in the queue ahead of this.

pinealservo commented 6 years ago

I don't have any objections, for whatever my say is worth. Anything that makes life easier for the people who are actively contributing sounds like a good thing to me.

FYI, I'm no longer employed by Harman and thus I can't speak with any real authority about it, but I'm happy to continue contributing a small amount of my time to code reviews and maintenance tasks until someone takes away my commit bit.

@ghindman, you might find it fruitful to poke the Intel folks who participate in the Avnu TWG to see if you can get some direction about Avnu's intentions for continued maintenance here from that channel.

ghindman commented 6 years ago

Thanks @pinealservo , yes we're in close coordination with the other Intel folks on the TWG.

avinash-palleti commented 6 years ago

Please check below personal repositories which gives an idea how the restructuring looks at the end,

https://github.com/avinash-palleti/OpenAvnu https://github.com/avinash-palleti/igb_avb https://github.com/avinash-palleti/gptp https://github.com/avinash-palleti/libavtp

To put briefly,

Let me know your suggestions/comments on the proposal.

andrew-elder commented 6 years ago

I looks ok to me. I have a few questions 1) Why isn't mrpd in it's own repo? 2) How will travis CI testing be maintained? 3) What do you think about doing this in stages? Move one repo at a time and make sure CI is still working and give folks a week to comment and then proceed to split off another repo. It will take a little longer, but most likely is more manageable. What do you think? 4) Do we lose commit history when we split the repo? I think there is a way to maintain it, but I haven't researched it.

ghindman commented 6 years ago

1) Why isn't mrpd in it's own repo?

2) How will travis CI testing be maintained?

3) What do you think about doing this in stages? Move one repo at a time and make sure CI is still working and give folks a week to comment and then proceed to split off another repo. It will take a little longer, but most likely is more manageable. What do you think?

4)Do we lose commit history when we split the repo? I think there is a way to maintain it, but I haven't researched it.

andrew-elder commented 6 years ago

Ok, so what do you need me to do next to support the libavtp move?

Then we could do gPTP and libigb ...

ghindman commented 6 years ago

I think the only remaining opens are: 1) who can create new repos in the Avnu project root 2) how do you want to establish maintainership/commit rights of the new repos? Are there official maintainers of gPTPd and igb_avb where they are now that we need to enlist? Do you want to give someone on my team (Avinash) commit rights for at least the initial move/copy?

andrew-elder commented 6 years ago

I'm happy to create initial root repos. Just tell me what you need.

@christopher-s-hall is the gPTP maintainer. I'm not sure who maintains igb_avb (probably someone at Intel). I'm open to giving an external person additional rights, but I'd still like like to see changes come in as pull requests rather than someone doing merges in the background.

avinash-palleti commented 6 years ago

Do we lose commit history when we split the repo? I think there is a way to maintain it, but I haven't researched it.

For daemons/gptp and lib/libavtp we can maintain full commit history using git-subtree command as they are going to be individual git repos. For lib/igb and kmod/igb, we cant have commit history in new repo as two folders going into single repo.

I'm happy to create initial root repos. Just tell me what you need.

As Gavin suggested we will start with libavtp, Can you please create a new repo libavtp. I will send two PR one to newly created repo and one to OpenAVnu to remove libavtp.

ghindman commented 6 years ago

perhaps we should not combine lib/igb and kmod/igb if it means losing the git history? Nothing requires us to merge them, it was just a bit cleaner since they should always be built together. Is there a preference?

andrew-elder commented 6 years ago

I have created libavtp. I think the description of the repo could be improved since it is not really a full implementation of avtp.

aguedes commented 6 years ago

Thanks, Andrew. IMO, the description is OK since in the README file of the project we specify what AVTP formats are currently supported.

ethan-stewart-iol commented 6 years ago

Combining lib/igb and kmod/igb while keeping git history should be possible (see http://gbayer.com/development/moving-files-from-one-git-repository-to-another-preserving-history/). I assume git-subtree would be used to move one folder, and then the (long) process in the link could be used to move the other.

andrew-elder commented 6 years ago

I would vote for preserving history - however that turns out to be implemented.

avinash-palleti commented 6 years ago

I have created libavtp.

@andrew-elder Can you please create an empty readme file in repo. I cant fork empty repo and send PR.

andrew-elder commented 6 years ago

readme.md, or just readme.txt, or something else?

avinash-palleti commented 6 years ago

Empty readme.md should be fine.

andrew-elder commented 6 years ago

Done, I think...

avinash-palleti commented 6 years ago

Done. Created two PRs one on this repo and other on libavtp repo. Please check. https://github.com/AVnu/OpenAvnu/pull/813 https://github.com/AVnu/libavtp/pull/1

avinash-palleti commented 6 years ago

If everyone is fine with above first set of patches. I will go ahead and submit another patch to move another folder out.

andrew-elder commented 6 years ago

libavtp doesn't integrate Travis CI yet....

avinash-palleti commented 6 years ago

@andrew-elder I see that you have added libavtp repo to travis (https://travis-ci.org/AVnu/libavtp) My PR already has travis files, that needs to be merged to trigger the build. https://github.com/AVnu/libavtp/pull/1/commits/8d645224342ebd9629c9eb35c38e6891ef6efaa4

andrew-elder commented 6 years ago

@avinash-palleti - I didn't do anything - weird!

aguedes commented 6 years ago

I think Travis adds github repos automatically, that's why libavtp repo shows up in https://travis-ci.org/AVnu already. To trigger automatic builds, we need to enable it in the profile settings. @andrew-elder, do you have access to it?

andrew-elder commented 6 years ago

@aguedes - I'll have more of a look at this over the weekend.

andrew-elder commented 6 years ago

I merged libavtp to master. Want branch/merge policy do we want for libavtp? OpenAvnu uses the convention that all pull requests come to open-avb-next. Do we want something similar in libavtp? I don't much mind what is decided as long as it is written up in the readme. Comments?

avinash-palleti commented 6 years ago

I merged libavtp to master.

Thanks @andrew-elder Can you please merge below patch as well https://github.com/AVnu/OpenAvnu/pull/814

And also can you create another new repo igb_avb with readme.md file.

andrew-elder commented 6 years ago

Done

avinash-palleti commented 6 years ago

@andrew-elder Created two other PRs, https://github.com/AVnu/OpenAvnu/pull/815 https://github.com/AVnu/igb_avb/pull/1

815 needs an update on submodule commit SHA after the second PR is merged.

avinash-palleti commented 6 years ago

To trigger automatic builds, we need to enable it in the profile settings.

@andrew-elder Can you please enable for igb-avb repo also.

avinash-palleti commented 6 years ago

@andrew-elder Can you please create another new repo "gptp" under AVnu and enable travis-ci for the same.

andrew-elder commented 6 years ago

Wow - no rest :) Is gptp the best name? That name isn't taken anywhere else?

avinash-palleti commented 6 years ago

There are some repos in external GitHub with gptp name but not under AVnu, i think that should be fine to have gptp name.

andrew-elder commented 6 years ago

ok

andrew-elder commented 6 years ago

Created gptp. It doesn't seem to show up in Travis yet....

avinash-palleti commented 6 years ago

@andrew-elder daemons/gptp folder has around ~250 commits in commit history. We need to rebase all of them on to new repo on top of initial commit. It is giving merge conflicts once for every ~2 commits. It will be difficult to resolve all conflicts. Can you please provide me admin privileges to that repo so that i can do force push all those ~250 commits and later you can reset permissions. Please let me know if there is any other way i can do this.

andrew-elder commented 6 years ago

Are all the conflicts in the readme.md file?

andrew-elder commented 6 years ago

Is it the gptp repo you need admin privileges for?

andrew-elder commented 6 years ago

@avinash-palleti - have you read https://demisx.github.io/git/rebase/2015/07/02/git-rebase-keep-my-branch-changes.html ?

andrew-elder commented 6 years ago

I guess you still need to do a push force after resolving the conflicts?

jdkoftinoff commented 6 years ago

No, don't do it that way. Follow this: https://help.github.com/articles/splitting-a-subfolder-out-into-a-new-repository/

avinash-palleti commented 6 years ago

No, don't do it that way. Follow this: https://help.github.com/articles/splitting-a-subfolder-out-into-a-new-repository/

I followed same steps. Now Problem is,

Its like deadlock. If i have admin rights on AVnu/gptp repo, i can set origin as upstream repo where i can submit all commit history

andrew-elder commented 6 years ago

@avinash-palleti - you should have push permission to gptp

jdkoftinoff commented 6 years ago

git rebase the forked repo on top of the initial commit.