search

AcademySoftwareFoundation / rez

An integrated package configuration, build and deployment system for software
https://rez.readthedocs.io
Apache License 2.0
951 stars 340 forks source link

Pulse of this repository #630

Closed mottosso closed 4 years ago

mottosso commented 5 years ago

Hi all, @nerdvegas in particular!

I just wanted to put a finger on the pulse for a second with regards to future development and pull-requests in particular; I've noticed a number of useful features that were agreed upon but never actually merged, like #355 and #597, some of which are years old, others weeks to months.

This isn't a critique! We all get busy and open source is often a labour of love with little to no accountability.

The issue however with leaving PRs for too long is that users start using their own forks in place of master, like Luma and mackievision, or worse yet limit themselves to a stale version. The benefit of having all PRs be blessed by a single maintainer can be consistency, but comes at a cost. Case in point, because I have so many unmerged feature branches, it prevents me from continuing development until these have been merged, else be forced to submit a "mega-PR" like the one for Python 3 support, #589.

You're lucky enough to have a pool of active contributors here, I would recommend assigning more responsibilities to one or more and take a more hands-off approach to development.

What I expect you'll find is a project that encourages growth, and takes some of the stress off from your end. In my experience, I've found that it's fine to let contributors break a build, even break a release; that's what versioning is for. So long as it's versioned, the end-user can choose to upgrade and roll-back at will.

There's benefit to letting contributors break a release; it gives them ownership and can make one feel part of a project. I've had my projects broken many times, only to be quickly rectified by the original developer, the developer having gained ownership and learnt a valuable lesson on backwards compatibility; with little to no cost to the end-user.

Now on the other hand, if this repository isn't your normal open source project, and is instead coupled with your particular studio and your studio has requirements that limit where the project is able to go - then that is useful information too. In that case, I would recommend going the Red Hat route of splitting the project into two; Red Hat and Fedora, for "coorporate" and "experimental". Where you let one grow unbounded, and merge features that work over a longer period of time into the other.

Another and perhaps more sensible option, if the studio really is an outlier, perhaps Rez should be open source, and the studio would maintain its own fork.

If possible, it would be great to see a 1-2 year plan of where you personally see things going, and what we could do to help.

Food for thought, let me know what you think!

Best, Marcus

KelSolaar commented 5 years ago

That makes me think that it might also be worth considering the Academy Software Foundation (ASWF): https://www.aswf.io/

nerdvegas commented 5 years ago

Hey all,

First, apologies for recent inactivity. Life, sickness etc. Will be getting up to speed on recent email threads pretty soon, and merges should start happening again.

RE a development plan, yes this is something I've been thinking about. I think it does make sense, and it'd be good to do a full writeup outlining the goals, and the features/fixes necessary to get there. I think I'll concentrate on this next. In order to get rez to where (I think) it should be, we'll need a combined effort. Some of the necessary work will be challenging.

RE ASWF, it is something I am considering. I think it'd be good to get the project into a more orderly state before going that route however. But in theory I have no issue with passing it into the ASWF and with changes to governance that would come with that. Ultimately it would help the project become a standard for our industry, and that's been my goal all along (because let's face it, who wants to keep writing their own tech to deal with this crap?!)

Thx all, A

On Tue, May 21, 2019 at 8:18 PM Thomas Mansencal notifications@github.com wrote:

That makes me think that it might also be worth considering the Academy Software Foundation (ASWF): https://www.aswf.io/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/630?email_source=notifications&email_token=AAMOUSTQ7G5DFOROYUXGHTDPWPD6ZA5CNFSM4HOJOBM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV3OLQI#issuecomment-494331329, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMOUSUSTLD73CALD6LZWWDPWPD6ZANCNFSM4HOJOBMQ .

mottosso commented 5 years ago

Thanks for getting back, and again, the problem isn't inactivity, but that inactivity leads to inaction. Inaction due to inactivity can be solved by having someone able to move the project forwards - especially on things you're happy with but just haven't got the time for (which again, is totally fine!).

For example, PRs like #355 could probably have been taken over and implemented by contributors. As another example, submitting this project for review to ASWF could also be done by a contributor; all you have to do is ask. :)

This isn't a power struggle; it's a way to avoid year-long PRs and internal forks from having to take place and be maintained at all. We're all solving the same problems, and if we band together we'll be that much stronger. Like a rez-bundle of sticks.

bpabel commented 5 years ago

Out of curiosity, does anyone besides @nerdvegas have permissions on this repo?

Even things as simple as going through the issues and closing, merging, tagging, and cleaning them up -- there are probably a good number of contributors that would gladly do that work, so at least there's a better idea of exactly what changes need to be made.

nerdvegas commented 5 years ago

I'm the only one that has merge permissions.

Let me look at things in GitHub soon and see what the options are. I would feel better about opening up access if GH can be configured to only allow merges after a certain number of approvals, things like that.

Thx A

On Thu, May 23, 2019 at 5:24 AM Brendan Abel notifications@github.com wrote:

Out of curiosity, does anyone besides @nerdvegas https://github.com/nerdvegas have permissions on this repo?

Even things as simple as going through the issues and closing, merging, tagging, and cleaning them up -- there are probably a good number of contributors that would gladly do that work, so at least there's a better idea of exactly what changes need to be made.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/630?email_source=notifications&email_token=AAMOUSSCA6PW2W3ZHA7UU2DPWWMV3A5CNFSM4HOJOBM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWABSYI#issuecomment-494934369, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMOUSSV4VGZCGJA2J4GNLTPWWMV3ANCNFSM4HOJOBMQ .

lambdaclan commented 5 years ago

Hello Allan

In terms of allowing merges after a certain number of approvals please see the Github documentation about required reviews on how to enable it.

I know for a fact that myself and many other contributors would like to help you manage the repository by doing tasks such as the ones mentioned by Brendan.

We are open for discussion so let's come up with a plan to improve the flow of the repository that will enable us to take rez to the next level.

I would like to see things like

I can come up with many other things but this is a good start.

Will be happy to help to put some of them together given the community and you agree with this direction. Also feedback and additional ideas are welcome!

Thank you.

nerdvegas commented 5 years ago

Thanks for that link Ira, I'll take a look. I'm also going to spend some time looking into GH in general and seeing what features we can take advantage of.

A

On Thu, May 23, 2019 at 10:25 AM Ira notifications@github.com wrote:

Hello Allan

In terms of allowing merges after a certain number of approvals please see the Github documentation about required reviews https://help.github.com/en/articles/enabling-required-reviews-for-pull-requests on how to enable it.

I know for a fact that myself and many other contributors would like to help you manage the repository by doing tasks such as the ones mentioned by Brendan.

We are open for discussion so let's come up with a plan to improve the flow of the repository that will enable us to take rez to the next level.

I would like to see things like

  • multiple maintainers/reviewers to get code into the repository quickly and allow people to work on new tasks
  • issue replies and management (closing, proper tagging such as type etc and assigning)
  • a pinned task list maybe in the form of a roadmap so that we can assign ourselves to tasks and avoid work duplication - maybe each task should include effort required so that if it is a big undertaking contributors can collaborate on the same task (mini teams)
  • well defined procedures and templates for everything that has to do with the repo - merge requests, issue creation, code styling, commit styling, new branch creation, naming, tags etc
  • revamp the wiki to include better contributor guidelines, links to new slack channel etc

I can come up with many other things but this is a good start.

Will be happy to help to put some of them together given the community and Allan agree with this direction. Also feedback and additional ideas are welcome!

Thank you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/630?email_source=notifications&email_token=AAMOUSW3YP7SXHPPS2TUG7DPWXQBBA5CNFSM4HOJOBM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWAXA2Y#issuecomment-495022187, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMOUSWMBIFSAYJIEPWUAYLPWXQBBANCNFSM4HOJOBMQ .

JeanChristopheMorinPerso commented 5 years ago

I agree with what all of you guys said.

multiple maintainers/reviewers to get code into the repository quickly and allow people to work on new tasks

And we will have to define multiple maintainers. Because we can't obviously open the gate to too much people as it just gets really hard to manage if too many people have maintainers right. To be discussed by everyone I guess.

Where do you guys think we should have these conversations? In the forum? Here? Video conference? Somewhere else? By smoke signals? (Yeah, I'm open to literally all options :stuck_out_tongue: ) Because the first thing to determine before starting a discussion is where we should do it and in which format. (at least to me it's the first step).

I'm super happy to see things moving!

jcmorin

lambdaclan commented 5 years ago

Hello Jc,

And we will have to define multiple maintainers. Because we can't obviously open the gate to too much people as it just gets really hard to manage if too many people have maintainers right. To be discussed by everyone I guess.

Agreed.

Where do you guys think we should have these conversations? In the forum? Here? Video conference? Somewhere else? By smoke signals? (Yeah, I'm open to literally all options stuck_out_tongue ) Because the first thing to determine before starting a discussion is where we should do it and in which format. (at least to me it's the first step).

Although smoke signals sound intriguing my vote goes for a video/audio conference. No need for people to show their faces if they don't want to, I respect everyone's privacy.

I think the discussion is just too involved and doing it via text messages or even worse forum posts will be time consuming and confusing with everyone typing messages frantically trying to get their opinion and ideas before someone changes the subject.

The way I see it,

  1. Decide which members of the community should be part of the discussion.
  2. Decide the topics to be discussed, an agenda (time included) is required to avoid getting into other topics not planned for.
  3. Decide on a meeting host to manage topics (focus on the topic on hand) and push ahead as conclusions are reached.
  4. Decide on a minute taker who will be taking notes as the outcome of the discussion will have to become a wiki post to inform the rest of the community.

I am also excited to get the ball rolling, here is hoping.

Ira

bpabel commented 5 years ago

@nerdvegas Yeah, the Github permission system can be a bit janky.

You can add Collaborators to give other people "write" access, which would allow management of Issues and Pull Requests, but also gives them write and push access to the repository.

You can use "Protected Branches" to prevent collaborators from pushing directly to master (or any branch), and you can use the CODEOWNERS file to set certain people (like yourself) as required Approvers on Pull Requests before they can be merged.

mottosso commented 5 years ago

Hi everyone,

Because of the amount of ongoing activity and until Allan finds the time to implement this plan, I'd like to offer to get started doing this in the bleeding-rez repo, in ways that are applicable to this repository and can be continued here once the wheels start turning.

Here are some of the things I'd like to see finalised asap.

These are the ones that apply to just my situation; which ones are you interested in seeing merged? In the chat, @wizofe mentioned wanting to work on new docs, let's set this up with GitHub pages as well.

Future PRs

Once these are put behind us, there are a few more things I've got in mind on my end.

$ rez wheel --install six Qt.py pyblish-base
$ rez choco --install python==3.7
$ rez vcpkg --install physx ilmbase qt==5.12
$ ...

(Note that those issues are still being worked out before being submitted here, primarily awaiting merging of the above PRs)

Next Steps

I'd like to invite you to submit PRs that Allan's approved but hasn't had the time to merge just yet, be it yours or anyone elses that you could benefit from, like the custom package orderer. Some of them involve refactoring. The PR section has got a few gems in it!

Then I'd like to assign you with maintainer roles so we can test out the above theory of continued collaboration, and enable the Kanban-board style of issue triaging to see where that goes.

By this time next week, I expect we'd be able to reduce the PR section by 50%, and have a fully functional version of Rez for Python 3 and Windows on PyPI and from there the sky's the limit!

Let me know your thoughts!

JeanChristopheMorinPerso commented 5 years ago

Hi @mottosso , I'd say we could probably wait a little bit more. I don't see anything urgent that forces us to move to another place yet. It's not like Allan didn't answer over a year or something or disagreed with changing the maintainer model, etc. He answered last weekend (ish) and showed some great openness.

Moving in your repo would also mean that if some PRs are merged there but are never merged here (for whatever the reason is), it might force you to continue your public fork and so might cause some confusions in the community as to which rez to use. This is just a theory yes, but it's a possibility and should be taken into account IMO.

But at the end, we probably all have forks with changes that will probably never made it to the upstream, because they are too specific to our workflows (Like custom repository types for example) or we know it wouldn't benefit the entire community or goes against it.

I would personally stick to basically what @lambdaclan said in https://github.com/nerdvegas/rez/issues/630#issuecomment-495035400. That is, plan things or at least discuss things before moving forward. (Yep, I'm also that boring guy that prefers to well plan things ahead)

JC

bpabel commented 5 years ago

@mottosso

How about moving development to a github organization? We'd have more control over permissions, and if we decide to go the route of creating a centralized package repository for common packages, then the repos could be organized together in one place.

@JeanChristopheMorinPerso

I'd say we could probably wait a little bit more. I don't see anything urgent that forces us to move to another place yet. It's not like Allan didn't answer over a year or something or disagreed with changing the maintainer model, etc. He answered last weekend (ish) and showed some great openness.

To be fair, this problem didn't start a week ago when this issue was opened. The long list of open PR's and Issues didn't appear overnight. I think it's fair to say that there are contributors and maintainers who would contribute more to rez development, but can't (or aren't motivated to) because they don't have permissions on the repo and their contributions aren't being given clear yes or no decisions.

might cause some confusions in the community as to which rez to use.

I think it's fair to say we're discussing moving community development to another repo, so that's kind of the point.

I would personally stick to basically what @lambdaclan said

We don't need to have a convention and meeting notes to fork a repo. And there's no reason we have to be particular about who can contribute. I'd be fine with forking the repo and giving anyone who's contributed in the last year write access if they want it. That's probably less than a dozen people.

mottosso commented 5 years ago

How about moving development to a github organization?

That's a good idea, it would also make the ownership a little less personal (hosted under an individual username) and a little more collaborative (multiple users in a given organisation).

I had a quick look and both rez and rezolve usernames are occupied. :( What's a good alternative organisation name?

Technically unimportant, but good for branding and new-user (and studio!) appeal.

bpabel commented 5 years ago

Out of those, I think I'd prefer rez-dev or getrez

There's also

nerdvegas commented 5 years ago

Hi all,

I will first echo what JC has said - I'm up for opening access and I just want to review GH options for doing that before I commit. I have time tonight to look into this, although I'll probably spend most of my time getting on top of the PRs that people are more interested in (as noted by Marcus earlier), until I hear some more feedback from this thread. But to reiterate - I am in favour of opening permissions on the project, I just want to ensure that the right amount of oversight is still in place when we do that.

The first step I am interested in taking, is opening merge access to contributors, with the caveat that PRs require an approval from me. This is just a first step - if everything goes well then permissions will be opened further. This isn't so much about me keeping control, it's more about keeping the project healthy. I can tell you there have definitely been a number of PRs in the past that initially made changes that would have had negative impact - rez is a big project now and without a lot of familiarity with it, it's hard to know what all the implications of one change are going to be. It's just inherently easier for newer contributors to make mistakes in this regard - I don't mean that as a judgement of anyone's abilities, I think it's just something that's inherently true in software development.

I don't think moving to another repo would be a good move at this stage. It may make sense from the point of view of developers who want to iterate on the project more often; but bear in mind that hundreds of studios (I guestimate) are using rez, and a lot of those are going to value stability over anything else. I fear that a different repo could cause confusion and a split in development that broadly splits between new/smaller studios and individuals, and larger established studios. However, another reason is because at this stage I'm still thinking that a move to an ASWF project is possibly on the cards too, and I don't think shifting the project twice would be helpful to anybody.

Happy to hear people's thoughts. A

Ps:

To touch on some specific points/PRs that have been raised here:

Py3 support: I would love to merge this PR, but if it includes all previous PRs, it's almost certainly going to break multiple things at once, that the tests don't happen to cover. This will probably result in a new rez version that some studios who use rez extensively, simply won't be able to use. I agree with the approach and am glad to see futurize used, as that was my thought also; but it would be better to merge this as its own feature and deal with the conflict on other PRs one-by-one, especially as I don't see that these conflicts would be too complicated to deal with.

CustomPackageOrderer: Unfortunately as elrond mentions, this PR is not compatible with master. The reason as I recall has to do with its support for applying different orderers on a per-family basis. Master does this already, but in a way that's more optimal. However, I completely agree with what this PR brings to the table (especially the ability to configure orderers in the config, which has been wanting for ages). I would like to work on a new PR incorporating this feature in a way that's compatible with the existing PackageOrderer classes (or would welcome someone else doing the same). IIRC elrond and I happened to work on package orderers around the same time, and this conflict in approaches has been there from the start.

Reversed version range: I think I will merge this one shortly. I had reservations, because I wanted to be confident that the change would not impact on resolve times. However, the only way to be confident of that is to have an extensive rez benchmarking program (something I've wanted to work on for some time now). What I intend for this is to take a copy Method's existing packages (payloads removed and names anonymized); then, produce a list of resolves that form the benchmark, and provide tooling for running these resolves and plotting out the results. You have to be careful making changes that can affect resolve times (as I have found in the past, sometimes painfully) - a change that looks like it's sped up resolves can sometimes also make slow resolves even slower, for example.

PyPi/wheel/rez-pip: I'm fine with merging these PRs, I just need to check what the latest state of things is first. I've voluntarily taken a backseat on this aspect of rez development, as I'm more interested in features that improve rez as a package manager in its own right, as opposed to bridging it with other managers. I'll spend some time tonight catching up on this. If GH allows it, it could make sense to have a branch opened up to merges from all contributors, and only clamp down on permissions when merging to master.

Windows support: Similar issue to above, I don't develop on Windows and I'm happy to accept PRs that improve the situation. A dedicated branch could also be good for this case.

Wiki: I don't follow what the issue is here. All the docs are in the wiki/ folder of the project, and there's a bash script there that pushes all updates to github wiki. Are you referring to updates to wiki/ in outstanding PRs, that haven't been pushed to github wiki yet?

Wrt performance improvements. That's quite a big conversation. I've done a lot of work on the solver (where the biggest gains are to be made) but there are still some improvements I have in mind. We should talk about this. One thing to bear in mind is that time spent loading package definitions off disk isn't very relevant - a production rez deployment should be using memcached, and in this setup, disk access is already heavily minimised.

On Sat, Jun 1, 2019 at 11:54 PM Jean-Christophe Morin < notifications@github.com> wrote:

Hi @mottosso https://github.com/mottosso , I'd say we could probably wait a little bit more. I don't see anything urgent that forces us to move to another place yet. It's not like Allan didn't answer over a year or something or disagreed with changing the maintainer model, etc. He answered last weekend (ish) and showed some great openness.

Moving in your repo would also mean that if some PRs are merged there but are never merged here (for whatever the reason is), it might force you to continue your public fork and so might cause some confusions in the community as to which rez to use. This is just a theory yes, but it's a possibility and should be taken into account IMO.

But at the end, we probably all have forks with changes that will probably never made it to the upstream, because they are too specific to our workflows (Like custom repository types for example) or we know it wouldn't benefit the entire community or goes against it.

I would personally stick to basically what @lambdaclan https://github.com/lambdaclan said in #630 (comment) https://github.com/nerdvegas/rez/issues/630#issuecomment-495035400. That is, plan things or at least discuss things before moving forward. (Yep, I'm also that boring guy that prefers to well plan things ahead)

JC

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/630?email_source=notifications&email_token=AAMOUSUB36OBBSO5XGDN3CTPYJ5PPA5CNFSM4HOJOBM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWXBDHA#issuecomment-497947036, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMOUSRC3PINBADNXYLX3N3PYJ5PPANCNFSM4HOJOBMQ .

KelSolaar commented 5 years ago

I don't think moving to another repo would be a good move at this stage.

Agreed, it would only cause confusion, the development might seem slow for some and could generate frustration for others (which I understand) but the reality is that Rez is certainly not stalled to the point where I think it requires shaking the whole tree.

Cheers,

Thomas

bpabel commented 5 years ago

@nerdvegas How about moving the repo to an organization? It gives us more options for permissions, including visibility into who has permissions, and we'd get a bunch of cool features like teams/discussions/projects/etc. If you need help setting up the Github permissions, let me know, they're really not that difficult.

I can tell you there have definitely been a number of PRs in the past that initially made changes that would have had negative impact - rez is a big project now and without a lot of familiarity with it, it's hard to know what all the implications of one change are going to be. It's just inherently easier for newer contributors to make mistakes in this regard - I don't mean that as a judgement of anyone's abilities, I think it's just something that's inherently true in software development.

I agree, That's even more of a reason that you want more people invested in doing code reviews and approving PR's. No one is going to submit perfect PR's the first time every time (including you), but the solution to that isn't to restrict merge approvals and code reviews to a single person.

I don't think moving to another repo would be a good move at this stage. It may make sense from the point of view of developers who want to iterate on the project more often;

Honestly, it's not really about how fast the project is moving. It's more about just having a clear direction and motivation to do more work on the repo. There's very little point for contributors to discuss issues and do code reviews if they have no authority to approve any changes and their PR's never get any feedback from the one person with merge access.

It's okay to say no to a PR. It's far worse to let PR's that you know you'll never merge stay open and let people keep working with that direction in mind.

nerdvegas commented 5 years ago

Below.

On Sun, Jun 2, 2019 at 3:41 PM Brendan Abel notifications@github.com wrote:

@nerdvegas https://github.com/nerdvegas How about moving the repo to an organization? It gives us more options for permissions, including visibility into who has permissions, and we'd get a bunch of cool features like teams/discussions/projects/etc. If you need help setting up the Github permissions, let me know, they're really not that difficult.

I'd rather enable more collaboration on the existing project first, and go from there.

I can tell you there have definitely been a number of PRs in the past that initially made changes that would have had negative impact - rez is a big project now and without a lot of familiarity with it, it's hard to know what all the implications of one change are going to be. It's just inherently easier for newer contributors to make mistakes in this regard - I don't mean that as a judgement of anyone's abilities, I think it's just something that's inherently true in software development.

I agree, That's even more of a reason that you want more people invested in doing code reviews and approving PR's. No one is going to submit perfect PR's the first time every time (including you), but the solution to that isn't to restrict merge approvals and code reviews to a single person.

I definitely do want more people involved in code review and approving PRs, and I think you're right that this will mean people are more invested in the work.

I don't think moving to another repo would be a good move at this stage. It may make sense from the point of view of developers who want to iterate on the project more often;

Honestly, it's not really about how fast the project is moving. It's more about just having a clear direction and motivation to do more work on the repo. There's very little point for contributors to discuss issues and do code reviews if they have no authority to approve any changes and their PR's never get any feedback from the one person with merge access.

It's okay to say no to a PR. It's far worse to let PR's that you know you'll never merge stay open and let people keep working with that direction in mind.

You can look at the history of the 258 merged PRs in this project to see that there's been a lot of feedback from me. And wrt the PRs that are outstanding, I don't think there are any that I intend not to merge - typically, older PRs have fallen into an "it's complicated" status for one reason or another, and the odd few have ended up as dead ends, as the project have developed over time. I have on occasion declined PRs.

What I want to do from here on in is the following:

To that end, I propose:

I'd like to see how the team actually does collaborate for a while, before we determine what the next steps are. Bear in mind that this is an 8 year old project, that's getting real use, and that studios regard as pretty stable (and that's fairing better than a lot of VFX OSS projects!). Any large and sudden changes (including code- and governance- wise) may put that at risk. Of course, it's a free world, and anyone is free to fork the project, but it'd be a shame to see it potentially diverge.

A

—

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/630?email_source=notifications&email_token=AAMOUSQ5OHG66LLQFAV444DPYNMQ3A5CNFSM4HOJOBM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWXOKRQ#issuecomment-498001222, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMOUSR5JZDNLWDEIRV3NLLPYNMQ3ANCNFSM4HOJOBMQ .

mottosso commented 5 years ago

Making some good progress here! Overall, it sounds like we're in agreement about one thing:

What remains is finding a suitable compromise.


Breaking Changes

Py3 Support ... This will probably result in a new rez version that some studios who use rez extensively, simply won't be able to use

I think we can all agree this is unwanted. Zooming out a bit, this has to do with making a potentially breaking change and how to deal with it.

From where I'm standing, I see two contradicting audiences to Rez.

  1. Extensive users want stability
  2. New users and developers want progress

Maybe there's a way to appeal to both. For example, we could introduce beta releases.

In practice, we could do something along the lines of:

That way, extensive users seeking stability use a stable version, and let the rest of us get on with development and release (beta) versions early and often.


The all-knowing contributor

rez is a big project now and without a lot of familiarity with it, it's hard to know what all the implications of one change are going to be.

This is something I've seen echoed a few times both on the mailing list and chat, and it's a problem.

So is there nothing we can do? I think there is. Tests only really have to cover expected behaviour, which is a much smaller subset of things to test. If everything expected is covered, then tests wouldn't catch all breakage (e.g. , but it would catch breakage of what is expected.

If we can manage that, then suddenly:

  1. The pace of PRs can increase, as we wouldn't have to worry about breakage or take the time to fully grasp the contents of the PR and implications of the PR into the greater whole.
  2. Contributors would be encouraged to try and implement something that both adds value and passes the tests. If it does, we all win.

Can it be done? Not as a one-off, I think. Expected behaviour changes and grows within any project, and the way to counter it is to write tests for (1) the behaviour you intend alongside any change or feature and (2) for what should have been, once a problem occurs.

This is something we could start doing right now. With beta-releases in place, swapping between stable and beta is made easy, which in turn make testing easy.

Wrt performance improvements. That's quite a big conversation.

Again, this comes back to having to understand everything about the system before making contributions. My guess is you probably didn't look at the issue, because if you did you'd see that not only does a rez env take 200 seconds, it also pinpoints exactly what is it that's taking so long, down to the individual call and their total contribution.

Right this second, we could pick one each, solve it, and be back in time for dinner.


Frontseat Driver

I've voluntarily taken a backseat .. I don't develop on Windows ..

Just a minor note on this as I've seen this echoed before; taking a backseat is one thing, getting out of the car and locking the door is another. :) If you really had a backseat, those would be merged by now. I invite you to try and actually take a backseat. If it backfires, you can always revert commits and give whomever is responsible the business.


Fork vs. Branch

Windows .. A dedicated branch could also be good for this case.

Could you elaborate on this? Do you mean a separate branch with its own release cycle and userbase? Like a fork? This seems counter productive. The PRs are ready now, and fit perfectly with Linux and Mac.


Moving House

I don't think moving to another repo would be a good move at this stage ... hundreds of studios are using rez

I hear what you and Thomas are saying, but this is a solvable problem.

  1. Whenever a project moves within GitHub, GitHub forwards any requests to a former location to the new one. This includes those coming from a browser and made via git clone/pull/push. The only noticeable change to the end-user is the address bar automatically changing in their browsers on visit.
  2. Alternatively, keep the old project here, and insert a "Project has moved to X!" banner.


ASWF

However, another reason is because at this stage I'm still thinking that a move to an ASWF project is possibly on the cards too

Surely, ASWF doesn't require that you move the repository? I had a quick look, and they do have an org with a few projects, some of which are hosted both there and elsewhere.

Frankly, the more I read about ASWF the less I think it is in our interest to join; they seem to enforce a great deal of stability (read: Process) into development which would bring us right back here again. It makes sense for software embedded in and shared amongst DCC vendors, like Alembic and VDB. But there is only ever one version of Rez, and that's the one right here. We should use that to our advantage.


Wiki

I don't follow what the issue is here.

Sorry, should have been more clear.

https://github.com/nerdvegas/rez/issues/600#issuecomment-494142265


Having said that

Just to cover all grounds, there's one other extreme I'd like to touch on.

It's possible Rez has over-matured. That what users has done with it over the past few years prevents or discourages change. It may even be in the interest of users to not see change, only maintenance. This isn't your average text editing app whereby you can easily swap it out for some alternative; whomever has committed to Rez has gone all-in and staked all of a companies software-assets on its continued survival.

If so, we need to set course for a future where this is less likely to happen again (e.g. #634) and either (a) increment the major version or (b) consider starting anew.

mottosso commented 5 years ago

You can look at the history of the 258 merged PRs in this project to see that there's been a lot of feedback from me.

Wanted to underline this as I know first-hand how hard it can be to hear you aren't doing enough when you're doing everything you can (noticing you wrote this at 3 AM on a Sunday). The issue isn't a lack of contribution, but a lack of parallelism. Nobody can say you haven't done enough or should do more. This is indeed a lot more active and used than a lot of open source projects and I think we all agree you're doing a marvellous job in keeping the project together and on track all this time. 🥇

nerdvegas commented 5 years ago

Just keeping it short and in point form as I'd like to catch up on all the email tonight..!

On Sun, Jun 2, 2019 at 8:01 PM Marcus Ottosson notifications@github.com wrote:

Making some good progress here! Overall, it sounds like we're in agreement about one thing:

  • We'd like to increase the development pace, but acknowledge that stability and project direction is important

What remains is finding a suitable compromise.

Breaking Changes

Py3 Support ... This will probably result in a new rez version that some studios who use rez extensively, simply won't be able to use

I think we can all agree this is unwanted. Zooming out a bit, this has to do with making a potentially breaking change and how to deal with it.

From where I'm standing, I see two contradicting audiences to Rez.

  1. Extensive users want stability
  2. New users and developers want progress

Maybe there's a way to appeal to both. For example, we could introduce beta releases.

  • rez-2.30.0 Latest stable release
  • rez-2.31.0b Latest beta release

In practice, we could do something along the lines of:

  • The master branch is made protected, and contains an always-stable, latest-released version of Rez, e.g. rez-2.31.0. No pushing here, only pull-requests, and merging is done rarely by one or few people.
  • A beta branch is writable by contributors and is considered beta-stability. Meaning no WIP features or partial commits, but breakage is permitted and to be expected from anyone using it. Releases made here are tagged beta, e.g. rez-2.31.0b

That way, extensive users seeking stability use a stable version, and let the rest of us get on with development and release (beta) versions early and often.

The all-knowing contributor

rez is a big project now and without a lot of familiarity with it, it's hard to know what all the implications of one change are going to be.

This is something I've seen echoed a few times both on the mailing list and chat, and it's a problem.

  • At one extreme, no change can be made without fully understanding everything. That's obviously bad and would prohibit contributions from anyone (including yourself).
  • At the other extreme any breaking change is picked up by tests. That would of course be great, but even tests cannot encompass everything.

So is there nothing we can do? I think there is. Tests only really have to cover expected behaviour, which is a much smaller subset of things to test. If everything expected is covered, then tests wouldn't catch all breakage (e.g. , but it would catch breakage of what is expected.

If we can manage that, then suddenly:

  1. The pace of PRs can increase, as we wouldn't have to worry about breakage or take the time to fully grasp the contents of the PR and implications of the PR into the greater whole.
  2. Contributors would be encouraged to try and implement something that both adds value and passes the tests. If it does, we all win.

Can it be done? Not as a one-off, I think. Expected behaviour changes and grows within any project, and the way to counter it is to write tests for (1) the behaviour you intend alongside any change or feature and (2) for what should have been, once a problem occurs.

This is something we could start doing right now. With beta-releases in place, swapping between stable and beta is made easy, which in turn make testing easy.

Wrt performance improvements. That's quite a big conversation.

Again, this comes back to having to understand everything about the system before making contributions. My guess is you probably didn't look at the issue, because if you did you'd see that not only does a rez env take 200 seconds, it also pinpoints exactly what is it that's taking so long, down to the individual call and their total contribution.

Right this second, we could pick one each, solve it, and be back in time for dinner.

Frontseat Driver

I've voluntarily taken a backseat .. I don't develop on Windows ..

Just a minor note on this as I've seen this echoed before; taking a backseat is one thing, getting out of the car and locking the door is another. :) If you really had a backseat, those would be merged by now. I invite you to try and actually take a backseat. If it backfires, you can always revert commits and give whomever is responsible the business.

Fork vs. Branch

Windows .. A dedicated branch could also be good for this case.

Could you elaborate on this? Do you mean a separate branch with its own release cycle and userbase? Like a fork? This seems counter productive. The PRs are ready now, and fit perfectly with Linux and Mac.

Moving House

I don't think moving to another repo would be a good move at this stage ... hundreds of studios are using rez

I hear what you and Thomas are saying, but this is a solvable problem.

  1. Whenever a project moves within GitHub, GitHub forwards any requests to a former location to the new one. This includes those coming from a browser and made via git clone/pull/push. The only noticeable change to the end-user is the address bar automatically changing in their browsers on visit.
  2. Alternatively, keep the old project here, and insert a "Project has moved to X!" banner.

ASWF

However, another reason is because at this stage I'm still thinking that a move to an ASWF project is possibly on the cards too

Surely, ASWF doesn't require that you move the repository? I had a quick look, and they do have an org with a few projects, some of which are hosted both there and elsewhere https://github.com/imageworks/OpenCue.

Frankly, the more I read about ASWF the less I think it is in our interest to join; they seem to enforce a great deal of stability (read: Process) into development which would bring us right back here again. It makes sense for software embedded in and shared amongst DCC vendors, like Alembic and VDB. But there is only ever one version of Rez, and that's the one right here. We should use that to our advantage.

Wiki

I don't follow what the issue is here.

Sorry, should have been more clear.

600 (comment)

https://github.com/nerdvegas/rez/issues/600#issuecomment-494142265

Having said that

Just to cover all grounds, there's one other extreme I'd like to touch on.

It's possible Rez has over-matured. That what users has done with it over the past few years prevents or discourages change. It may even be in the interest of users to not see change, only maintenance. This isn't your average text editing app whereby you can easily swap it out for some alternative; whomever has committed to Rez has gone all-in and staked all of a companies software-assets on its continued survival.

If so, we need to set course for a future where this is less likely to happen again (e.g. #634 https://github.com/nerdvegas/rez/issues/634) and either (a) increment the major version or (b) consider starting anew.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/630?email_source=notifications&email_token=AAMOUSQ2USZTFP5UFCC5N53PYOK57A5CNFSM4HOJOBM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWXSD6Q#issuecomment-498016762, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMOUSUYW4ISKJWPSYWSQV3PYOK57ANCNFSM4HOJOBMQ .

JeanChristopheMorinPerso commented 5 years ago

Hey guys, so here are my thoughts on the last couple of messages (and hope my answers will be clear and not too unordered. It's getting hard to comment with these quite big comments).

I agree with almost everything that @nerdvegas said in https://github.com/nerdvegas/rez/issues/630#issuecomment-497997916. Especially with:

The first step I am interested in taking, is opening merge access to contributors, with the caveat that PRs require an approval from me. This is just a first step - if everything goes well then permissions will be opened further. This isn't so much about me keeping control, it's more about keeping the project healthy. I can tell you there have definitely been a number of PRs in the past that initially made changes that would have had negative impact - rez is a big project now and without a lot of familiarity with it, it's hard to know what all the implications of one change are going to be. It's just inherently easier for newer contributors to make mistakes in this regard - I don't mean that as a judgement of anyone's abilities, I think it's just something that's inherently true in software development.

Stability is extremely important for open source projects. I would even go one step further and say that the general line of thought of a project is also really important and needs to be kept as much as possible. In other words, if we let everyone merge what they want, the project will just become a big pile of features that do not work well together and that might not have a common vision.

Talking about stability, if rez becomes too unstable, we won't be able to do any progress. A really good recent example in the open source is pipenv. Pipenv didn't see a release in over a year. One of the reason is it's stability. The maintainers had to deal with a lot of instabilities (in pipenv itself and in their dependencies (setuptools, pip, and the other 80 dependencies)). So they had to stop the development of new features and focus on stability. (Note that this is a brief overview and it won't represent the entire/real story of why they didn't release in over a year as the real answer is way too long to write in what will be a way too long post of my own in here)

From Allan:

I don't think moving to another repo would be a good move at this stage. It may make sense from the point of view of developers who want to iterate on the project more often; but bear in mind that hundreds of studios (I guestimate) are using rez, and a lot of those are going to value stability over anything else. I fear that a different repo could cause confusion and a split in development that broadly splits between new/smaller studios and individuals, and larger established studios.

I can't agree more (as I've already stated earlier).

From Allan:

What I want to do from here on in is the following:

  • Open up more collaboration in a controlled manner. This means making a change to workflow / permissions, assessing how that goes (including feedback from contributors), and iterating;
  • Work on a roadmap detailing where I see the project going, and creating tickets related to this. There are some major features that need to be worked on to turn rez into a fully fledged package manager, and it'd be great to see a focused effort on this. In order to really drive adoption in the industry, I think people need to be able to easily install entire dependency chains of packages from a public repository using rez. We aren't there yet.
  • Dedicate more of my time to moving the project forward. What I would like to do is spend less time on development, and more time giving feedback on PRs and getting those merged.

To that end, I propose:

  • Adding as collaborator anyone who has done rez development thus far and who expresses interest in becoming a collaborator;
  • Restricting merges to master, for the time being.

That too I agree. And I re-iterate that I'm willing to work on the new features like having a shared repository, localization, DB repos, refactoring, etc. But I'm also looking at rewriting the test suites too. Or working on the installer.

From @mottosso :

Making some good progress here! Overall, it sounds like we're in agreement about one thing:

  • We'd like to increase the development pace, but acknowledge that stability and project direction is important

What remains is finding a suitable compromise.

@mottosso And here I agree too :) ! I think the compromises is what Allan is trying to do right now. And to me is proposition is a good first step. Note the word first. I don't think we would be able to come up with something that is perfect the first time.

From @mottosso :

From where I'm standing, I see two contradicting audiences to Rez.

  • Extensive users want stability
  • New users and developers want progress

From what I see, extensive users want stability yes, but they also want progress (and quite a lot). New users will probably want stability first and then progress. Because if you start using rez and it's unstable, it's highly possible you will ditch it away. Though your 2 points also represents the audience too.

As for the beta proposition you are making @mottosso, I think it makes sense, but only if we had 100% (or almost) test coverage (unit and integration and behavior). Though we can still have a beta branch that studios can try.

Also from @mottosso :

The all-knowing contributor rez is a big project now and without a lot of familiarity with it, it's hard to know what all the implications of one change are going to be.

This is something I've seen echoed a few times both on the mailing list and chat, and it's a problem.

This is why projects have maintainers. Maintainers exists for this really particular reason. And it's what differentiate them from contributors. It doesn't mean one maintainer needs to know the whole project on the tip of his fingers. Multiple maintainers should share their knowledge on different parts (or common parts) of the project, and summing all this knowledge gives a full understanding of the project spread across multiple individuals. Contributors then comes in to help developing new cool features, fixing bugs, adding docs (which are all things a maintainers can do).

At one extreme, no change can be made without fully understanding everything. That's obviously bad and would prohibit contributions from anyone (including yourself).

As I just said, this could be solved by having more maintainers that shares knowledge. But I don't agree that this problem would prohibit someone from contributing. It's normal that not all contributions will have the same level of difficulty. Normally the core parts of a project (in a package manager we have solvers) will be well protected and won't be touched by a lot of people, due to their complexity and implications. This is fortunately a solvable problem. Documentation of the internals is normally a good first step into solving this problem. A good contribution guide can also help quite a lot. A complete test suite is also fundamental as you say below.

At the other extreme any breaking change is picked up by tests. That would of course be great, but even tests cannot encompass everything. So is there nothing we can do? I think there is. Tests only really have to cover expected behaviour, which is a much smaller subset of things to test. If everything expected is covered, then tests wouldn't catch all breakage (e.g. , but it would catch breakage of what is expected.

If we can manage that, then suddenly:

  • The pace of PRs can increase, as we wouldn't have to worry about breakage or take the time to fully grasp the contents of the PR and implications of the PR into the greater whole.
  • Contributors would be encouraged to try and implement something that both adds value and passes the tests. If it does, we all win.

Yes tests will help a lot into removing fears of breakage, but someone or multiple people still have to sit and think about the changes and how well they will fit with any future work or how well it fits with the general philosophy of the project. So the "take the time to fully grasp the contents of the PR and implications of the PR into the greater whole" part cannot be removed of the PR process.

It's possible Rez has over-matured. That what users has done with it over the past few years prevents or discourages change. It may even be in the interest of users to not see change, only maintenance. This isn't your average text editing app whereby you can easily swap it out for some alternative; whomever has committed to Rez has gone all-in and staked all of a companies software-assets on its continued survival.

If so, we need to set course for a future where this is less likely to happen again (e.g. #634) and either (a) increment the major version or (b) consider starting anew.

This is effectively something that came up a couple of times, from time to time. I already showed my interest in helping getting a complete rewrite, and so for @lambdaclan. Though as Allan said, that would mean no new features for a quite long period of time, and probably find some money to pay the people that would work on this.

As for issue #634, I think we need to do quite a lot of cleaning before really starting to expose a specific set of the API as being public. Without going into a full rewrite, we can rewrite/refactor parts of rez while developing new features, and while doing this, determine what is to be exposed and how. At least that's how I see things. To be discusses in the issue itself I guess.

All in all, I'm happy to see these conversations going :)

mottosso commented 5 years ago

Starting again. This takes far longer than you'd think..

that would mean no new features for a quite long period of time

To clarify, I'm not suggesting we throw anything away.

What I meant was (a) increment the major version, and thus allow for breakage and new features or (b) fork Rez, call it Tez and more forwards. Either way, we maintain momentum without sacrificing stability. Best of both worlds!

But again, this only applies if Rez really has over-matured, which I don't think it has.

bpabel commented 5 years ago

Stability is extremely important for open source projects... if we let everyone merge what they want, the project will just become a big pile of features that do not work well together and that might not have a common vision.

No one wants rez to be unstable. Thousands of open source projects manage to function with multiple maintainers and their projects don't fall apart. In fact, I'd venture to say that nearly every successful open source project has multiple people responsible for maintaining them.

Also, if the flipside to "stability" is that a significant portion of your users have to work out of their own forks, in production, that's not healthy for the project either.

from @mottosso @JeanChristopheMorinPerso

A beta branch is writable by contributors and is considered beta-stability.

I think we really need to commit to deciding what changes we want, and making sure they get pushed to master. If a change isn't good enough to go to master, it really shouldn't be going into beta either. I just don't want to get into a situation where we're deferring decisions and instead just merging things to beta until it diverges so much from master that it can't really ever be merged.

But it brings up a good point, which is that adding maintainers without also granting merge access to master doesn't really solve our problem. It might be more convenient to work out of the same repo instead of trying to manage a half dozen different remotes, but functionally, it doesn't change anything about PR's. Unless something else changes, we're going to be in the exact same boat a week from now.

@nerdvegas I understand the desire to introduce these workflow changes slowly instead of all at once. That makes sense. But I don't want to have this same discussion every other week. These workflow changes are only going to work if you're the one championing them and moving them forward.

lambdaclan commented 5 years ago

Hello everyone,

compulsory this is my opinion statement, you are free to agree or disagree with it, not trying to impose anything on anyone

I am happy to see the discussion picking up stream. I am sure at end of the day everybody wants the same outcome, keep improving rez for all its users (stable and bleeding edge), expand its userbase and ultimately attract more contributors and increase the potential pool of maintainers.

Overall I am in line with JC as commented here but I would like to offer a broader view of looking at things to make you all consider a few other important points.

I see a lot of people requesting more maintainers (which I am also in favour of if done appropriately), putting themselves down for it but let us not forget that a maintainer is not only responsible for merging pull requests or putting tags on issues. Also there should be more emphasis about who becomes a maintainer and why, becoming a maintainer should take effort and not happen just because of dire repo situation. Do they have the time and skills necessary to do so? Keep in mind programming skill is not the main skill required albeit an important one. Communication is key.

I would like to point you all to a very nice article about managing open source projects written by someone (developer of swaywm a linux window manager and sourcehut an alternative to Github) who through successful management of a FOSS projects has managed to gain enough support to do it full time so I am assuming he is doing something right. I highly recommend you give it a read.

There are far bigger projects than rez with thousands of open issues and countless PRs that manage perfectly fine to maintain a healthy release schedule adding stability and new features for all users without multiple forks syncing with each other. It is all done on a single central repository.

How is that possible?

Call me old school but as JC put it

Yep, I'm also that boring guy that prefers to well plan things ahead

There needs to be a plan, there needs to be a roadmap, there needs to be a scope, there needs to be order in the repository and the community as a whole.

For example

Problem:

Allan

Windows support: Similar issue to above, I don't develop on Windows and I'm happy to accept PRs that improve the situation. A dedicated branch could also be good for this case.

Marcus

Just a minor note on this as I've seen this echoed before; taking a backseat is one thing, getting out of the car and locking the door is another. :) If you really had a backseat, those would be merged by now. I invite you to try and actually take a backseat. If it backfires, you can always revert commits and give whomever is responsible the business.

Answer: Is Windows support in scope? Is it in the roadmap? Is it a frequently requested feature that maintainers should consider adding?

Without a roadmap, plan or a scope is hard to come up to a conclusion, anyone is free to walk in as a contributor and request a feature to be added to master, it is tested, maintains backwards compatibility and the code is good. What do we do? Accept it? Reject it? What do we base that decision on? If we accept anything regardless of how good it is there will be mayhem. The project needs to stick to its plan decided by the maintainers anything else goes into the consideration list.

Now if you want a feature that is not in scope or in the roadmap then fork is the way to go. There needs to be some limitations and constraints otherwise rez will diverge from its main purpose. Having a roadmap will make it immediately apparent to prospective contributors what can be potentially merged to master before working on it.

This requires an open mind from both the maintainers to listen to what the person requesting/ offering a feature PR has to say and the contributor to accept and respect the maintainers decision to comply or not comply.

An example from pyenv:

Issue: Windows Support Solution: [Wiki] --> If you're on Windows, consider using @kirankotari's pyenv-win fork. (pyenv does not work on windows outside the Windows Subsystem for Linux)

Stability is extremely important for open source projects... if we let everyone merge what they want, the project will just become a big pile of features that do not work well together and that might not have a common vision.

There is no vision right now apart the one in our heads, everything is go which is why we see this happening.

What I am trying to say is that handling an open source project is much more than coding, there needs to be clear communication between the community members, an understanding of the overall goals of the project and unison amongst the maintainers in order to "maintain" order and keep the project moving forwards.

From Brendan

I think we really need to commit to deciding what changes we want, and making sure they get pushed to master. If a change isn't good enough to go to master, it really shouldn't be going into beta either. I just don't want to get into a situation where we're deferring decisions and instead just merging things to beta until it diverges so much from master that it can't really ever be merged.

Agreed, most importantly:

I think we really need to commit to deciding what changes we want

Plan, roadmap, scope please.

From JC

This is why projects have maintainers. Maintainers exists for this really particular reason. And it's what differentiate them from contributors. It doesn't mean one maintainer needs to know the whole project on the tip of his fingers. Multiple maintainers should share their knowledge on different parts (or common parts) of the project, and summing all this knowledge gives a full understanding of the project spread across multiple individuals. Contributors then comes in to help developing new cool features, fixing bugs, adding docs (which are all things a maintainers can do).

Agreed 100% and very close to what the linked article above boils to.

An open source project, or in fact any software project cannot please everyone or meet every studios needs. It is virtually impossible. There bounds to be a specific requirement that will be so closely related to an internal workflow which does not make sense to be part of rez. rez has allowed multiple studios to achieve amazing things in terms of automation and that is great but if at one point a studio needs to diverge then is perfectly fine, this is how new projects are being born and many times later merged again or continue on as individual projects to meet the needs of two distinct markets.

Someone needs to decide where that line is.

From Brendan:

We don't need to have a convention and meeting notes to fork a repo. And there's no reason we have to be particular about who can contribute. I'd be fine with forking the repo and giving anyone who's contributed in the last year write access if they want it. That's probably less than a dozen people.

This is what I would call a diverge, doing so is not problem but we should not expect anything in that new repo to end up in master since development on the new repo is not overseen by the maintainers on the main repo who need to be following a plan, and be in close communication.

If we are going to create branches then I propose:

  1. having a python 3 support branch to allow people to work on that while mainline is kept stable, only after thorough testing and code reviews will it be merged in master

  2. having a refactor branch where an overall overhaul of rez is slowly taking place dare I say re-write?

Starting again. This takes far longer than you'd think.. that would mean no new features for a quite long period

From JC already showed my interest in helping getting a complete rewrite, and so for @lambdaclan. Though as Allan said, that would mean no new features for a quite long period of time, and probably find some money to pay the people that would work on this.

It's a very tempting idea, but it's also often a huge undertaking that doesn't have anything to show for a long time.

From Brendan

I agree, That's even more of a reason that you want more people invested in doing code reviews and approving PR's. No one is going to submit perfect PR's the first time every time (including you), but the solution to that isn't to restrict merge approvals and code reviews to a single person.

I don't think moving to another repo would be a good move at this stage. It may make sense from the point of view of developers who want to iterate on the project more often;

Honestly, it's not really about how fast the project is moving. It's more about just having a clear direction and motivation to do more work on the repo. There's very little point for contributors to discuss issues and do code reviews if they have no authority to approve any changes and their PR's never get any feedback from the one person with merge access.

The highlights speak for themselves.

The state of the repository is very important as a first impression for a new contributor. That needs improvement as well.

well defined procedures and templates for everything that has to do with the repo - merge requests, issue creation, code styling, commit styling, new branch creation, naming, tags etc revamp the wiki to include better contributor guidelines, links to new slack channel etc

Most importantly a healthy and friendly community striving to make the application better by working towards a common goal and encourage people to get involved.

I am all up for discussions to keep going but sooner than later we will need to take some action as well for the benefit of the project.

First important step to achieve this, as stated from Allan

Work on a roadmap detailing where I see the project going, and creating tickets related to this. There are some major features that need to be worked on to turn rez into a fully fledged package manager, and it'd be great to see a focused effort on this. In order to really drive adoption in the industry, I think people need to be able to easily install entire dependency chains of packages from a public repository using rez. We aren't there yet.

👍

--

Ira

mottosso commented 5 years ago

I'd be interested to hear @instinct-vfx opinion on all of this.

instinct-vfx commented 5 years ago

Rest assured that i am following this discussion. There is a LOT being written and it is getting hard to properly answer all the points. With some internal Rez milestones this week and a lot of personal stuff happening i am a bit constrained this week. So i will just talk to a few things that i consider very important.

In conclusion i would once again like to repeat and stress that the project needs a vision and a scope. What does Rez do and what does it NOT do (even more importantly). What are the core concepts and principles Rez wants to maintain? What are supported platforms and environments? etc.

I think based on that it also gets easier to answer other questions like "Rewrite or not" ;)

As a side-note: I think that making more things plugin-based may help as it allows more things to be done in a custom way without polluting the core. It can also be a curse though with everyone running their own flavour of the WSL integration and causing ticket influx.

In the end i would like to say that i really do welcome seeing this discussion happening and also see it happen in a constructive way. Keep it coming!

nerdvegas commented 5 years ago

I also think a roadmap is important, and I'm going to prioritise getting that done soon. Watch this space (or rather, wiki).

I'm going to get through a couple of tickets now, and there'll be some more activity tomorrow.

Thx A

On Mon, Jun 3, 2019 at 5:12 PM Thorsten Kaufmann notifications@github.com wrote:

Rest assured that i am following this discussion. There is a LOT being written and it is getting hard to properly answer all the points. With some internal Rez milestones this week and a lot of personal stuff happening i am a bit constrained this week. So i will just talk to a few things that i consider very important.

-

While it seems "easy enough" to give people merge permissions there are a lot of pitfalls and as Allan noted above it can be very tricky to oversee the potential side-effects of changes, or even more importantly when you are implementing "against" some of the core principles of Rez. I am guilty of that many times and i was really happy to get Feedback from Allan in these cases. As much as i am frustrated by some PRs being stale for longer periods of time (and the work this creates to update them to master again etc.) i still see quite a bit of a risk there. I kind of like the idea of having a sign-off and others doing the actual merge work etc. as a first step. In the long run? I don't know, maybe Rez needs a BDFL?

I am rather strongly opposing forking in a way that was described here. Because even if not intended this will diverge away quickly and we'd end up with contesting code bases. I have seen that both out in the public and even internally (with quite some care being taken heh). Also see my first point. As the fork(s) would lack Allan's input i am pretty sure they'd reach a state that is in conflict with upstream master rather sooner than later.

I am agree violently that the project needs a vision, a roadmap and planning. Also some more in-depth guardrails would help to get more people up to a point where they can better judge the impact of PRs etc. Things like "Complete encapsulation of Rez's env and the envs it provides" etc. should be available in written form and be part of the developers guide. (I may be missing that that exists even? If so it ain't visible enough ;) ). Some random points here:

The github issues are perfectly fine for task planning besides just bug tracking i think. No need to go elsewhere (at least to start with).

Github issues can also be used for release planning and roadmapping to some extent (see bigger projects like the vscode project for example. They currently have nearly 5500 OPEN issues and yet manage good pace and monthly releases.

Speaking of monthly releases: I really like these (gitlab does that, vscode does that). It does not have to be monthly, but having a release schedule also helps to plan features and fixes into these releases and bring predictability.

Managing issues and grooming them, labling them etc. is a lot of work in itself and something that can be done by active contributors right away i'd guess?

WRT a rewrite. Tempting. A full rewrite is always a tempting thing getting more and more tempting the older a code-base gets. But it is also quite a risk and quite an effort, even with funding and resources. Remember XSI 1.0? ;) That said i am not in opposition here, just needs a lot of careful planning to make sure we're not abandoning 2.x without creating 3.x

As mentioned above, proper tests and test coverage would ease my mind. Setting up CI would help there. (There is a workstream in ASWF currently on CI, maybe their best practices can help).

In conclusion i would once again like to repeat and stress that the project needs a vision and a scope. What does Rez do and what does it NOT do (even more importantly). What are the core concepts and principles Rez wants to maintain? What are supported platforms and environments? etc.

I think based on that it also gets easier to answer other questions like "Rewrite or not" ;)

As a side-note: I think that making more things plugin-based may help as it allows more things to be done in a custom way without polluting the core. It can also be a curse though with everyone running their own flavour of the WSL integration and causing ticket influx.

In the end i would like to say that i really do welcome seeing this discussion happening and also see it happen in a constructive way. Keep it coming!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/630?email_source=notifications&email_token=AAMOUSSDH36TKHC6IP2H33TPYS74JA5CNFSM4HOJOBM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWYQFEA#issuecomment-498139792, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMOUSU45GBJR57JEMC3TILPYS74JANCNFSM4HOJOBMQ .

bpabel commented 5 years ago

@nerdvegas Saw this on #629, but wanted to comment here

Been looking into permissions. Unfortunately it isn't possible to restrict access to master in personal GH repos

I think this should be possible by making master a protected branch and checking the "Require pull request reviews before merging" and "Require review from Code Owners" options

image

And then adding a CODEOWNERS in master and making yourself the default code owner for all files.

* @nerdvegas

Then you could selectively grant merge rights to portions of the repo by adding other code owners for different files and directories.

/src/rezplugins/  @bpabel
nerdvegas commented 5 years ago

Oh, good find. I didn't drill down that far, but curiously this suggestion didn't come up after doing some googling either. I'll sort this out soon.

Wrt collaborators. I'm still looking into what the best approach is. Like any OSS project, opening collaboration directly to master for all new developers would not be a good idea. Don't think of this like me considering giving you that access - think of it in general terms, ie how do the collaborators of this project want to control access to new members, moving forward? How we determine who becomes a collaborator and when should be a known process that we all understand from day one. I would like to avoid a situation where I'm arbitrarily choosing who has what access - I think it would be better for everyone if that process is objective. This is what I'm looking into and getting advice on at the moment.

Cheers A

On Fri, Jun 7, 2019 at 3:38 AM Brendan Abel notifications@github.com wrote:

@nerdvegas https://github.com/nerdvegas Saw this on #629 https://github.com/nerdvegas/rez/pull/629, but wanted to comment here

Been looking into permissions. Unfortunately it isn't possible to restrict access to master in personal GH repos

I think this should be possible by making master a protected branch and checking the "Require pull request reviews before merging" and "Require review from Code Owners" options

[image: image] https://user-images.githubusercontent.com/622914/59053407-3e8c9680-8846-11e9-85e9-9e892e063b19.png

And then adding a CODEOWNERS https://help.github.com/en/articles/about-code-owners#example-of-a-codeowners-file in master and making yourself the default code owner for all files.

  • @nerdvegas

Then you could selectively grant merge rights to portions of the repo by adding other code owners for different files and directories.

/src/rezplugins/ @bpabel

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/630?email_source=notifications&email_token=AAMOUSVF4RZBEEEAC26YVKDPZFDS3A5CNFSM4HOJOBM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXDTPCI#issuecomment-499595145, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMOUSULRF2ICB2NBOS57EDPZFDS3ANCNFSM4HOJOBMQ .

instinct-vfx commented 5 years ago

How we determine who becomes a collaborator and when should be a known process that we all understand from day one.

Quoted for agreement!

mottosso commented 5 years ago

Hi all,

I've enjoyed this topic and others like it, but I've been informed that my PRs and issues take up too much of Allan's time, so much so that they negatively affect his ability to develop his roadmap. So I'll take my leave and divert attention to the bleeding-rez project instead.

Best case, progress made there can help inspire and push progress here (no point pushing in parallel). Worst case it's a fork on a different path (no point pulling from both directions). Neither of which is all that bad, both of which means progress. Have a good weekend! :)

nerdvegas commented 5 years ago

That's unfortunate, but it's a free world. However, I think this misrepresents what I actually said.

Excerpt follows:

[AJ] Well, I think rather than spreading our time over a lot of threads, we should spend more time on the ones with traction, and get things done, then revisit things that don't have consensus yet. There's a finite amount of time to do things, right now I feel like we're making slow progress. I have an hour or two, 5ish days a week so I literally can't spend it on long threads all the time - if it's a long back and forth then that can be two days gone, then perhaps the PR/issue gets closed and I've not made any progress.

I don't mean it as a criticism that you're communicating too much per se, more that I just need to have a way to get things done within a finite amount of time. Typically I'd be doing development myself to ensure progress is being made, but I'm trying to move to a place where I'm not doing so much development, in order to keep checkins going faster. But I currently find myself in a catch-22 where there's not much progress.

So, for example right now there's good work being done in 3 different PRs (installer, preprocessor-defined-in-config and request-iterator-in-rex). Two of those haven't had any input for some time now - so I think it's a good example of where we can concentrate more effort (one is also holding up another developer). Then things are getting done and I can spend more time merging older PRs, responding to the longer conversations, and developing (sometimes maybe! There's some interesting meaty problems to solve).

A

On Sat, Jun 22, 2019 at 5:06 AM Marcus Ottosson notifications@github.com wrote:

Hi all,

I've enjoyed this topic and others like it, but I've been informed that my PRs and issues take up too much of Allan's time, so much so that they negatively affect his ability to develop his roadmap. So I'll take my leave and divert attention to the bleeding-rez https://github.com/mottosso/bleeding-rez project instead.

Best case, progress made there can help inspire and push progress here (no point pushing in parallel). Worst case it's a fork on a different path (no point pulling from both directions). Neither of which is all that bad, both of which means progress. Have a good weekend! :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/630?email_source=notifications&email_token=AAMOUSV3EDWBILYEFJ2KAPDP3URDJA5CNFSM4HOJOBM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYJKSLA#issuecomment-504539436, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMOUSQGGDVGWKFHE7EHPKLP3URDJANCNFSM4HOJOBMQ .

KelSolaar commented 5 years ago

A bit sad to come back to this thread only to realise that there has been a hard fork of Rez. Feels like wasted energy to me, instead of having a single repository we essentially now have two competing ones.

@mottosso : I haven't followed precisely the changes you made in bleeding-rez but are you at a point where the codebases deviated so much that they cannot be reconciled and merged together?

mottosso commented 5 years ago

@mottosso : I haven't followed precisely the changes you made in bleeding-rez but are you at a point where the codebases deviated so much that they cannot be reconciled and merged together?

Hey @KelSolaar, I think it isn't as much about whether it's possible, but about whether it's wanted. Generally, I'm less interested in the 100 studios using Rez as I am in the 100,000 studios not using it; that is, I'm pushing towards lowering the barrier of adoption and making it useful to a widget audience. That involves using Rez for project configurations, the simplified installation method (pip install bleeding-rez), solving the problems associated with that and generally implementing features that only make sense to new users.

I wouldn't say one is better, but that the two projects address different requirements.

And as someone not bound by any particular studio, I'm also able to provide dedicated support to those on a deadline, feel free to get in touch.