Infrastructure sharing method needed

[baumanj]

Since the habitat repo has been split into habitat, builder, core, etc., we have some duplicated infrastructure. This has led to some issues such as the travis libsodium build being broken. Issues like this are likely to occur more in the future, so it would be helpful to find a sustainable way to share this code

Aha! Link: https://chef.aha.io/features/APPDL-51

[christophermaier]

Data point: the configuration of which Clippy lints we choose to fail a build for. Ideally, these would be the same across all our Rust repositories.

[baumanj]

I did a bit of investigation on methods here and I think we probably want to use submodules, but confirming that the workflow is user-friendly before we commit would be nice. Here's a good summary of the difference between submodules and subtrees. The upshot is that since submodules make changes to the shared content easier, that's probably what would work best for us.

[christophermaier]

@baumanj Yeah, I was thinking submodules might be the way forward, too.

[raskchanky]

Submodules you say?

[raskchanky]

Another thought - we originally split the repos up because the tools we were using to run our release process didn't play well with a large repo. Most of those tools have gone away at this point, and with the impending move to BK, I wonder if moving back to a monorepo would be worthwhile. There's certainly a lot of extra coordination and hassle associated with multiple repos - this issue being one, but also things like changes to the core crate being prerequisites for changes to other crates in separate repos, Cargo.toml patch shenanigans, etc.

[christophermaier]

I don't think I've been burned by submodules before in the way that everyone else on the internet seems to have been, so I'm OK with that solution. I look forward to eating my words, though. :smile:

Remerging the repos is a very interesting proposal. It might be worth gaming that out.

(paging @chefsalim and @eeyun ☝️ )

[raskchanky]

I have a pretty strong preference for re-merging the repos, assuming our release tools at this point are strong enough to not care. There was significantly less hassle that way.

Submodules are fine if you do them right. I've used them with success before. My main issue with them is they add a non-trivial amount of complexity to the process. It's totally manageable, but it's just one more thing that everyone needs to be able to do effectively. I've seen them trip a lot of people up in the past.

[stevendanna]

I have a pretty strong preference for re-merging the repos, assuming our release tools at this point are strong enough to not care. There was significantly less hassle that way.

As an "external"-ish contributor, I can confirm that even when I know what to do, anytime I have to modify anything that crosses 2 repositories, it kills a bit of my enthusiasm just because if I haven't been working on Habitat that week, I almost surely have to overcome some new build environment issue that only crops up when trying to build both repositories at HEAD.

[jeremymv2]

Here's someone we know very well making a great case for mono repos! https://medium.com/@adamhjk/monorepo-please-do-3657e08a4b70

[scotthain]

I am very on board with the monorepo concept in this case. It would definitely allow us to consolidate configuration and get very granular and explicit about what we are building, shipping, and testing.

[baumanj]

Mo' money, mo' norepo

[chefsalim]

I'm in support of merging hab and core (assuming theres no new dependency madness that surfaces). For Builder things are not clear cut and my preference is to keep it independent. Builder (and its on-prem offshoot) are significant enough projects in their own right now, and the release cadence, versioning, issue handling, development environment, and many other things are going to need to operate independently. In addition, the effort to extract and clean up the dependencies that came out of the original repo split was very worthwhile and helped simplify and clean up a lot of extra un-necessary crud. Merging things in again is likely to take us back to the world of spaghetti deps that we had earlier in Builder.

[christophermaier]

@chefsalim That should be possible... Builder would need to tweak how it pulls in the core crates, since they'd be embedded in the habitat repo, but if that works, then it seems like a happy middle path.

[scotthain]

@chefsalim I think there are some options here we can talk about with regard to having builder be a separate thing depending on things in habitat versus having the builder code in the same repo and having releases (of builder, supervisor, and "habitat") be subsets. I'd like to know more about the versioning and release cadence concerns (and what we would like to be testing for each of these components when we release).

[chefsalim]

@cm 👍 on the happy medium

@scotthain I'm not clear on the test question, however it may be orthogonal to this discussion. The release cadence I'm talking about is things like how fast and when Builder updates itself, procedures such as merge freezes, etc. As long as things are in separate repos we can let those pipelines operate independently based on their needs. It's not a problem today since deps are explicitly pinned across the repos.

[scotthain]

@chefsalim totally - this is probably a better in person/mouth-words question, and it is orthogonal to this discussion.

[raskchanky]

We decided today to merge the core repo back into habitat, but leave builder separate for now. We'll revisit the decision to merge builder at a later date, when our release process is more streamlined.

I'm going to close this for now, as I don't think there's anything actionable left to do, short of the actual repo merging (which will happen as soon as all the PRs in core are merged).

[baumanj]

Merging core back into habitat reduces the problem, but isn't there still a fair bit of infra logic we'd like to share between habitat and builder (clippy, shellcheck, rustfmt)? If we don't care about keeping the two in sync, that's fine, but if we do, I'd say this issue is still relevant.

[raskchanky]

My (possibly incorrect) assumption based on the conversation yesterday was that things would stay as they are currently (in a semi-duplicated state) until we revisit the decision to merge builder back into habitat. If that's not the case, though, and we want to put more effort into this in the interim, then yeah, we should re-open.

[scotthain]

There absolutely is some shared logic, as you mention. In this case I'd personally rather we focus on getting the prereqs done to be able to merge builder back in rather than investing in a syncing/sharing mechanism that we will eventually throw away. (I don't think that's what you were implying but that's what I think about when we start talking about syncing/etc 😄 )

[baumanj]

I like the plan or merging builder back in eventually and it would fully address this issue. Any objection to re-opening it pending that? If a few months go by and the builder repo hasn't re-merged, we can reevaluate whether it's likely to happen or whether it's worth pursuing another method for sharing.

[scotthain]

Personally I'm fine with reopening or opening a more focused issue. I'll defer to the team!

[raskchanky]

Heck yeah, let's reopen it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale[bot]]

This issue has been automatically closed after being stale for 400 days. We still value your input and contribution. Please re-open the issue if desired and leave a comment with details.