Make a monorepo and/or gitlab

faddat commented 2 years ago

I have an earlier issue on this, and I think that we decided not to do it, but I'd like to walk through reasoning on this a second time, I hope that no one minds.

Basically, this repository here should contain osmosis.

The reason for that is visibility. Just for example, I really don't know much about the front end development process for osmosis, though it is an area where I could contribute (but please not to visual elements, it would look like a first grader did it).

So let's go ahead and define osmosis:

the blockchain code
the genesis state (which we may want to consider using Go's embed to put into the binary itself)
the front end
the javascript libraries
the documentation
the installer
the dashboard

This could help Osmosis to evolve more as a focused product, because-- and maybe this is just a consequence of my personal style-- it is easy to lose track of work going on in other repositories. When I check in on osmosis, I do so here, in this repository. Also, this repository is used to track issues on other repositories, like the cosmos-sdk fork that we use.

Why maybe gitlab

Github's support for self-hosted runners is awful. If we were on GitLab, much more effective integration testing could be done, and I think that I saw some discussion of that from @daniel-farina.

I have a ton of experience with gitlab ci pipelines and I'd be happy to get this kicked off

faddat commented 2 years ago

I think that this issue is important to accomplishing this one:

https://github.com/osmosis-labs/osmosis/issues/1014

and that we should also start to do "speed runs" from genesis when we change code and to encourage the maitenance of Osmosis v3, v4, v6, and v7

I have a bit of the speed run setup put together, but it is definitely not good enough yet.

faddat commented 2 years ago

Hey, thinking about this, I think that it's best that I "JFDI" it and show what it would look like.

For context, I'm going to use the Optane drive we bought for testing. This should allow us to do many things that we otherwise could have. While we could use a self-hosted gitlab environment, I don't think that is the best path. What we did for blurt was to just connect one beefy machine, and ah, it is not far from me as I speak, so maybe I just begin having runners again, this would help the blurt CI situation, too.

faddat commented 2 years ago

I made: gitlab.com/osmosis-labs and I invited @ValarDragon as an owner.

I'm OK w/ stepping down as owner once owners here have been assigned as owners.

faddat commented 2 years ago

OK, so pursuant to this I made a PR to the archive script:

osmosis-labs/osmosis#1270

For now, I am going to get a gitlab workflow going on my big machine:

threadripper pro 32 core
optane
256gb

and I am going to focus on automating speedruns, this way as we backport work from @p0mvn, we can track changes.

faddat commented 2 years ago

Hey, I think that speedruns for comparative db performance testing are more important/pressing than a monorepo right now.

I still like the monorepo thing, I still have the gitlab and can hand it over to whomever, and currently I just want to put some automated performance tests in place so that we can:

test db's against each other
test how much different changes to iavl change performance
compare osmosis and gaia

Please let me know if these sound like decent priorities

daniel-farina commented 2 years ago

In regards to the monorepo: I think the place to define each component would be docs.osmosis.zone (https://github.com/osmosis-labs/docs). This is work in progress and once the refactor of the front-end is complete we should have better docs for the front-end as well.

Gitlab Why do you need custom workers in order to run jobs? can we use docker and run images in Github actions for example? I think automated performance testing is def something that needs to be done. Also see. https://github.com/osmosis-labs/osmosis/issues/209 & osmosis-labs/osmosis#1181

This is def a topic I would like to learn more about, thank you for all the thoughts and ideas!

faddat commented 2 years ago

Hey sure thing let me break it down for you:

Gitlab

Unless something has changed in the past 2-3 months, when you self-host a github runner, it is not possible to run jobs concurrently-- in fact, that is pretty much the only reason for any of this.

I think that I would like to kind of scale back my plan for the gitlab right now though, to just include some automated speedruns so that I can get the environment set up correctly.

I reckon that we need self-hosted runners because we really demand vast iops.

I have tried to do this on github in the past and run into endless issues, but it was quite smooth with gitlab.

It also may be possible to just mirror this repo on gitlab, and then run CI against it, but that's got limitations. Another alternative could be "Concourse CI".

Monorepo

This is almost entirely a visibility hack, which would make sure that the canonical docs and UI are always present and accessible.

I think it also helps some with product definition.

example about the visibility hack: I wasn't aware of the dashboard repo, basically because it lives outside of what I routinely check (this one)

My current plan

As we speak I'm confirming viability for rocksdb and should end up with a decent archive node running rocks. It looks like it is on the path to success. Once I've manually confirmed that it works, I am going to figure out how to "speedrun" the different databases against each other. Another benefit of this is that the logs will be preserved for review, and we will be able to monitor performance changes over time. This'll also allow me to figure out if gitlab is worth the effort.

ValarDragon commented 2 years ago

Theres several components to this issue, trying to breakdown my understanding of it:

Gitlab

Were unsatisfied with CI speed. This seems like something that must be fixable within githubs frameworks though, e.g. adding more things to workflows, or making github actions trigger events on other machines, parallel tesing, etc.

The transition costs of going to gitlab are immense. I think we need to put much more effort in understanding whats the thing we want to do in CI right now but can't, and assess options for fixing it, if CI is the reason to do a switch.

Monorepo

Modularity is imo key. This repository serves one specific purpose, defining the Osmosis state machine, and tooling to run the state machine, and extract data from it.

There is no canonical osmosis UI. Theres already multiple front-ends, that do different components of the stack. Dozens of staking dashboards. For trading & LP'ing, theres at least 6. app.citadel.one, cosmostation wallet, app.osmosis.zone, frontier.osmosis.zone, ping.pub, {bit songs}

Anyone can make more that connect to the core protocol! I know there are several teams building vaults, and arbitrage bots. Hopefully more emerge that build aggregators, ala 1-inch. I am pretty opposed to any front-end being in this repo, none should get canonicalized within the same repo. Imo, at most we can have JS tooling in here, if theres a structure that makes sense.

DB performance

Thanks for working on this! Very cool, and agreed as near term important!

osmosis-labs / feature-requests