[ENH] - Add ansible playbook for dev

nebari-dev / nebari

🪴 Nebari - your open source data science platform

https://nebari.dev

BSD 3-Clause "New" or "Revised" License

279 stars 92 forks source link

[ENH] - Add ansible playbook for dev #2345

Open dcmcand opened 7 months ago

dcmcand commented 7 months ago

Feature description

Add ansible playbook for setting up a dev environment for nebari

Value and/or benefit

Easier onboarding and getting started

Anything else?

No response

Adam-D-Lewis commented 2 months ago

Does any other open source python project do this? It seems a bit of pain requiring ansible to run a few shell commands to create a conda environment and run pip install -e .[dev].

marcelovilla commented 2 months ago

It's been quite a few months since we discussed this, but my main takeaway was that the playbook would be useful to create a local dev cluster and a dev AMI (potentially using packer) with that. The AMI would then be used to run tests again an "existing" cluster, without the need of deploying a cluster every single time we want to run the local integration tests.

I think this would be highly beneficial but we would need to think when to run the deploy/destroy tests. Some options are:

Every time a PR is merged
Every time we cut a release (candidate)

Reading the issue description, I'm now wondering if what I just outlined was actually scoped or not.

Adam-D-Lewis commented 2 months ago

@marcelovilla Ahh, I took environment in the original feature description to mean "conda environment". I see what you mean now. Thanks for clarifying.

viniciusdc commented 2 months ago

as @marcelovilla said the main use case would be the AMI setup, and I also think we could merge #2554 and use that in our benefit to reduce the complexity of the ansible tasks

viniciusdc commented 1 month ago

So, I've done some testing around this, and here are my main points: PROS:

Using Pecker with the playbook was an incredible tool to create the instance with a running nebari on it; CONS:
Running Nebari commands internally within the playbook works, but its a nightmare to properly instepec the logs from Terraform, as Ansible puts all of them inside a dictionary (also only shows up once the task concludes)

So my takeaway on this would be:

Have the makefile PR merged without changing any of our current tests, so that we can further enhance it if needed;
Use the Ansible playbook to create a suitable environment within the EC2 instance without running any nebari commands
Explore a way to run the nebari deploy within the VM outside of the ansible playbook and catch the messages in a more visible format (maybe an ssh exec)

Adam-D-Lewis commented 1 month ago

BTW, Packer's license changed recently along with Terraform's so I assume we'll need to use a version of Packer prior to the license change to BSL in August 2023.