search

codebuddies / backend

CodeBuddies back-end
https://codebuddies.org
GNU General Public License v3.0
20 stars 25 forks source link

chore: Add Ansible, molecule and pytest-bdd #173

Closed tgrrr closed 3 years ago

tgrrr commented 4 years ago

What

Why

What type of PR is this? (check all applicable)

Context

Add ansible, pytest-bdd, and mocule

Eg. for ticket https://github.com/codebuddies/backend/issues/111 [User Registration] Make email required on backend #111

The requirements are:

- [ ] Change the line in the model to make sure email is required
- [ ] Make migrations file and run migrations
- [ ] Add models test
- [ ] Make sure tests pass

Other Related Tickets & Documents (as needed)

Implementation Details

What was your thought process as you changed the code?

Looking at the current issues for the MVP tagged P1 for MVP

BDD will help with that

Ansible and molecule will help with that

What does someone need to consider in reviewing it?

If there's any reason not to include Ansible If we should use behave vs python-bdd

New Libraries/Dependancies Introduced (Fill out as needed)

ansible==2.9.12 # https://github.com/ansible/ansible pytest-bdd==3.4.0 # https://github.com/pytest-dev/pytest-bdd molecule==3.0.8 # https://github.com/ansible-community/molecule

If you have added libraries or other dependancies, please list them (and links to their repos) below:

Any new migration files added?

Did you add tests?

Code added or changed without test coverage or good reason for exemption won't be approved.

TODO: Did you add documentation?

codecov[bot] commented 4 years ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@551d43e). Click here to learn what that means. The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #173   +/-   ##
=======================================
  Coverage        ?   83.78%           
=======================================
  Files           ?       32           
  Lines           ?      518           
  Branches        ?        0           
=======================================
  Hits            ?      434           
  Misses          ?       84           
  Partials        ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more Ξ” = absolute <relative> (impact), ΓΈ = not affected, ? = missing data Powered by Codecov. Last update 551d43e...f194279. Read the comment docs.

BethanyG commented 4 years ago

@tgrrr -- Not convinced that Ansible gives us more than it adds in config and extra libraries to install. Do you have links to articles/use cases I could look at?

We already test that migrations are present using GitHub Actions, so if that is the only use case at the moment for Ansible/molecule, I would pass -- or go with a purpose-build test package like django-test-migrations, should we need to test the application of migrations (roll forward/roll back/order of execution, etc).

I also want to make sure that this setup is not "locking in" BDD - while it is a useful paradigm, it is not the only paradigm - nor is it a one size fits all solution. I do not want to get into a situation where tests already written will have to be re-written either in the BDD paradigm or vice-versa. Anything we choose should be compatible with Pytest/pytest-django as a test runner, and interoperable with the other tests and frameworks we've used.

From some looking around, it appears that Behave/behave-django and Behave/django-behave are not compatible with pytest as a test runner at all (pytest fails to discover tests). So my preference would be for pytest-bdd -- but I'd like some concrete details on why that necessitates Ansible/molecule.

tgrrr commented 4 years ago

Not convinced that Ansible gives us more than it adds in config and extra libraries to install. Do you have links to articles/use cases I could look at?

This is fairly simple, but I feel this is a good article https://developer.ibm.com/tutorials/d-bbd-guide-iac/

@BethanyG I might get @bengineerdavis to weigh in on this

My overall goal is to run tests across the entire application, including Docker. So, if someone new runs the setup, and something fails, it'll point them exactly (and hopefully clearly + explicitly) towards how to fix the problem.

I also want to make sure that this setup is not "locking in" BDD - while it is a useful paradigm, it is not the only paradigm - nor is it a one size fits all solution. I do not want to get into a situation where tests already written will have to be re-written either in the BDD paradigm or vice-versa. Anything we choose should be compatible with Pytest/pytest-django as a test runner, and interoperable with the other tests and frameworks we've used.

I agree with all of this

From some looking around, it appears that Behave/behave-django and Behave/django-behave are not compatible with pytest as a test runner at all (pytest fails to discover tests). So my preference would be for pytest-bdd -- but I'd like some concrete details on why that necessitates Ansible/molecule.

That sounds like an important discovery. Do you mind posting the sources?

Thanks,

Phil

lpatmo commented 4 years ago

I also want to make sure that this setup is not "locking in" BDD - while it is a useful paradigm, it is not the only paradigm - nor is it a one size fits all solution. I do not want to get into a situation where tests already written will have to be re-written either in the BDD paradigm or vice-versa. Anything we choose should be compatible with Pytest/pytest-django as a test runner, and interoperable with the other tests and frameworks we've used.

Agree as well!

I do have a couple of followup questions:

My overall goal is to run tests across the entire application, including Docker. So, if someone new runs the setup, and something fails, it'll point them exactly (and hopefully clearly + explicitly) towards how to fix the problem.

I've helped a couple of people get set up on the backend, and when there's a docker failure it's been either:

In those cases, would the BDD tests help illuminate more details about what the user should be doing to resolve an issue?

Thanks!

BethanyG commented 4 years ago

@tgrrr After reading through the article, it seems that Ansible is providing infrastructure for provisioning servers. While that may be useful for deployment, and for integration testing in a staging environment (which for us will be one or more Digital Ocean Droplets), we are talking here about a development environment and the writing/running of unit tests for backend and API code.

Ansible might help me provision a droplet on DO so that I can deploy the "stack" and make sure I got the process correct -- but it will do nothing for me in troubleshooting docker permission problems on localhost for when I need to talk to the DB, nor is it useful in diagnosing a missing registration in the Django Admin app, or a serialization error. It also adds several additional layers of things (playbooks, YAML, supporting libraries) for newcomers to learn on top of an already complex-feeling stack.

I am not saying it won't be useful in an integration testing/QA/deployment toolchain -- I am saying we are not at the stage of writing integration tests. Right now, we need to make sure everyone writing code for the application writes proper uint tests for that code - and that those unit tests are as easy as possible to think through, make, and run.

here are some of the articles/SO posts on BDD I've scanned through. I put particular emphasis on Automation Panda, since his "shtick" is Python-flavored test automation.

EDIT: Adding this course from Automation Panda which looks really interesting: Automation University BDD with pytest-BDD

bengineerdavis commented 4 years ago

Hi everyone, I'm just catching up on this conversation.

It didn't occur to me we would be adding ansible into Main or for wider adoption just yet. Ansible was supposed to be a tool that @tgrrr and I were using for ourselves while working within the context of the Backend repository. Just to be clear--I have no intention of committing to a team-wide adoption of Ansible until we both had tried it, applied it, and had a wider discussion (and user tests) on it. If anything, I would prefer this be its own experimental branch, should we reach that consensus. If needed, we'll work off our own fork before making any proposals.

Let's clarify expectations around Ansible: we're talking about a configuration management tool--it can neither solve problems inside Django, nor replace the functionality of our current infrastructure. It does relief us (potentially) of some of the manual labor surrounding dependencies, abstract infrastructure management, and automate setup for the local and production environments.

It also doesn't impose itself on team members who want to use the individual tools, such as compose, and for most members of the team, they would be running one or two commands they can copy and paste.

When talking about Ansible complexity, we can argue the same thing for docker(-compose), which takes some experience to manage and customize directly, but takes almost no effort to run in its intended state--as a background process; given my experience level, I'm a good barometer for user-testing this.

Ansible can also help you set up your local development environment, and can be valuable for automating the dev prerequisites needed for new and less experience devs to get "up and running". This would give a similar abstraction to the dev environment (installing docker, compose, etc.), in a turn-key approach as docker-compose does for managing/abstracting our services architecture, and does so using yml formatting. The added benefit is we can use the same tool when deploying or troubleshooting issues in our production environment with virtually the same configurations (plus a few variables/templates for abstraction).

Among other benefits:

I think a potential deal-breaker would be if we couldn't leverage the existing docker-compose files, then that would impose Ansible as a requirement, rather than an enhancement--which I would prefer to avoid.

bengineerdavis commented 4 years ago

My intent for this exercise is to focus on unit tests. Pytest doesn't do integration tests very well, anyway--at least from what I've read.

Separately, I would strongly discourage Behave, because that is an entirely different testing suite than what we have: pytest-bdd at least guarantees that any tests can be maintained/recovered if we moved away from the bdd plugin, because the test themselves should be regular pytest. My understanding is that BDD can be a good way to discover unit tests by deducing them from features and potential integration tests.

lpatmo commented 4 years ago

Ansible was supposed to be a tool that @tgrrr and I were using for ourselves while working within the context of the Backend repository.

Totally fine to fork and explore as a personal project! :)

In terms of product priorities, I think we should be focusing on a couple other things like getting API endpoints up (currently being explored in /projects in this tracking issue: https://github.com/codebuddies/backend/issues/79), email verification, etc.

tgrrr commented 4 years ago

I think there's 2 conversation threads going on here.

  1. Ansible
  2. BDD

I think the best solution is to split this into 2 PR's, any objections?

tgrrr commented 4 years ago

I think there's 2 conversation threads going on here.

  1. Ansible
  2. BDD

I think the best solution is to split this into 2 PR's, any objections?

BethanyG commented 4 years ago

I think the best course of action here would be to not check in Ansible/molecule at all, given that we are working on using GitHub Actions for deployment to DO at the moment (see Draft PR #176). They can of course be used locally - but I don't think they're a priority for this project. (see @lpatmo comment above).

For pytest-bdd, recommend that once a PR for an issue is developed and the tests written, it be included there. From the note on #82, it looks like that's the one you're going to tackle? We closed #111, because the authentication scheme we are using doesn't trigger off an email on the custom user model. See comments there and on discussion number 178.

@lpatmo Thoughts here?

lpatmo commented 4 years ago

I think there's 2 conversation threads going on here.

  • Ansible
  • BDD

Good call out!

I think the best course of action here would be to not check in Ansible/molecule at all, given that we are working on using GitHub Actions for deployment to DO at the moment (see Draft PR #176). They can of course be used locally - but I don't think they're a priority for this project.

Agree -- but as I mentioned before, you're welcome to fork/continue exploring it personally if it's something you're interested in.

Re: BDD -- I'm down to explore it since I've never written a backend BDD test before (I did read the links you shared!), and I still want to wrap my head around this question:

Side note: ultimately it's great if we have good tooling, but I also worry about getting lost in the weeds of infrastructure shiny things/"best practices" vs. focusing on the getting API endpoints up, teaching (partly myself lol) and everyone how to write good tests, and a working prototype. :) Right now I see the backend priorities as being:

tgrrr commented 3 years ago

Lets delete this PR @lpatmo

lpatmo commented 3 years ago

(Note: generally, feel free to close PRs you open! Don't need my permission to close a PR you've created :))