Open hwine opened 3 years ago
I think we could:
frost/
directory and update the packaging manifest (refs: https://github.com/mozilla/frost/blob/master/frost/cli.py#L16)switch_to_frost_parent_directory
frost test
to include the first-party tests path by default (with a flag to disable it), but run from the directory it's invoked in to include third party / local / dev tests I think there is also the challenge of keeping pytest happy with it's perception of the world at frost execution time.
Normally, pytest runs from the root of the test suite, and all conftest.py
and other files are only in that directory tree. We'll be asking it to consider some disjoint directory (wherever the frost package was installed) as the "top root", and then the normal test root. I.e. we need pytest to include the frost conftest.py file as part of the chain of magic conftest imports it does.
@g-k I think I understand #3, and that wasn't a case I had considered. But we should define the behavior we support.
I think we're conflating "using" with "developing". With most python packages, if I'm only going to use the package, I just install it via pipx
or some such. If I also want to develop on it, I clone the repo, create a venv, and install in "develop" mode.
We can make a case for there being a third mode for frost "install system wide, but use my own test suite", but I'd be concerned that the costs of supporting that "unusual mode" might be high with all the magic already happening.
Let me define some behavior in original comment -- we can debate & update that, so we have clarity going forward.
@g-k @ajvb - I proposed a behavior specification in the opening comment, just to get a discussion going in a visible place (i.e. we'll update that when we get agreement). Have at the comments!
I think it'd be nice to support developing and running custom tests that:
I agree it would be nice. However, I don't think supporting that use case is as urgent as some other items.
@g-k Let me ask this: do you see specifying the "traditional package dev model" as preventing an eventual move to support this use case?
If part of your concern is the cost of a dev setup for someone who isn't a python user, I think we could do a docker image and/or add vscode support to make it "one click" for non-devs.
I agree it would be nice. However, I don't think supporting that use case is as urgent as some other items.
Absolutely, I know you and AJ have a roadmap.
@g-k Let me ask this: do you see specifying the "traditional package dev model" as preventing an eventual move to support this use case?
I don't. We might not want to support it anyway, since it'd probably result in confusing end user behavior and drawing a stricter distinction between running and developing might be good especially if we went another direction like allowing users to provide tests via plugin.
I mainly wanted to mention items 1 and 2 from https://github.com/mozilla/frost/issues/411#issuecomment-756992325 since it's tech debt that impacts how frost is packaged and run.
[implementation note: pytest-pythonpath may help with "pointing to the right place". See/listen/read to segment 2 of episode 220 of Python Bytes podcast for a better description of a use case that may be similar to frost.]
Originally posted by @ajvb in https://github.com/mozilla/frost/issues/401#issuecomment-724184034
Currently, while
frost
can be installed from pypi, that isn't very meaningful, as you still have to executefrost
from the top of a clone of the frost repository. (Due to customizedconftest.py
, etc.)Allowing
frost
to be run against an arbitrary set of tests would be useful, but it's unclear if the packaging can be munged around to do that cleanly.Desired Behavior
When a user installs
frost
from pypi, it comes bundled with the current test suites. (Enhancement, package each service's test suite as a separate extension.)frost
will execute only the tests bundled with the installation. Bothfrost
and the test suites will be updated by a normal update from pypi.This implies that when a user is developing tests, they are effectively developing frost. To do so, they would need to be working on a checked out version of the
frost
repository, and ensure they run the frost in the checkout via appropriate isolation techniques (virtualenv, docker, etc.)Scheduling
This enhancement is unlikely to be considered until a Phase 2 or Phase 3 implementation time frame.