getting the valid input in mock

[dvu4]

Hi,

I tried to mock the input type of a function create_service in services.py .

from mockito import verify, when, ANY, not_
from .api import services
from .models import Token

class TestCreateServicePrincipal(TestCase):
    def test_input_name_service_principal(self):
        when(services).create_service(ANY(Token), ANY(str), None, None, "p",False).thenReturn(1)
        when(services).create_service(not_(ANY(Token)), ANY(str), None, None, "p",False).thenRaise(TypeError)
        when(services).create_service(ANY(Token), not_( ANY(str)), None, None, "p",False).thenRaise(TypeError)
        when(services).create_service(not_(ANY(Token)), not_( ANY(str)), None, None, "p",False).thenRaise(TypeError)
        verify(services).create_service(ANY(Token), ANY(str), None, None, "p",False)
                #verify(services).create_service(not_(ANY(Token)), ANY(str), None, None, "p",False)
                #verify(services).create_service(ANY(Token), not_( ANY(str)), None, None, "p",False).thenRaise(TypeError)
        #verify(services).create_service(not_(ANY(Token)), not_( ANY(str)), None, None, "p",False).thenRaise(TypeError)

I am setting up a mock with when and then using verify to evaluate the valid or invalid inputs but I got the following error. Is there any way I can fix the error?

mockito.verification.VerificationError: Wanted but not invoked: create_service(<Any: <class 'Token'>>, <Any: <class 'str'>>, None, None, 'p', False) Instead got: Nothing

[kaste]

I think there is a misconception here. You first when something and then immediately verify the usage of the mock but you actually never used the mock. T.i. there is no execution phase in the test. (when = setup, verify = assert, the execution of the system under test is missing)

[dvu4]

Can you explain more what is missing in the execution of the system under test?

[kaste]

Typically

when(services).create_service(...).thenReturn(1)  # setup
assert services.create_service(1, 2, 3, 4, 5, 6) == 1  # usage
# now you could verify; you don't need though because you already asserted
unstub()  #  clean up (undo the stubbing) after the test, usually in the tearDown()

[dvu4]

Thank you for clarification. Is that any way I can assert it raises the correct exception like TypeError. I tried with invalid Token class

data = {
        "token_type": "mock_token", 
        "expires_in": 2023, 
        "ext_expires_in": 2025, 
        "access_token": 1 # --> valid one should be str , I changed it to int
        }
token = Token(**data)
long_name = "foo"
when(services).create_service(not_(ANY(Token)), ANY(str), None, None, "p",False).thenRaise(TypeError)  # setup
assert services.create_service(token, long_name, None, None, "p",False)  # usage

But I got the error

mockito.invocation.InvocationError: 
Called but not expected:

    create_service(Token(token_type='mock_token', expires_in=2023, ext_expires_in=2025, access_token='1'), 'foo', None, None, 'p', False)

Stubbed invocations are:

    create_service(<Not: <Any: <class 'models.Token'>>>, <Any: <class 'str'>>, None, None, 'p', False)

[kaste]

I still think there is something odd here. Are you experienced with mock driven development or TDD etc. ? The when(services).create_serceive(not_(any_()),...) seems untypical specific. The matchers not_(ANY(...))) translate to if not isinstance(token, Token: ... raise TypeError but then you actually call it with a token so this never matches the expected/stubbed invocation. If you define one very specific when invocation, mockito expects and only responds iff the system under tests calls and uses that exact same call/invocation signature. Otherwise it throws "unexpected invocation".

E.g.

when(os.path).exists("foo").thenReturn("No")
# usage
os.path.exists("bar")   # <= this will throw

For your code, typically the real implementation of create_service will throw TypeError if it is called with the wrong arguments. I assume you don't want to execute the code in create_service. How does the code look you actually test here? You don't test create_service here but the function which calls create_service. Am I right?

I see that the token you instantiate is an invalid token, but it is an instance of Token. Who decides this token is invalid?

[dvu4]

I am not experienced with mock driven development or TDD. My main goal is to test if the input is valid or not. The instance Token is defined below,

class Token(BaseModel):
    token_type: str
    expires_in: int
    ext_expires_in: int
    access_token: Optional[str] = None

I don't want to execute the code in create_service. I just want to throw exception such as TypeError if input is incorrect. I am not sure if mockito allows me to define the correct type such as when(services).create_serceive(not_(any_()),...), then test if I add a specific input to test the validity of the input.

I am wrong when calling token with a Token in The matchers not_(ANY(Token))), so I changed token to a random string

    def test_invalid_token_valid_long_name(self):
        token = "mock_token"
        long_name = "foo"

        when(service).create_service(not_(ANY(Token)), ANY(str), None, None, "p",False).thenRaise(TypeError)
        service.create_service(token,long_name, None, None, "p", False)

Is it normal if this error occur?

  File "/Users/.venv/lib/python3.9/site-packages/mockito/invocation.py", line 410, in raise_
    raise exception
TypeError

[kaste]

Well now you get the TypeError you want. However, I think the testing strategy should be different. We should focus on the function you're actually testing first.

Maybe it looks like:

def make_service(token, service):
    try:
        return service.create_service(token, ...)
    except TypeError:
        return None

If this is the function under test, then one test could look like:

def test_returns_none_if_create_service_fails(self):
    when(service).create_service(...).thenRaise(TypeError)
    assert make_service(mock(), service) is None

def test_passes_token_down(self):
    token = mock()
    expect(service).create_service(token, ...)
    make_service(token, service)
    verify(service).create_service(token, ...)

These are just examples. To help you out, you should focus and maybe post the function-under-test. This function uses a dependency (service) or a resource, or does a unwanted side-effect we want to capture. (Anything that is not a pure computation. Something that is not a mathematical function probably.)

Everything else, we also have arg_that, which lets you check an argument basically to program your own matcher, e.g.:

def check_token(token):
    if not isinstance(token.access_token, int): 
        return False
    return True
when(service).create_service(arg_that(check_token), ...).thenRaise(TypeError)
service.create_service(_some_invalid_token_)  # this should throw now

[dvu4]

The function I test is create_service in service.py which importing some external dependencies and may cause some unwanted side-effect

import  exclude_dict, service_names, call_api, 
from .models import AK,  SDetails, Token

class ServiceExists(Exception):
    pass

def create_service(
    token: Token,
    long_name: str,
    dn: Optional[str] = None,
    sn: Optional[str] = None,
    e: Optional[str] = None,
    t: Optional[str] = "p",
    d: bool = False
) -> Optional[AK]:

    if dn is None and (sn is None or e is None):
        raise ValueError(
            "You must set the `dn` or you must set both `sn and e`.")

    sp_list = service_names(token)

    if display_name in sp_list:
        raise ServiceExists(f"Service  {dn} exists, please create different name")

    sp_details = SDetails(
        dN=dn,
        tR=f"{long_name}",
        cS="D",
    )

    data = [exclude_dict(sp_details)]
    response = call_api(token, "post", "./service", json=data)
    result = parse_obj_as(AK, response)

    return result

I need to test all the valid inputs, output type is AK , all raise exception and correct message is showed if exception is raised and maybe call_api to see if it executed successfully or not.

[kaste]

Ok, when create_service is the system-under-test then you don't mock it out. You mock its dependencies. Of course create_service is already a complex function and it already has been written before the tests so this is a bit painful.

You look at the dependencies: call_api, service_names, parse_obj_as, exclude_dict. Typically some of them are pure (maybe: parse_obj_as) and could run in a test and some shouldn't really run in a test scenario (probably: call_api). Start with mocking the latter.

In test_create_service.py:

import service as sut   # sut = system-under-test
def test_happy_path(self):
    token, response, result = mock(), mock(), mock()
    when(sut).service_names(token).thenReturn([])
    when(sut).call_api(token, ...).thenReturn(response)   
    when(sut).parse_obj_as(sut.AK, response).thenReturn(result)

    assert sut.create_service(token, "long_name", "dn") == result

Make this test somehow pass.

It gets easier if the first happy path passes.

def test_no_dn_throws(self):
    token, response, result = mock(), mock(), mock()
    when(sut).service_names(token).thenReturn([])
    when(sut).call_api(token, ...).thenReturn(response)   
    when(sut).parse_obj_as(sut.AK, response).thenReturn(result)

    with self.assertRaises(ValueError):
        sut.create_service(token, "long_name")

And from there you test permutations and focus on edge cases and failing inputs. (There is probably already an error in the code as setting dn, sn, and e should throw.)

Does this gets you started?

[dvu4]

I got the error when running test_happy_path(self). Do I need to mock the SDetails and exclude_dict since the output of them will be used as json in call_api ?

Called but not expected:

    call_api(<Dummy id=34323>, 'post', /services', json=[{'app_Other_Type': 'long_name', 'calling_Service': 'DA', 'dN': 'dn'], debug=False)

Stubbed invocations are:

    call_api(<Dummy id=34323>, 'post', '/services', debug=False)

[kaste]

I thought SDetails and exclude_dict are "pure" and you can just fill out the json value manually. Isn't this what the function is about? Taking the arguments, and build a http request from them?

To get you started I actually wrote it with ... (the Ellipsis) -- call_api(token, ...) because in the first test you maybe don't care about the other arguments.

[dvu4]

Thank you for clarifying that for me. I filled out the json value manually and don't care about the arguments (...). I tried to test if the output of service_names is empty, then create a new service .

class AK(BaseModel):
    dM: Any
    code: Optional[int]
    message: str
    status: str

I want to confirm the output of test function, it works.

import service as sut   # sut = system-under-test

def test_happy_path(self):
        when(sut).service_names(...).thenReturn([])
    response = {
        "dM" : "foo",
            "code" : 200,
            "message":  "ok",
            "status" : "200"
            }

    when(sut).call_api(...).thenReturn(response)
        result = AK(**response)
        when(sut).parse_obj_as(AK, response).thenReturn(result)
    assert sut.create_service("token","long_name", "dn") == result

[dvu4]

I also tried to test if exception is raised, the correct message is showed. It works with pytest. Is there anyway I can do the same in mockito ?

class ServiceExists(Exception):
    Service already exists. Cannot create without the --force flag

import service as sut   # sut = system-under-test
import ServiceExists

def test_exception_message_existed_service(self):
        with pytest.raises(ServiceExists) as err:
            with mock.patch('sut.service_names') as patched_function:
                patched_function.return_value = ["ps", "rt", "vc", "ro"]
            sut.create_service("token_foo", "ln_foo", "rt", "sn_foo", "e_foo", "p", False)

    msg = "Service already exists. Cannot create without the --force flag"
    assert msg in str(err.value)
    assert err.type == ServiceExists

[kaste]

I don't see why there should be any difference in the last example:

when(sut).service_names(...).thenReturn(["rt"])
with pytest.raises(ServiceExists) as err:
    sut.create_service("token_foo", "ln_foo", "rt", "sn_foo", "e_foo", "p", False)
# ...

[dvu4]

That works. Thank you !!!

[kaste]

Ideally, parse_obj_as is "pure" and should not be mocked. E.g.:

import service as sut   # sut = system-under-test

def test_happy_path(self):
    token = mock()
    response = {
        "dM": "foo",
        "code": 200,
        "message": "ok",
        "status": "200"
        }
    when(sut).service_names(token).thenReturn([])
    when(sut).call_api(token, ...).thenReturn(response)

    result = AK(**response)
    assert sut.create_service(token, "long_name", "dn") == result

would be more readable. Keep in mind that you don't want ... everywhere, only for stuff you actually don't test here but somewhere else. Typically you test a function which takes some arguments and you take these arguments, maybe transform them, and then pass them to a colaborator. The test should assure that you pass the correct things down.

So as soon as the test is green with ... filled in everywhere you basically know which parts of the code under tests are pure, t.i. don't need to be mocked, and which parts must be mocked. You should (after the happy path) spell out the call_api() call explicitly (t.i. without the ...) for every possible input of the create_service function because it seems that transformation (arguments for create_service -> arguments for call_api) is the main responsibility of create_service. (And it throws for same (token, dN) pairs; such a pair forms a singleton as it seems.)

[dvu4]

When I mock token, I got the error <Dummy id=34323>.

mockito.invocation.InvocationError: 
Called but not expected:

    service_names('token')

Stubbed invocations are:

    service_names(<Dummy id=4335639360>)

I guess because token is the input for system under test so we should not mock it because it is not created by the other dependency. How can I set up for token correctly?

data = {
"token_type": "mock_token", 
"expires_in": 2023, 
"ext_expires_in": "mock_ext_expires_in", 
"access_token": "mock_access_token"
}
token = Token(**data)

[kaste]

Called but not expected:

    service_names('token')
                  ^^^^^^^ a literal string

Stubbed invocations are:

    service_names(<Dummy id=4335639360>)

that basically means you called assert sut.create_service("token", "long_name", "dn") instead of sut.create_service(token, "long_name", "dn"). Instead of creating a mock token you can of course create a real token if it easy to do. (The only plus of using a mock here is that it makes it clear that the passed in token is only used to pass it further down to other functions; it is not examined by the function under test -- t.i. a simple unconfigured mock() is enough to make the test green.)

[dvu4]

That is my mistake when passing a string "token" instead of token. I set up the unconfigured mock() for token and it works

[dvu4]

I try to paginate through all the services created and added new option cursor and limit in call_api(token, "get", "/services/?cursor={cursor}&limit={limit}") so user can see all the services. I do unit test for this new feature in call_api for the default and the overwritten values for limit and cursor

class ASPResponse(BaseModel):
    aSPDetails: List[ASPDetail]
    count: int
    detailedMessage: str
    httpCode: int
    message: str
    status: str
    totalCount: int

class ASPDetail(BaseModel):

I tried to check if the default limit and cursor are correct but got the TypeError: 'AnswerSelector' object is not subscriptable. Is there any way I can check the default and overwritten values like validator in Mockito?

 def test_default_limit_and_cursor(self):
        token = mock()
        path = "/services/?cursor=1&limit=50"
        response = {
            "ASPDetails": [],
            "count": 50,
            "detailedMessage": "foo",
            "httpCode": 200,
            "message": "ok",
            "status": "200",
            "totalCount": 1400}

        when(services).call_api(token, "get", path, ...).thenReturn(response)
        result = ASPResponse(**response)
        default_result = when(service_principals).list_service_principals(token, ...)
        assert default_result["count"] == result["count"]

 def test_overwritten_limit_and_cursor(self):
        token = mock()
        path = "/services/?cursor=3&limit=100"
        response = {
            "ASPDetails": [],
            "count": 100,
            "detailedMessage": "foo",
            "httpCode": 200,
            "message": "ok",
            "status": "200",
            "totalCount": 1400}

        when(services).call_api(token, "get", path, ...).thenReturn(response)
        result = ASPResponse(**response)
        overwritten_result = when(service_principals).list_service_principals(token, ...)
        assert overwritten["count"] == result["count"]

[kaste]

when() returns such an AnswerSelector. You probably want to execute the function under test:

        default_result = service_principals.list_service_principals(token)

More idiomatic test would be, especially the assert would compare something from the result from the function under test with a constant:

        when(services).call_api(token, "get", path, ...).thenReturn(response)
        default_result = service_principals.list_service_principals(token)
        assert default_result["count"] == 100

[dvu4]

yes it works :)

[dvu4]

Hi, I just wonder if I can use Mockito to mock the output of CLI

main.py

import typer
from services import create_service
import output_json
import is_valid, show_message
import sys
app = typer.Typer()

@app.command()
def create(
    ctx: typer.Context,
    ln: str,
    dn: Optional[str] = None,
    sn: Optional[str] = None,
    e: Optional[str] = None,
    g: Optional[str] = "p",
    f: Optional[bool] = False,
) -> None:

    response = create_service(token, ln, dn, sn, e, g, debug=ctx.obj.debug, f=f)
    if is_valid(response):
        output_json(ctx.obj.output, response)
    else:
        typer.echo(show_message(response))
        sys.exit(1)

def output_json(response: Optional[str])
          typer.echo(response.json(indent=2))

Because create only print out the response, so I need to assert the message is correct. I tried to mock context (typer.Context) with context = mock() but got the error AttributeError: 'functools.partial' object has no attribute 'debug'

Is there any way I can mock the attributes ctx.obj.debug or ctx.obj.output from the inner function when building CLI with typer ?

import main
from mockito import when, mock
import contextlib
from io import StringIO

    def test_invalid_response(self):
        token = mock()
        context = mock()
        response = {
            "aSPDetails": [],
            "count": 10,
            "detailedMessage":
                "App_Other_Type: size must be between 0 and 10, Target_Resource: size must be between 1 and 20",
            "httpCode": 400,
            "message": "Please check the values entered for attributes",
            "status": " Failure",
            "totalCount": 40}

        msg = "App_Other_Type: size must be between 0 and 10, Target_Resource: size must be between 1 and 20"
        result = AKS(**response)

        when(main).create_service(token, "too_long_name", ...).thenReturn(result)
        when(main).is_valid(...).thenReturn(False)
        when(main).show_message(...).thenReturn(msg)

        temp_stdout = StringIO()
        with contextlib.redirect_stdout(temp_stdout):
            main.create(context, "too_long_name", "dn", "sn", "e", "p", False)
        output = temp_stdout.getvalue().strip()
        assert output == msg

[kaste]

You need to configure the context mock.

E.g.

obj = mock({"debug": False})
context = mock({"obj": obj})

[dvu4]

Thank you for pointing out. It works :)

[dvu4]

I tried to test output of CLI but my approach seems incorrect when I import main and got the error `

import main
from mockito import when, mock
import contextlib
from io import StringIO

    def test_invalid_response(self):
        token = mock()
        obj = mock({"debug": False})
        context = mock({"obj": obj})
        response = {
            "aSPDetails": [],
            "count": 10,
            "detailedMessage":
                "App_Other_Type: size must be between 0 and 10, Target_Resource: size must be between 1 and 20",
            "httpCode": 400,
            "message": "Please check the values entered for attributes",
            "status": " Failure",
            "totalCount": 40}

        msg = "App_Other_Type: size must be between 0 and 10, Target_Resource: size must be between 1 and 20"
        result = AKS(**response)

        when(main).create_service(token, "too_long_name", ...).thenReturn(result)
        when(main).is_valid(...).thenReturn(False)
        when(main).show_message(...).thenReturn(msg)

        temp_stdout = StringIO()
        with contextlib.redirect_stdout(temp_stdout):
            main.create(context, "too_long_name", "dn", "sn", "e", "p", False)
        output = temp_stdout.getvalue().strip()
        assert output == msg

______________________ ERROR collecting tests/test_cli.py ______________________
tests/test_cli.py:4: in <module>
     import main

To test the output of CLI app with Typer , create a CliRunner and this runner is what will "invoke" or "call" command line application. For example

main.py

import typer

def main(name: str = "World"):
    print(f"Hello {name}")

if __name__ == "__main__":
    typer.run(main)

import typer
from typer.testing import CliRunner

from .main import main

app = typer.Typer()
app.command()(main)

runner = CliRunner()

def test_app():
    result = runner.invoke(app, ["--name", "Camila"])
    assert result.exit_code == 0
    assert "Hello Camila" in result.stdout

I want to test the output of command create from app to check if it prints the correct result.

import typer
from services import create_service
import output_json
import is_valid, show_message
import sys
app = typer.Typer()

@app.command()
def create(
    ctx: typer.Context,
    ln: str,
    dn: Optional[str] = None,
    sn: Optional[str] = None,
    e: Optional[str] = None,
    g: Optional[str] = "p",
    f: Optional[bool] = False,
) -> None:

    response = create_service(token, ln, dn, sn, e, g, debug=ctx.obj.debug, f=f)
    if is_valid(response):
        output_json(ctx.obj.output, response)
    else:
        typer.echo(show_message(response))
        sys.exit(1)

I tried this approach but the runner invokes the app instead of calling main module so I am not sure if I can usemockito in this case.

from unittest import TestCase
from mockito import when, mock, unstub
from .main import app
from typer.testing import CliRunner
runner = CliRunner()

class TestCli(TestCase):
    def test_invalid_response(self):

        token = mock()
        obj = mock({"debug": False})
        context = mock({"obj": obj})
        response = {
            "aSPDetails": [],
            "count": 10,
            "detailedMessage":
                "App_Other_Type: size must be between 0 and 10, Target_Resource: size must be between 1 and 20",
            "httpCode": 400,
            "message": "Please check the values entered for attributes",
            "status": " Failure",
            "totalCount": 40}

        msg = "App_Other_Type: size must be between 0 and 10, Target_Resource: size must be between 1 and 20"
        result = AKS(**response)

        # when(main).create_service(token, "too_long_name", ...).thenReturn(result)
        # when(main).is_valid(...).thenReturn(False)
        # when(main).show_message(...).thenReturn(msg)

       # main.create(context, "too_long_name", "dn", "sn", "e", "p", False)

        result = runner.invoke(app, ["create"], input="too_long_name")
        assert result.exit_code == 0
        assert msg in result.stdout

[kaste]

That is very specific to ask for. And you don't actually write down what throws, what works.

I assume

def test_app():
    result = runner.invoke(app, ["--name", "Camila"])
    assert result.exit_code == 0
    assert "Hello Camila" in result.stdout

this stuff works because it is the example from the documentation. From their you start doing the same thing but make it a subcommand "create" for example. You at least must get a feeling and direction in your work.

Generally, mocking here works as always. It looks like you're always using globals, no dependency injection, always patching the module functions. This should be doable. You've did this before.

[dvu4]

I wonder if I inject dependency create_service, is_valid, show_message in the subcommand create, how will I do that in mockito? like when(app).is_valid(...).thenReturn(False) or when(app.create).is_valid(...).thenReturn(False) because it is only invoked after runner.invoke(app, ["create"], input="too_long_name")

[kaste]

I don't understand your last comment. You still don't show if you can follow the tutorial for Typer. Between the tutorial runner.invoke(app, ["--name", "Camila"]) and the rather huge TestCli.test_invalid_response, well where are the steps inbetween? At what point it starts to fail, or you fail to understand the error messages or general code path.

And what actually is thrown/shown when you run test_invalid_response? Why are all patch calls when(...) commented out?

BTW: In this test you try to replace everything with mocks. I don't think you need to compute a result = AKS(**response) here.

response = mock()
when(main).create_service(token, "too_long_name", ...).thenReturn(response)
when(main).is_valid(response).thenReturn(False)
when(main).show_message(response).thenReturn("MOCKED MESSAGE")
...
assert "MOCKED MESSAGE" in result.stdout

[dvu4]

I think it fails in invoking the subcommand create because the output is empt string.

from mockito import when, mock, unstub
import main
from main import app
from typer.testing import CliRunner

 def test_invalid_response(self):

        runner = CliRunner()
        response = mock()
        when(service_principals).create_service(...).thenReturn(response)
        when(main).is_valid(response).thenReturn(False)
        when(main).show_message(response).thenReturn("MOCKED MESSAGE")

        result = runner.invoke(app, ["create", "too_long_name", "--display-name", "foo"])
        assert result.exit_code == 0
        assert "MOCKED MESSAGE" in result.stdout

And got the error

tests/test_cli.py:39: AssertionError
=========================================================================================================== short test summary info ===========================================================================================================
FAILED tests/test_cli.py::TestCli::test_invalid_response - AssertionError: assert 'MOCKED MESSAGE' in ''

FAILED tests/test_cli.py::TestCli::test_invalid_response - AssertionError: assert 1 == 0

[kaste]

Spooky when(service_principals).create_service(...).thenReturn(response) where is the service_principals coming from? You can just sprinkle some 1/0 in create and see where it throws. (If the exit_code is 1 something failed. Look at stderr.)

[dvu4]

I’m sorry. It should be when(main).create_service(...).thenReturn(response). I wonder when I mock the other dependencies in sub-command ‘create’ and invoke ‘create’ with Typer, it doesn’t create any output

[kaste]

Did you look at stderr as well?

[dvu4]

Yes I set ‘runner = CliRunner(mix_stderr=False)’

[kaste]

Well you don't see the exceptions from the code your running.

result = runner.invoke(app, ["create", "too_long_name", "--display-name", "foo"], catch_exceptions=False)

from their documentation.

Not sure why this isn't the default. 🤷

[dvu4]

I added the exception option and mock context from ctx: typer.Context

 def test_invalid_response(self):
        runner = CliRunner(mix_stderr=False)
        obj = mock({"debug": False})
        context = mock({"obj": obj})
       response = mock()

        when(main).create_service(...).thenReturn(response)
        when(main).is_valid(response).thenReturn(False)
        when(main).show_message(response).thenReturn("msg")

        result = runner.invoke(app,
                               ["create", context, "too_long_name", "--display-name", "foo"],
                               catch_exceptions=False)
        # print(result.stderr)
        assert result.exit_code == 0
        assert "msg" in result.stdout

And got the error

>       result = runner.invoke(app,
                               ["create", context, "too_long_name", "--display-name", "p-prodfix-ds-pfoo-svcp"],
                               catch_exceptions=False)

tests/test_cli.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/typer/testing.py:21: in invoke
    return super().invoke(
../Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/click/testing.py:408: in invoke
    return_value = cli.main(args=args or (), prog_name=prog_name, **extra)
../Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/click/core.py:1053: in main
    rv = self.invoke(ctx)
../Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/click/core.py:1657: in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
../Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/click/core.py:914: in make_context
    self.parse_args(ctx, args)
../Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/click/core.py:1367: in parse_args
    opts, args, param_order = parser.parse_args(args=args)
../Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/click/parser.py:337: in parse_args
    self._process_args_for_options(state)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <click.parser.OptionParser object at 0x105bba230>, state = <click.parser.ParsingState object at 0x105bba1d0>

    def _process_args_for_options(self, state: ParsingState) -> None:
        while state.rargs:
            arg = state.rargs.pop(0)
>           arglen = len(arg)
E           TypeError: object of type 'Dummy' has no len()

../Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/click/parser.py:358: TypeError
======================================================================================== short test summary info ========================================================================================
FAILED tests/test_cli.py::TestCli::test_invalid_response - TypeError: object of type 'Dummy' has no len()

[kaste]

From my understanding you can't pass an object where invoke actually expects strings. (Because on the command line you can't pass in objects as well just strings.)

[dvu4]

If I do not pass an object in invoke, I got the error. So I am not sure how mock will work in the test

E       AttributeError: 'NoneType' object has no attribute 'debug'

src/cli/main.py:31: AttributeError
=========================================================================================================== short test summary info ===========================================================================================================
FAILED tests/test_cli.py::TestCli::test_invalid_response - AttributeError: 'NoneType' object has no attribute 'debug'

[kaste]

Yeah sure, part of what Typer seems to do is that it injects a context object here. You need to find out where this ctx comes from. Why is there an obj on it. Where does the debug (ctx.obj.debug) come from. And how can you modify and control the typer.Context in your tests.

[dvu4]

I think I can test the output of CLI with this method below. config.py

class Settings(BaseSettings):
    client_id: str
    thumbprint: str
    private_key_file: str
    authority: str = "https://example"
    scope: List[str] = ["example"]

    class Config:
        env_file = "~/.env"

And~/.env is like this

PRIVATE_KEY_FILE="key.pem"
THUMBPRINT="grgeerhe"
CLIENT_ID="few"

main.py

import typer
from services import create_service
from config import Settings
import get_token
import output_json
import is_valid, show_message
import sys
app = typer.Typer()

@app.command()
def create(
    ctx: typer.Context,
    ln: str,
    dn: Optional[str] = None,
    sn: Optional[str] = None,
    e: Optional[str] = None,
    g: Optional[str] = "p",
    f: Optional[bool] = False,
) -> None:

    config = Settings()
    token = get_token(config) 
    response = create_service(token, ln, dn, sn, e, g, debug=ctx.obj.debug, f=f)
    if is_valid(response):
        output_json(ctx.obj.output, response)
    else:
        typer.echo(show_message(response))
        sys.exit(1)

In test test_cli.py

import main

    def test_invalid_response(self):

         config = mock({"client_id": "str", "thumbprint": "str", "private_key_file": "str"})
        access_token = mock()

        obj = mock({"debug": False})
        context = mock({"obj": obj})
        response = mock()
        msg = "App_Other_Type: size must be between 0 and 10, Target_Resource: size must be between 1 and 20"

        when(main).Settings().thenReturn(config)
        when(main).get_token(config).thenReturn(access_token)
        when(main).create_service(...).thenReturn(response)
        when(main).is_valid(response).thenReturn(False)
        when(main).show_message(response).thenReturn(msg)

        out = StringIO()
        sys.stdout = out
        main.create(context, "too_long_name", "display_name", "short_name", "env", "p", False)
        output = out.getvalue().strip()
        assert output == msg
        unstub()

But I got pipeline issue when main is imported. it will read config.py and I tried to mock it but did not successful. Is there any way I can bypass client_id, thumbprint and private_key_file

ERROR: test_invalid_response (test_cli.TestCLI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/tests/test_cli.py", line 18, in test_invalid_response
    when(main).Settings().thenReturn(config)
  File "/Users/Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/mockito/invocation.py", line 342, in __call__
    self.ensure_signature_matches(
  File "/Users/Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/mockito/invocation.py", line 337, in ensure_signature_matches
    signature.match_signature_allowing_placeholders(sig, args, kwargs)
  File "/Users/Library/Caches/pypoetry/virtualenvs/lib/python3.10/site-packages/mockito/signature.py", line 115, in match_signature_allowing_placeholders
    sig.bind(*args, **kwargs)
  File "/opt/homebrew/Cellar/python@3.10/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/inspect.py", line 3179, in bind
    return self._bind(args, kwargs)
  File "/opt/homebrew/Cellar/python@3.10/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/inspect.py", line 3149, in _bind
    raise TypeError('missing a required argument: {arg!r}'. \
TypeError: missing a required argument: 'client_id'

----------------------------------------------------------------------
Ran 1 test in 0.027s

FAILED (errors=1)

[kaste]

I don't follow. You now wrote the command differently. Why?

result = runner.invoke(app,
                               ["create", context, "too_long_name", "--display-name", "foo"],
                               obj=mock({"debug": False}, catch_exceptions=False)

Isn't that how you inject a obj part of ctx into Typer?

In your app command the first line is:

config = Settings()

Is that valid? Without any arguments? mockito, actually python's inspect module is used here, thinks you need to pass in arguments. What's the purpose of this?

Even if you mock with when..Settings(...).then... the code will fail because it still thinks Settings() is an invalid call.

[dvu4]

Settings()takes Environmental variables in .env formatted file (client_id, thumbprint, private_key_file) as arguments so I want to mock them but am not sure how to

class Settings(BaseSettings):
    client_id: str
    thumbprint: str
    private_key_file: str
    authority: str = "https://example"
    scope: List[str] = ["example"]

    class Config:
        env_file = "~/.env"

I have an pipeline issue when doing PR

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_cli.py ______________________
tests/test_cli.py:3: in <module>
     import main
    from .config import CONFIG
pydantic/env_settings.py:38: in pydantic.env_settings.BaseSettings.__init__
    ???
pydantic/main.py:331: in pydantic.main.BaseModel.__init__
    ???
E   pydantic.error_wrappers.ValidationError: 3 validation errors for Settings
E   client_id
E     field required (type=value_error.missing)
E   thumbprint
E     field required (type=value_error.missing)
E   private_key_file
E     field required (type=value_error.missing)

[kaste]

I don't understand what you're after here. I think the command code is invalid as it has a plain Settings() in it.

[dvu4]

I just updated Settings() which reads client_id, thumbprint, private_key_file from env_file = "~/.env" And~/.env is like this

PRIVATE_KEY_FILE="key.pem"
THUMBPRINT="grgeerhe"
CLIENT_ID="few"

My guess is that when I create PR, it could not find the env file and I want to find a way to bypass it

unit test passed in my local machine but got the error on Azure Pipeline

self = <test_cli.TestCLI testMethod=test_invalid_response>

    def test_invalid_response(self):
        response = mock()
        msg = "App_Other_Type: size must be between 0 and 10, Target_Resource: size must be between 1 and 20"

        when(main).create_service(...).thenReturn(response)
        when(main).is_valid(response).thenReturn(False)
        when(main).show_message(response).thenReturn(msg)

>       result = runner.invoke(app,
                               ["create", "too_long_name", "--display-name", "foo"],
                               obj=mock({"debug": False}),
                               catch_exceptions=False)

tests/test_cli.py:18: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.venv/lib/python3.9/site-packages/typer/testing.py:21: in invoke
    return super().invoke(
.venv/lib/python3.9/site-packages/click/testing.py:408: in invoke
    return_value = cli.main(args=args or (), prog_name=prog_name, **extra)
.venv/lib/python3.9/site-packages/click/core.py:1053: in main
    rv = self.invoke(ctx)
.venv/lib/python3.9/site-packages/click/core.py:1659: in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
.venv/lib/python3.9/site-packages/click/core.py:1395: in invoke
    return ctx.invoke(self.callback, **ctx.params)
.venv/lib/python3.9/site-packages/click/core.py:754: in invoke
    return __callback(*args, **kwargs)
.venv/lib/python3.9/site-packages/typer/main.py:500: in wrapper
    return callback(**use_params)  # type: ignore
main.py:28: in create
    config = Settings()
pydantic/env_settings.py:38: in pydantic.env_settings.BaseSettings.__init__
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pydantic.error_wrappers.ValidationError: 3 validation errors for Settings
E   client_id
E     field required (type=value_error.missing)
E   thumbprint
E     field required (type=value_error.missing)
E   private_key_file
E     field required (type=value_error.missing)

[kaste]

Are you sure you know what you're doing here? 😁

Settings() is throwing because its parent BaseSettings.__init__ requires arguments. That's what I'm reading here. Where do you updated Settings() so it reads from some file? And how is this related to mockito?

[dvu4]

class Settings(BaseSettings):
    client_id: str
    thumbprint: str
    private_key_file: str
    authority: str = "https://example"
    scope: List[str] = ["example"]

    class Config:
        env_file = "~/.env"

And~/.env is like this

PRIVATE_KEY_FILE="key.pem"
THUMBPRINT="grgeerhe"
CLIENT_ID="few"

so when I call config = Settings(), config will be updated from .env file. I hope there are some way I can mock client_id, thumbprint, private_key_file without reading .env file because there is no .env file at pipeline job level in azure devops

[kaste]

So you want to use a mock in production? Actually it looks like it's throwing on your computer and not on azure devops.

[dvu4]

mock is passed on my computer and the error is throwing on azure devops. I wonder if I can use mock in production.

[kaste]

I don't think you should use mock in production, that's not a good design. It still looks like it is not in production though. The tests fail when they run on azure.

It feels brittle to have a plain Settings() here (without any args) work in production because it in the end magically reads from somewhere. I think I would put them on the context which I then inject in the tests.

You can always do

when(main, strict=False).Settings().thenReturn(config)

to suppress the signature check (which gives you the TypeError).

I don't know why you put this call in the command. Settings is probably a singleton, so naturally you initiate it once and put it on module level ("global" but simple) and because that usually stinks you call it once and put it on the context. But you do and then strict=False might be what you're looking for 🤷

[dvu4]

I put Settings in the command because I need to get the access_token to create the service. I also tried to initiate it in "global" but the error is throwing as soon as I import main, so calling it inside command, I can mock it and avoid the error on Azure.

Adding strict=False finally works because the error is not throwing any more 😄