pulumi.runtime.Mocks::call works only partly

[zhukovgreen]

What happened?

When use pulumi.runtime.set_mocks(mock), and mock overrides call method, which returns some dict, then not all fields of this dict are reflected in the output object.

Steps to reproduce

import pulumi

from pulumi_aws import ec2

class MockPulumiResources(pulumi.runtime.Mocks):
    def new_resource(self, args: pulumi.runtime.MockResourceArgs):
        return args.name + "_id", args.inputs

    def call(self, args: pulumi.runtime.MockCallArgs):
        if args.token == "aws:ec2/getSubnet:getSubnet":
            return {
                "arn": "arn:aws:ec2:us-east-2:495555555796:subnet/subnet-01e95555555070ee1",
                "assign_ipv6_address_on_creation": False,
                "availability_zone": "us-east-2a",
                "availability_zone_id": "use2-az1",
                "available_ip_address_count": 10,
                "cidr_block": "155.55.5.5/26",
                "customer_owned_ipv4_pool": "",
                "default_for_az": False,
                "enable_dns64": False,
                "enable_resource_name_dns_a_record_on_launch": False,
                "enable_resource_name_dns_aaaa_record_on_launch": False,
                "filters": [
                    {
                        "name": "tag:Name",
                        "values": ["DataEngineering-Dev Subnet A"],
                    }
                ],
                "id": "subnet-01555555555070ee1",
                "ipv6_cidr_block": "",
                "ipv6_cidr_block_association_id": "",
                "ipv6_native": False,
                "map_customer_owned_ip_on_launch": False,
                "map_public_ip_on_launch": False,
                "outpost_arn": "",
                "owner_id": "495555555796",
                "private_dns_hostname_type_on_launch": "ip-name",
                "state": "available",
                "tags": {
                    "application-group": "Reporting",
                    "environment": "DEV",
                    "Name": "DataEngineering-Dev Subnet A",
                    "domain": "DataEngineering",
                    "environment-group": "DEV",
                },
                "vpc_id": "vpc-019cf555555555552",
            }
        raise NotImplementedError(f"No stub for {args.token}")

pulumi.runtime.set_mocks(MockPulumiResources())

subnet = ec2.get_subnet(id="some_id_001")

@pulumi.runtime.test
def test_example_not_working():
    def inner(args):
        assert args[0]

    return pulumi.Output.all(subnet.availability_zone).apply(inner)

@pulumi.runtime.test
def test_example_working():
    def inner(args):
        assert args[0]

    return pulumi.Output.all(subnet.arn).apply(inner)

Expected Behavior

test_example_not_working should pass

Actual Behavior

test_mvp.py::test_example_not_working FAILED
tests/test_mvp.py:71 (test_example_not_working)
Traceback (most recent call last):
  File "/Users/azhukov/Documents/paylocity/dataplatform.reportingstore-migration/tests/test_mvp.py", line 75, in inner
    assert args[0]
AssertionError: assert None

Output of pulumi about

CLI          
Version      3.43.0
Go Version   go1.19.2
Go Compiler  gc

Plugins
NAME    VERSION
aws     5.17.0
python  unknown

Host     
OS       darwin
Version  12.6
Arch     arm64

This project is written in python: executable='/Users/username/Documents/companyname/projectname/.venv/bin/python3' version='3.10.6
'

Current Stack: some-stack-name

Found no resources associated with some-stack-name
Found no pending operations associated with some-stack-name

Backend        
Name           my-pc-name
URL            s3://...
User           username
Organizations  

Dependencies:
NAME                                   VERSION
black                                  22.10.0
my-repo                           1.0.0
env                                    0.1.0
isort                                  5.10.1
mypy                                   0.982.0
pre-commit                             2.20.0
pytest-asyncio                         0.19.0
pytest-cov                             4.0.0
pyupgrade                              3.1.0
ruff                                   0.0.75
types-protobuf                         3.20.4
types-PyYAML                           6.0.12
types-six                              1.16.21

Additional context

Seems like this issue causing https://github.com/pulumi/pulumi/discussions/11016

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[RobbieMcKinstry]

Hi @zhukovgreen

Ah, interesting. I'm able to reproduce locally. I'm seeing a suspicious warning message:

/site-packages/pulumi/runtime/sync_await.py:94: DeprecationWarning: There is no current event loop
  loop = asyncio.get_event_loop()

I wonder if there's a race condition on reading the second value in an async context, though I suppose there shouldn't be, since Output.all should wait until all futures resolve.

[zhukovgreen]

Hi @RobbieMcKinstry this issue should be addressed for python3.11 (see https://stackoverflow.com/questions/70303895/python-3-10-asyncio-gather-shows-deprecationwarning-there-is-no-current-event). But it seems it is not the reason of the current issue

[RobbieMcKinstry]

@zhukovgreen after a little digging, we discovered that the mocks are expected to return data in the format of the AWS response, not in the format of the Python object.

Both tests will pass if you change:

 "availability_zone": "us-east-2a",

to...

 "availabilityZone": "us-east-2a",

You can see in the source code that the decorator name maps to the expected field. I'm unsure how else you could figure out the name you need to use in the mock. This may be a bug in the SDK. I will pass this issue along to the team that authors the AWS provider and ask them if it's intentional or not. But in the meantime, you can use this as a workaround.

[zhukovgreen]

Thank you @RobbieMcKinstry . Do you think it would be a good idea to raise an exception on not existing attributes? I'd try to contribute to this feature.

[RobbieMcKinstry]

Hi @zhukovgreen I'm not sure! We're talking internally about finding a way to correct this mapping so it's less confusing. I'm not sure if we want to introduce a new Exception, though. I'm leaning towards "yes".