Properties of maps get mangled in translation from Pulumi to TF provider

[lukehoban]

From @mmdriley on September 23, 2017 0:0

As CloudFormation's docs explain, the Name tag on the VPC resource has special semantics:

Tags An arbitrary set of tags (key–value pairs) for this VPC. To name a VPC resource, specify a value for the Name key.

When trying to create a VPC with Pulumi:

let vpc = new aws.ec2.Vpc("vpc", {
    cidrBlock: "10.0.0.0/16",
    tags: {
        Name: 'Pulumi VPC',
    },
})

We actually end up creating:

$ aws ec2 describe-vpcs --vpc-id vpc-8aa4cbec --query 'Vpcs[].Tags'
[
    [
        {
            "Value": "Pulumi VPC", 
            "Key": "name"
        }
    ]
]

Notice the lowercase name. Unfortunately this name doesn't appear in the VPC console:

Here's the corresponding console output, which corresponds to what we see above:

$ pulumi plan
Planning changes:
+ aws:ec2/vpc:Vpc: (create)
      [urn=urn:pulumi:pulumi-service::pulumi-service::aws:ec2/vpc:Vpc::vpc]
      assignGeneratedIpv6CidrBlock: false
      cidrBlock                   : "10.0.0.0/16"
      enableDnsHostnames          : true
      enableDnsSupport            : true
      tags                        : {
          Name: "Pulumi VPC"
      }
info: 1 change planned:
    + 1 resource to create

$ pulumi deploy
Deploying changes:
+ aws:ec2/vpc:Vpc: (create)
      [urn=urn:pulumi:pulumi-service::pulumi-service::aws:ec2/vpc:Vpc::vpc]
      assignGeneratedIpv6CidrBlock: false
      cidrBlock                   : "10.0.0.0/16"
      enableDnsHostnames          : true
      enableDnsSupport            : true
      tags                        : {
          Name: "Pulumi VPC"
      }
      ---outputs:---
      assignGeneratedIpv6CidrBlock: false
      defaultNetworkAclId         : "acl-30555456"
      defaultRouteTableId         : "rtb-7ee63307"
      defaultSecurityGroupId      : "sg-d3c700ae"
      dhcpOptionsId               : "dopt-0b37a96c"
      enableClassiclink           : false
      enableClassiclinkDnsSupport : false
      enableDnsSupport            : true
      id                          : "vpc-8aa4cbec"
      instanceTenancy             : "default"
      mainRouteTableId            : "rtb-7ee63307"
      tags                        : {
          name: "Pulumi VPC"
      }
info: 1 change deployed:
    + 1 resource created
Deployment duration: 4.898564607s

Copied from original issue: pulumi/pulumi#363

[lukehoban]

From @joeduffy on September 23, 2017 5:33

Good issue! And thank you for the thorough details. I think we can solve this with an overlay in the AWS package, but I will take a close look in the morning.

[lukehoban]

From @joeduffy on September 23, 2017 12:37

Actually, I think this is a deeper issue (in the bridge). Right now we unconditionally transform property names deeply throughout the object so that code can use "idiomatic" JavaScript names. But this should only apply to object properties, and not the keys within maps. Because this is JavaScript, however, these look the same to the provider. I think we'll need to use translation that is sensitive to the destination type; in other words, don't mangle elements of a schema.TypeMap.

[lukehoban]

Hit another case of this with the configuration property on CodePipeline: https://www.terraform.io/docs/providers/aws/r/codepipeline.html#configuration

Here, the input:

configuration: {
    S3Bucket: bucket.id,
    S3ObjectKey: "source.zip",
},

produces an error about the snake_cased s3_bucket being not allowed:

error PU2003: Plan apply failed: rpc error: code = Unknown desc = [ERROR] Error creating CodePipeline: InvalidActionDeclarationException: Action configuration for action 'App' contains unknown configuration 's3_bucket'
    status code: 400, request id: ad25f023-a7a0-11e7-97d0-ff721b58e353

Unfortunately, for cases like this there is no workaround - so putting this in 0.7.

[joeduffy]

Assigning to myself, and I will definitely get to this this week. I started looking and it won't be terribly difficult, since we can key the decision of whether to auto-case based on whether it's a TF map or not (something we know dynamically).

[lukehoban]

Looks like this didn't fix the case I was hitting.

The resource I am using is here: https://github.com/terraform-providers/terraform-provider-aws/blob/b8d4e1570fc43a2acee6b6e47f63c9db6b067fa2/aws/resource_aws_codepipeline.go#L92

This code in pulumi/terraform appears to be the culprit:

var etfs *schema.Schema
if tfs != nil {
    if sch, issch := tfs.Elem.(*schema.Schema); issch {
        etfs = sch
    } 
}

For the resource I'm using, tfs.Elem is a *schema.Resource, not a *schema.Schema. So this code sets tfs to nil all the way down the walk, ignoring all the rest of the type-direct name mangling decisions.

[lukehoban]

A beta customer hit a case of this again:

I ran into something strange with tagging, it very well could be me being a noob with JS though. I created a bunch of default tags but when applied the casing and naming was changed to include underscores. let default_tags = {Environment: "Dev", Creator: "Brad", DummyTag: "Val1", DummyTag2: "Val2"};

[mmdriley]

with this program:

import * as aws from "@pulumi/aws";

let x = new aws.ec2.SecurityGroup("test1", {
    tags: {
        One: "1",
        Two: "2",
        ThreeFourAndFive: "345",
    }
});

I see this diff:

$ pulumi preview --diff
Previewing changes:
* pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:abcde::pulumi-template::pulumi:pulumi:Stack::pulumi-template-abcde]
    + aws:ec2/securityGroup:SecurityGroup: (create)
        [urn=urn:pulumi:abcde::pulumi-template::aws:ec2/securityGroup:SecurityGroup::test1]
        description        : "Managed by Pulumi"
        name               : "test1-fa0bab9"
        revokeRulesOnDelete: false
        tags               : {
            One             : "1"
            ThreeFourAndFive: "345"
            Two             : "2"
        }
info: 1 change previewed:
    + 1 resource to create
      1 resource unchanged

and performing the update creates this security group:

[mmdriley]

Reached out to Brad to see if I can get more info on how default_tags ended up flowing into a resource.

[mmdriley]

Here's the code:

"use strict";

const aws = require("@pulumi/aws");

let tag = {};
let default_tags = {Environment: "Dev", Creator: "Brad", DummyTag: "Val1", DummyTag2: "Val2"};

let size = "t2.micro";    
let ami  = "";

let group = new aws.ec2.SecurityGroup("webserver-secgrp", {
   ingress: [
       { protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
   ],
   tags: [
       default_tags,
       {RandomTag: "Foo"},
   ],
});

let server = new aws.ec2.Instance("webserver-www", {
   instanceType: size,
   securityGroups: [ group.name ], // reference the security group resource above
   ami: ami,
});

exports.publicIp = server.publicIp;
exports.publicDns = server.publicDns;

Notably: tags are passed as an array of objects, and then somehow that works...

[mmdriley]

Using plain Javascript:

let x = new aws.ec2.SecurityGroup("test1", {
    tags: [
        {
            One: "1",
            Two: "2",
            ThreeFourAndFive: "345",
        },
        {
            Six: "6",
        },
    ],
});

preview:

Previewing changes:
+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:abcde::pulumi-template::pulumi:pulumi:Stack::pulumi-template-abcde]
    + aws:ec2/securityGroup:SecurityGroup: (create)
        [urn=urn:pulumi:abcde::pulumi-template::aws:ec2/securityGroup:SecurityGroup::test1]
        description        : "Managed by Pulumi"
        name               : "test1-7b162b1"
        revokeRulesOnDelete: false
        tags               : [
            [0]: {
                one             : "1"
                threeFourAndFive: "345"
                two             : "2"
            }
            [1]: {
                six: "6"
            }
        ]
info: 2 changes previewed:
    + 2 resources to create

created object:

[mmdriley]

Opened #147 for the easy type-confusion issue. Closing this since the original bug was fixed.

[omidraha]

I have the same issue, the name value is empty in the VPC aws console panel:

import pulumi_aws

# VPC
def create_vpc():
    log.info('[base.vpc.create_vpc]')
    vpc = pulumi_aws.ec2.Vpc(
        "vpc-st",
        tags={"name": "vpc-st-tag"},
        cidr_block='12.0.0.0/16',
    )
    return vpc

Update:

Fixed with upper casing the name:

tags={"Name": "vpc-st-tag"},