search

cloudtools / troposphere

troposphere - Python library to create AWS CloudFormation descriptions
BSD 2-Clause "Simplified" License
4.92k stars 1.45k forks source link

Update glue.py with SNOWFLAKE validate type #2247

Closed evilensky closed 1 month ago

evilensky commented 1 month ago

The SNOWFLAKE validation type allows an AWS::Glue::Connection with the ConnectionType of SNOWFLAKE to be created.

This is not yet in any documentation nor is the underlying API documented :(

https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1810

ewascent commented 1 month ago

I think you need to add a target class in glue.py (e.g. JDBCTarget or MongoDBTarget) and add that to the list of targets in the Targets class dict.

For this connection I do not think this needs a default port in constants.py. Snowflake has a default port of 443. We both have a default https_port value, and the default of the Snowflake connection properties assume the port is 443 here

markpeek commented 1 month ago

Thank you for the PR. As @ewascent mentions, there are likely more changes needed to fully support Snowflake. With this change were you able to bring up a valid CloudFormation stack with Snowflake?

evilensky commented 1 month ago

Thank you for considering the PR.

We aren't currently using Crawler or Target so I'm unable to verify at this time.

and just like the SNOWFLAKE connection type, I cannot find documentation for SNOWFLAKE support in these objects either in Cloudformation or underlying API documentation, only this time there is no Roadmap card for them.

Right now I'm monkey patching the Validator with this change, and yes, it creates a snowflake Connection successfully.

image

There are some interesting internal validators in the Glue API: 

SparkProperties.sfUrl: does not match the regex pattern .+[.]snowflakecomputing[.]com (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException; Request ID: 79f731f2-89f7-4f04-b32f-9eda6b6f3e02; Proxy: null)

I'm curious if there is a pattern for troposphere to validate this since the API type is String but the string must be of a Json of a particular shape, and some validation rules apply to the values.

markpeek commented 1 month ago

I'm going to pull in this change since it doesn't really hurt and your screenshot shows it seems to work.

To your validation question, for validation of interconnected properties you would want to create a "class validator" for the object in validators/glue.py. I'll highlight this easy example of the EC2 Subnet class validator here that ensures Ipv6CidrBlock is set if AssignIpv6AddressOnCreation is used in the object.