Sketch out a platform API

[joeduffy]

We agreed that in 0.4 we'd try to sketch out the broad brush strokes of the API so that we can generate additional work items and investigations. This work item tracks doing this.

FWIW, here is a very old exercise I did to explore what cross-sections might look like. I realize we are in a very different place these days, but it still could come in handy as a reference:

Platform Ideas

A brainstorm of what categories of services we might offer:

• Apps
  • Containers
  • Logging
  • Queueing
  • Pub/sub
  • RPC
  • Service discovery
  • Serverless / API Gateway
• Data
  • Cache
  • Blob Store
  • Key/value Store
  • SQL Database
  • NoSQL Database
  • NoSQL Data Warehouse
  • Secret Store
• Services
  • Email
  • SMS
  • Search
  • Job Scheduling
  • Workflow
  • MapReduce
• Infra Services
  • DNS
  • Load Balancing
  • CDN
  • Container Registry

Cloud Catalog

Here's an exhaustive list of services offered by "the big three," and an attempt to correlate them.

COMPUTE	AWS	Azure	Google
IaaS	EC2	Virtual Machines	Compute Engine
Container Registry	EC2 Container Registry		Container Registry
Container Service	EC2 Container Service	Container Service	Container Engine
PaaS	Elastic Beanstalk	Cloud Services / Service Fabric	App Engine
Serverless	Lambda	Functions / WebJobs	Cloud Functions
Job Scheduling		Scheduler / Batch	Compute Engine Tasks
Queueing	Simple Queueing Service (SQS)	Queue Storage / ServiceBus
Workflow	Simple Workflow Service (SWS)	LogicApps

STORAGE	AWS	Azure	Google
Blob Storage	S3	Storage	Cloud Storage (Standard)
Low-Cost Archival	Glacier		Cloud Storage (Nearline)
Mountable Storage	Elastic File System	File Storage
Data Import/Export	Snowball	Import/Export
On-Prem-to-Cloud	Storage Gateway	StorSimple
Secrets	Key Management Service (KMS)	Key Vault

DATABASE	AWS	Azure	Google
Database (SQL)	Relational DB Service (RDS)	SQL Database	Cloud SQL
Database (NoSQL)	DynamoDB	Document DB / Table Storage	Cloud Bigtable / Cloud Datastore
Cache	ElastiCache	Managed Cache / Redis Cache
Data Warehouse (SQL)	Redshift	SQL Data Warehouse
Data Warehouse (NoSQL)		Data Lake Store	BigQuery
Data Migration	DB Migration Service (DMS)	SQL DB Migration Wizard

NETWORKING	AWS	Azure	Google
Load Balancer	EC2 Load Balancer	Load Balancer	Cloud Load Balancing
CDN	CloudFront	Azure CDN	Cloud CDN
Network Mgmt	Virtual Private Cloud (VPC)	Virtual Network	Cloud Virtual Network
VPN	Direct Connect	ExpressRoute	Cloud Interconnect
DNS	Route 53	DNS	Cloud DNS

DEVELOPER TOOLS	AWS	Azure	Google
Git Hosting	CodeCommit	VSTS	Cloud Source Repositories
C/I	C/D	CodeDeploy	VSTS
C/I	C/D Workflow	CodePipeline

MANAGEMENT TOOLS	AWS	Azure	Google
E2E Management	CloudWatch		Stackdriver
Auditing	CloudTrail
Monitoring	-	VS AppInsights	Monitoring
Logging	-	Log Analytics	Logging
Error Reporting	-		Error Reporting
Perf Tracing	-		Trace
Debugging	-		Debugger
Mgmt Templates	CloudFormation		Deployment Manager
Governance	Config
Ops	OpsWorks (Chef)	Resource Manager / Automation
Security Templates	Service Catalog
Service Optimization	Trusted Advisor

SECURITY & IDENTITY	AWS	Azure	Google
Identity & Access	Identity & Access Mgmt		Cloud IAM
LDAP / AD	Directory Service (AD)	Active Directory	Cloud Resource Manager
Security Analysis	Inspector	Security Center	Cloud Security Scanner
DoS/Malicious Guards	WAF
SSL/TLS Cert Mgmt	Certificate Manager

BIG DATA / ANALYTICS	AWS	Azure	Google
MapReduce	Elastic MapReduce (Hadoop)	HDInsight	Cloud Dataproc
Data Processing	Data Pipeline	Data Factory
Search	Elasticsearch Service	Search
Streams Processing	Kinesis	Stream Analytics	Cloud Dataflow
Data Exploration	-	PowerBI / Data Lake Analytics	Cloud Datalab
Pub/Sub/Push Notify	Simple Notif. Service (SNS)	Notif Hub Topics / Event Hubs	Cloud Pub/Sub
Big Science	-		Cloud Genomics

MACHINE LEARNING	AWS	Azure	Google
ML Platform	Machine Learning	Machine Learning	Cloud ML Platform
ML/AI Services		Cognitive Services	Vision, Speech, NL, Translate

INTERNET OF THINGS	AWS	Azure	Google
IoT	IoT	IoT Hub	IoT

GAME DEVELOPMENT	AWS	Azure	Google
Games	GameLift

MOBILE SERVICES	AWS	Azure	Google
Mobile E2E	Mobile Hub
Mobile Auth	Cognito	Multifactor Auth
Mobile Testing	Device Farm	DevTest Labs	Cloud Test Lab
Mobile Analytics	Mobile Analytics	HockeyApp

APP SERVICES	AWS	Azure	Google
API Gateway	API Gateway	API Management	Google Cloud Endpoints
Remote Desktop	AppStream		RemoteApp
Search	CloudSearch	Search
Media Transcoding	Elastic Transcoder	Media Services
Email	Simple Email Service (SES)

ENTERPRISE APPS	AWS	Azure	Google
Remote Desktop	WorkSpaces
Document Sharing	WorkDocs
Office (Email/Calendar)	WorkMail

[lukehoban]

Here's an initial proposal for prioritization of core Pulumi platform features.

Provider	API	AWS Implementation	Priority
Pulumi Platform	ES6 Arrows	Lambda Function	1
Pulumi Platform	Table	DynamoDB Table	1
Pulumi Platform	EventHub	SNS Topic	1
Pulumi Platform	HttpAPI	APIGateway RestAPI	1
Pulumi Platform	ES6 globals	None	1
Pulumi Platform	Cron	Cloudwatch Events ScheduledEvent	2
Pulumi Platform	Bucket	S3 Bucket	1
Pulumi Platform	Email	SES	3
Pulumi Platform	Logs	CloudWatch LogGroup	2
Pulumi Platform	Container	ECS Task	3
Pulumi Platform	Cache	ElastiCache	4
Pulumi Platform	SQLDB	Aurora	4
Pulumi Platform	Search	Elasticsearch Service	3
Pulumi Platform	Instance	EC2 Instance	4
Pulumi Platform	FileSystem	EFS FileSystem	5

We expect to adjust priorities as we make progress, but feedback on anything P1 or P2 that is not on this list would be good to get early.

[lukehoban]

A pulumi.d.ts covering the P1 items in the table above is available at https://github.com/pulumi/lumi-platform/commits/master/doc/pulumi.d.ts for review.

[lukehoban]

This is done for now. We'll track the additional updates and new APIs as separate issues.