[ECS] ECS Development in IntelliJ, PyCharm, and Visual Studio Code

[cbbarclay]

Be able to create, test, and deploy an ECS application in the IDE. Create a new application that includes source code, Dockerfile, and a CloudFormation template that defines your application's resources including an ECS task definition and ECS service. Test your application in the IDE’s debugger with AWS cloud dependencies like a container credentials endpoint, a task metadata endpoint, and task definition attributes like environment variables and secrets. Deploy your application directly through the IDE or with a continuous delivery pipeline.

[talawahtech]

One quick request I have is that IntelliJ support should not just include templates and debugging support for Java/Python applications but also NodeJS (and maybe even PHP). Even though IntelliJ started out as a Java IDE, the community version supports development in Python as well and the commercial (Ultimate) editions supports a large number of languages including NodeJS and PHP.

The current AWS Toolkit for IntelliJ which supports serverless development using SAM seems to only focus on Java and Python development with no support for NodeJS.

[garrardkitchen]

Could the scope be extended to include the Rider IDE with .NET Core debugging for both C# & F#?

[clareliguori]

@garrardkitchen Are you developing Linux or Windows containers?

[garrardkitchen]

Hi @clareliguori, we develop Linux containers to run on ECS Fargate.

[nicwise]

Could it be in the form of a IntelliJ plugin - ie, for all the platforms? We use a terraform one across Pycharm, Webstorm (node) and Rider (C#). They have slightly different project management structures the (eg Rider has a project file, the others just have a folder). Would work for Goland too (Go) etc.

I've always shied away from IDE-driven development like this, mostly since Visual Studio did a GREAT job of it in the past, but for doing actual production work: oh hells no. It instilled a lot of bad habits in a lot of people. Good to play around with in development tho, to get started.

for platforms, we use ECS, with linux containers, using Rider mostly (for our platform) or Pycharm (for SRE, where I am). Lambda or ECS debugging on either of those would be mint.

[webframp]

If there’s any way to implement it as an lsp according to https://github.com/Microsoft/language-server-protocol that might make it more generically useful to different editors and IDEs

[virajpadte]

Updating task definition, ability to deploy to test cluster through dev pipeline and looking at ECS container logs in the IDE might be really cool features.

[kiiadi]

@talawahtech / @nicwise / @garrardkitchen - we are definitely looking at extending the JetBrains plugin to support other languages/JetBrains IDE flavours. Please help us prioritize by +1-ing the various tracking issues in our repo:

• Kotlin: https://github.com/aws/aws-toolkit-jetbrains/issues/142
• NodeJS / WebStorm: https://github.com/aws/aws-toolkit-jetbrains/issues/187
• Go / GoLand: https://github.com/aws/aws-toolkit-jetbrains/issues/649
• Ruby / RubyMine: https://github.com/aws/aws-toolkit-jetbrains/issues/682
• .NET / Rider: https://github.com/aws/aws-toolkit-jetbrains/issues/683
• PHP / PHPStorm: https://github.com/aws/aws-toolkit-jetbrains/issues/847

[garrardkitchen]

Thanks @kiiadi, I have now +1 WebStorm and Rider.

[talawahtech]

@kiiadi

Done. Can you add PHPStorm to the list as well please: https://github.com/aws/aws-toolkit-jetbrains/issues/847

[kiiadi]

@talawahtech have added it to my list above.

[gholt-aws]

I've always shied away from IDE-driven development like this, mostly since Visual Studio did a GREAT job of it in the past, but for doing actual production work: oh hells no. It instilled a lot of bad habits in a lot of people. Good to play around with in development tho, to get started.

@nicwise , I'm curious about this. What kinds of activities (in IDEs) encouraged bad habits?

One of our goals is to promote best practices. That said, we want to be careful not to "get in the way" - for example if customers are prototyping (where deployment from the IDE is helpful). Any feedback or suggestions you have here would be helpful.

[gholt-aws]

A question for the thread... which platforms do you run your IDEs on?

[talawahtech]

@gholt-aws Windows 10

[garrardkitchen]

@gholt-aws macOS

[nicwise]

@gholt-aws

What kinds of activities (in IDEs) encouraged bad habits?

One of our goals is to promote best practices. That said, we want to be careful not to "get in the way" - for example if customers are prototyping (where deployment from the IDE is helpful). Any feedback or suggestions you have here would be helpful.

I've seen people refuse to learn how to use SVN (or Git), and just use what VS provided. "oh, we can't branch cos merging in VS is too hard". 🤦‍♂

Unreproducible production deployments from a developers IDE, basically people being so dependant on the IDE, they could do literally nothing without it.

Personally, I do like using an IDE for a lot of things, but I also use the command line and other tools, esp for things like Git. It is handy for prototyping, I agree, but to some degree, I'd perfer an IDE plug in to do less and guide people into something which could be reproducible and scriptable, than hide it all away.

[kiiadi]

@nicwise you've hit the nail on the head! That's exactly what we're trying to do with the AWS IDE toolkits - enable capabilities that help you through your development process but then steer you towards best practice. The majority of the server-less functionality (e.g. new project, deploy, run local etc) in the new JetBrains / VS Code toolkits is a thin wrapper on-top of the SAM CLI.

I suspect the approach we take with containers would be the same - we want customers to be able to have consistent reproducible processes both in an IDE and in their terminal or CI/CD pipelines.

+1 for using git at the command-line!

[ADGrant]

I would like to see AWS support for Rider.

[cbbarclay]

@ADGrant please +1 https://github.com/aws/aws-toolkit-jetbrains/issues/683

[clareliguori]

The AWS Toolkit for IntelliJ, PyCharm, Rider, and WebStorm now support Cloud Debugging (in beta) for ECS services running on AWS Fargate. You can list your ECS services in the IDE, view logs, set breakpoints, and get a terminal into the running container. https://aws.amazon.com/about-aws/whats-new/2019/11/announcing-cloud-debugging-beta/

Feedback very welcome!

cc @kiiadi

[aidansteele]

@clareliguori This is the best thing since sliced bread. I love where you're heading with this. A few questions - let me know if a different venue is more appropriate.

In order to investigate any security implications of the cloud debugging, I went digging. It's great to see the plugin is open source, but the juicy functionality seems to be locked away in a tool called cloud-debug that isn't included in the plugin.

The plugin makes a handful of HTTP requests and eventually ends up downloading a zip file. In that zip file, it has some release notes that are probably of interest to users? (expand arrow below to see)

RELEASE_NOTES.md

# Release Notes ## CloudDebug CLI version 1.0.151 - 2019-11-06 **Warning**: Do not use in a production environment. Cloud debugging changes the state of resources in your account and could lead to issues if you try to use it in production. ### Checksum * The checksums for each individual Cloud Debug artifact can be found in the metadata file associated with the artifact for that specific platform. The metadata file is found in the S3 bucket that contains the Cloud Debug production artifacts, under the 'release/metadata' prefix. ### What's new? * The state of the instrumentation/revert operations is now modelled in our S3 DAO * Add retry strategy to the WaitUntilServicesStable ECS call ### Supported Amazon ECS service attributes: * Launch type: Fargate, EC2 * Load balancing: ALB, NLB, None * Scheduler strategy: Replica or Daemon * Deployment controller: ECS, CodeDeploy * Repositories: public and ECR * Platform: Linux containers * Service type: Java ### Unsupported Amazon ECS service attributes: * Deployment controller: external * ECS Tasks (RunTask) * Containers from private repos (except for ECR) * Service startup: shell scripts (service must be started directly with `java`) * Platform: Windows containers * Using Cloud Debug on more than 10 services at once ### Other limitations and considerations: * Auto scaling will be turned off during debugging to improve the debugging experience (scaling will be set to 1) * Health checks will be turned off during debugging * The command (CMD parameter) must be explicitly specified for a Task Definition (won't be auto-detected) * Changes to the Docker file of an ECS Service being debugged (e.g. adding a new dependency) require disabling and re-enabling Cloud Debugging on the respective service

Is there a reason these aren't made more readily available? The zip file also includes two binaries: cloud-debug and session-manager-plugin. Is there any plan to make these open source? It's been extremely helpful in convincing internal security teams that AWS tools can be trusted due to their code being on GitHub -- it's harder to do this (either in AWS or on our laptops!) when the source doesn't appear to be available.

Thanks again for an awesome tool.

[jicowan]

@cbbarclay are there plans to move this from beta to GA?

[JohnPreston]

Sorry if that is not the appropriate place for this. I was just checking in on the roadmap and saw this issue, thought this might help some of you.

I have written an open source project similar to what ecs-cliv1/docker ecs-plugin does, parsing docker-compose files, to render AWS CloudFormation templates (and auto defines ECS Task Definitions/Service Definitions/ IAM roles etc) called ECS Compose-X

But it does not just define ECS resources, you can also define other AWS resources(RDS, SQS, DocDB, DyDB, ELB, certs etc.), volumes, network and secrets, which I would imagine for a number of web developers could be of help to get their apps to AWS ECS quickly. I automatically takes care of Security groups ingress and IAM permissions to keep things simple to define. Supports AWS SAM policies and if you have used docker ecs-plugins markup (x-aws-) it also will interpret these.

Not quite a IntelliJ plugin and I have never written a plugin for intelliJ, but given this is something I have written for developers first, I hope this will help some of you.