search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.41k stars 9.5k forks source link

Support AWS Data Pipeline #4077

Closed danielcompton closed 7 years ago

danielcompton commented 8 years ago

It would be nice to be able to use AWS Data Pipeline with Terraform, especially for things like updating the AMI id when Packer builds a new one.

http://docs.aws.amazon.com/datapipeline/latest/APIReference/Welcome.html

pawelsawicz commented 8 years ago

Hey,

I am looking at it, will try to implement this, as I am new in Go it can take a while.

stack72 commented 8 years ago

@pawelsawicz are you still working on this? If so, how is the progress? :)

desmondmorris commented 8 years ago

+1 This would be great.

jamescarr commented 8 years ago

Just checking in again on this. We'd benefit from it as well.

pawelsawicz commented 8 years ago

I am changing jobs now so I had to do a break on it.

jamescarr commented 8 years ago

That's fine... I'm curious if hashicorp has this on their roadmap?

radeksimko commented 8 years ago

@jamescarr I can't talk on behalf of Hashicorp, but as for myself, it is somewhere on my to-do list, below Lambda related resources, CloudWatch Events and API Gateway. Possibly fixing a few reported RDS & CloudFormation & ECS bugs may also take priority. I can't say when I will get to it, I'm merely expressing the interest in getting it done.

If you really need it and don't want to wait, it's probably best to send a PR. Here are some hints on what can you do to speed things up when sending the PR (i.e. what makes a good PR that has higher chance of being merged sooner): https://github.com/hashicorp/terraform/blob/master/.github/CONTRIBUTING.md#pull-requests

ApsOps commented 8 years ago

+1 It would be nice, for taking/updating dynamodb backups as well. Any updates on this?

pawelsawicz commented 8 years ago

After a while, I get back to that. I just want to double check if it's still free to pick up. I assume that nobody send PR yet ?

stack72 commented 8 years ago

Hi @pawelsawicz

Feel free to pick this up still :)

Paul

pawelsawicz commented 8 years ago

@stack72 yes I am working on this at my fork

I have some problems with installing SDK's to intellij to fully tests it, but I will drop you a line to keep this conversation clean.

stack72 commented 8 years ago

Nice :)

rchibiscus commented 8 years ago

looking forward to this feature!!!

razic commented 8 years ago

oh sad face :(

apparentlymart commented 7 years ago

Hi all,

Thanks for opening this request and to @pawelsawicz for taking a stab at implementing it.

We (the Terraform team) would love to support AWS Data Pipeline, but it's a bit of a beast to implement and we don't have any plans to work on it in the short term. We're trying to prune enhancement requests that are stale and likely to remain that way for the foreseeable future, so I'm going to close this.

However, if a motivated person were interested in working on this we'd be happy to provide guidance and review a PR.

howdoicomputer commented 7 years ago

@pawelsawicz @apparentlymart My company has a need for data pipeline support. As of right now, we're wrapping a cloud formation template but I would love it if there was native support. If nobody is chucking code at this, would a pull request be welcome?

apparentlymart commented 7 years ago

Hi @howdoicomputer!

A PR would indeed be welcome. It would be nice to have data pipeline support.

Given that data pipeline is a pretty complex service, I'd suggest trying to attack it in several smaller steps if possible. Usually we try to start with one or two most important resources, or a useful subset there of, and then build up from there. If there are particular subsets of functionality you need then starting there seems reasonable! Just need to do our best to leave room in the design for other functionality to be added later.

Thanks for your interest!

howdoicomputer commented 7 years ago

@apparentlymart Hooray! It'll take me a bit between work responsibilities and terrashovel but I'd be super happy to help.

apparentlymart commented 7 years ago

Sorry, forgot to mention this yesterday: since this issue was opened the AWS provider development has moved into a separate repo, any new issues or PRs for this will need to be opened there in future. This is part of the provider/core split.

arevell89 commented 7 years ago

any update on this?

eugenejen commented 7 years ago

I haven't seen any one raises the issue regarding data pipeline in terraform-provider-aws

sorjef commented 7 years ago

Hi there!

I had a similar problem and I've come up with a script to convert data pipelines templates to CloudFormation templates and a Terraform module to help integrate it into the configuration.

The script is written on node.js, so it might not work for everybody, but still, I hope it helps somebody.

I've added more context in terms of my setup and the link to the solution in this StackOverflow post.

ishantanu commented 6 years ago

+1 This would be great. It's Oct, 2017 and still waiting for the support.

melbourne2991 commented 6 years ago

+1 would be really helpful to have this

radeksimko commented 6 years ago

As mentioned above AWS provider now has its own repository, please follow https://github.com/terraform-providers/terraform-provider-aws/issues/1538 Thanks.