search

atomita / fluent-plugin-aws-elasticsearch-service

This repository is extends of https://github.com/uken/fluent-plugin-elasticsearch/ which made connectable to Amazon Elassticsearch Service using Aws Signers V4. Please check the requirements to connect to Amazon Elassticsearch Service without using Aws Signers V4 for before starting use. (reference https://groups.google.com/forum/#!msg/fluentd/uW87VAOqxeE/cfhenicEBAAJ)
MIT License
122 stars 49 forks source link

gem install fails for fluent-plugin-aws-elasticsearch-service #39

Closed randomyug closed 5 years ago

randomyug commented 6 years ago

Hi,

I am building an AMI ( with pre installed fluentd and this plugin ). It was working fine for several months. But recently it seems to be broken. I see that dev activity picked up again.

I was using this command /usr/sbin/td-agent-gem install fluent-plugin-aws-elasticsearch-service

But now it fails with compilation of strptime-0.1.9 and in the corresponding mkmf.log I see gcc trying to compile something. We do not have gcc in our AMI.

Is this expected ? Dependency on strptime-0.1.9 .. is it newly added?

How do I install the latest fluent-plugin-aws-elasticsearch-service ?

As a workaround, I am using this command /usr/sbin/td-agent-gem install fluent-plugin-aws-elasticsearch-service -v 0.1.6

But want to use the latest fluent-plugin-aws-elasticsearch-service

cosmo0920 commented 6 years ago

Is this expected ? Dependency on strptime-0.1.9 .. is it newly added?

How do I install the latest fluent-plugin-aws-elasticsearch-service ?

Yes. It is expected. Because Fluentd v0.14 depends on strptime gem which is written in C. strptime is introduced in Fluentd v0.14.0 for supporting EventTime msgpack extension type in Fluentd.

You should install ruby-dev (or ruby-devel?) package and gcc related package and then, remove development packages.

sriramsv commented 6 years ago

I tried to install ruby-devel and gcc related packages, but still install fails, I also tried install strptime gem 0.2.3 I am using fluentd-1.0.2.

sriramsv commented 6 years ago

@cosmo0920 Actually, after further experimentation, the real problem with installation failing with version 1.1.0 and above is td-agent-gem tries to pull all the aws-sdk-* dependencies in parallel thus causing td-agent-gem installer to run OOM and fail out.

Using an instance type > t2.micro seems to have solved this issue.

The workaround to this problem is to migrate the aws-sdk-core to v3 and defining only the required aws-sdk dependencies in the gemfile.

cosmo0920 commented 5 years ago

fluent-plugin-aws-elasticsearch-service stops to depends a lots of aws-sdk related gems. Could you try fluent-plugin-aws-elasticsearch-service v2.0.0 in your environment?

cosmo0920 commented 5 years ago

49 is merged. Closing.