fluent / fluent-plugin-sql

SQL input/output plugin for Fluentd
108 stars 57 forks source link

How to connect to sql server #76

Open eamonnmcevoy opened 4 years ago

eamonnmcevoy commented 4 years ago

I have tried installing tiny_tds, also tried activerecord-sqlserver-adapter but both give me the following error:

2019-10-14 07:21:34 +0000 [error]: #0 fluent/log.rb:362:error: Could not load the 'tiny_tds' Active Record adapter. Ensure that the adapter is spelled correctly in config/database.yml and that you've added the necessary adapter gem to your Gemfile. path=nil error_class=LoadError error="Could not load the 'tiny_tds' Active Record adapter. Ensure that the adapter is spelled correctly in config/database.yml and that you've added the necessary adapter gem to your Gemfile."
2019-10-14 07:21:34 +0000 [error]: #0 fluent/log.rb:362:error: Could not load the 'activerecord-sqlserver-adapter' Active Record adapter. Ensure that the adapter is spelled correctly in config/database.yml and that you've added the necessary adapter gem to your Gemfile. path=nil error_class=LoadError error="Could not load the 'activerecord-sqlserver-adapter' Active Record adapter. Ensure that the adapter is spelled correctly in config/database.yml and that you've added the necessary adapter gem to your Gemfile."

Dockerfile:

# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /Dockerfile.template.erb

FROM alpine:3.9
LABEL maintainer "Fluentd developers <fluentd@googlegroups.com>"
LABEL Description="Fluentd docker image" Vendor="Fluent Organization" Version="1.7.3"

# Do not split this into multiple RUN!
# Docker creates a layer for every RUN-Statement
# therefore an 'apk delete' has no effect
RUN apk update \
 && apk add --no-cache \
        ca-certificates linux-headers \
        ruby ruby-irb ruby-etc ruby-webrick \
        tini \
 && apk add --no-cache --virtual .build-deps \
        build-base \
        ruby-dev gnupg \
 && apk add freetds freetds-dev \
 && echo 'gem: --no-document' >> /etc/gemrc \
 && gem install tiny_tds -v 2.1.0 \
 && gem install fluent-plugin-sql -v 1.1.1 \
 && gem install activerecord-sqlserver-adapter -v 5.2.0 \
 && gem install oj -v 3.3.10 \
 && gem install json -v 2.2.0 \
 && gem install async-http -v 0.46.3 \
 && gem install fluentd -v 1.6.3 \
 && gem install bigdecimal -v 1.3.5 \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

RUN addgroup -S fluent && adduser -S -g fluent fluent \
    # for log storage (maybe shared with host)
    && mkdir -p /fluentd/log \
    # configuration/plugins path (default: copied from .)
    && mkdir -p /fluentd/etc /fluentd/plugins \
    && chown -R fluent /fluentd && chgrp -R fluent /fluentd

COPY fluent.conf /fluentd/etc/
COPY entrypoint.sh /bin/

ENV FLUENTD_CONF="fluent.conf"

ENV LD_PRELOAD=""
EXPOSE 24224 5140

USER fluent
ENTRYPOINT ["tini",  "--", "/bin/entrypoint.sh"]
CMD ["fluentd"]

conf

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match **>
  @type stdout
</match>
<match *>
  @type sql
  host ...
  port 1433
  database fluentd
  adapter tiny_tds / activerecord-sqlserver-adapter
  username ...
  password ...

  <table>
    table dbo.logs
    column_mapping 'timestamp:created_at,fluentdata1:dbcol1,fluentdata2:dbcol2,fluentdata3:dbcol3'
    # This is the default table because it has no "pattern" argument in <table>
    # The logic is such that if all non-default <table> blocks
    # do not match, the default one is chosen.
    # The default table is required.
  </table>

</match>
tuiaswath commented 4 years ago

If you are still facing this issue or anyone facing the issue in the future, i hope this helps

Try changing the adapter to sqlserver. it works for me. this is only difference i have with my configuration.

......
  database fluentd
  adapter sqlserver
  username ...
......
RAJMODI038 commented 1 year ago

@eamonnmcevoy - were you able to resolve this issue? I'm facing the similar for snowflake! I see you're using activerecord-sqlserver-adapter for sqlserver. so do you know if there any activerecord adapter available for snowflake?