search

rails-sqlserver / tiny_tds

TinyTDS - Simple and fast FreeTDS bindings for Ruby using DB-Library.
Other
607 stars 189 forks source link

Intermittent `SystemStackError: stack level too deep` and `TinyTds::Error: Server name not found in configuration files` occuring #442

Closed pbhogan closed 4 years ago

pbhogan commented 5 years ago

Environment

Operating System

Distributor ID: Debian
Description:    Debian GNU/Linux 9.9 (stretch)
Release:    9.9
Codename:   stretch
Linux rws02-1 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux

Also happens on:

ProductName:    Mac OS X
ProductVersion: 10.14.6
BuildVersion:   18G95

TinyTDS Version and Information

gem "tiny_tds", "~> 2.1.2"

                            Version: freetds v1.00.104
             freetds.conf directory: /etc/freetds
     MS db-lib source compatibility: no
        Sybase binary compatibility: yes
                      Thread safety: yes
                      iconv library: yes
                        TDS version: 4.2
                              iODBC: no
                           unixodbc: yes
              SSPI "trusted" logins: no
                           Kerberos: yes
                            OpenSSL: no
                             GnuTLS: yes
                               MARS: no

FreeTDS Version

On Debian:

freetds-bin  1.00.104-1~bpo9+1 
freetds-common  1.00.104-1~bpo9+1 
freetds-dev  1.00.104-1~bpo9+1

On macOS:

freetds: stable 1.1.15 (installed through homebrew)

Description

We are running a series of background jobs (with Sidekiq) to push information to a database with tiny_tds. There are around 50,000 jobs.

The vast majority of the jobs work perfectly, so it's not a configuration error. But on any given run there are about 5 to 10 TinyTds::Error: Server name not found in configuration files errors occuring.

And then at some point a SystemStackError: stack level too deep occurs and brings down the Sidekiq process entirely which then has to be restarted, after which the remainder of the jobs finish up without an issue (but this could just be due to the rarity of the error).

Our connection is made like so:

TinyTds::Client.new(
    :username => CONFIG.dynamics.username,
    :password => CONFIG.dynamics.password,
    :host     => CONFIG.dynamics.host,
    :port     => 1433,
    :timeout  => 300,
)

We've had it happen with and without a connection pool (using the connection_pool gem), and with the jobs running parallelized or with them running one at a time.

It's also happening both in production on Debian 9 and in development on macOS 10.14.

Can you shed any light on why these errors might be occurring? The more critical one to solve is the SystemStackError as it's bringing down our job system entirely. It occurs in https://github.com/rails-sqlserver/tiny_tds/blob/f2fceb222e5bc8d461ec563c4a096730b5a8e7b3/lib/tiny_tds/client.rb#L60 which appears to be a call into the native extension.

Why would connect throw a SystemStackError? It seems like any kind of connection error should just be a simple error that can be retried later instead of what appears to be a recursion bug.

aharpervc commented 5 years ago

The connect method is here, for reference: https://github.com/rails-sqlserver/tiny_tds/blob/master/ext/tiny_tds/client.c#L308

You'd probably need to provide the full stack trace to get farther. The problem might not even be in tinytds (could be freetds).

The only other thing that jumps out at me is your protocol version 4.2, which is old. Is that set on purpose?

pbhogan commented 5 years ago

Protocol version 4.2 is not set on purpose. It's probably just the default when installing the package. We've customized nothing beyond just installing packages.