Proxy crashing

[sthomp]

Im trying to connect to cloud sql from a compute engine instance in my python application. Im running the proxy like this:

nohup ./cloud_sql_proxy -dir=/cloudsql --instances=my-project:us-central1:my-sql-instance=tcp:3307 &

I can successfully connect using the MySql client:

mysql -u root --host 127.0.0.1 --port 3307 -p
mysql> select 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.00 sec)

I can also connect from a python repl:

>>> from sqlalchemy import create_engine
>>> from sqlalchemy import text
>>> engine = create_engine('mysql+pymysql://root:password@localhost:3307/db')
>>> engine.execute(text("SELECT 1")).fetchone()
(1,)

But, when I run this same code from my application I'll get Network Unreachable errors:

  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1990, in execute
    connection = self.contextual_connect(close_with_result=True)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
    e, dialect, self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
    self.connection = self.__connect()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
    connection = self.__pool._invoke_creator(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 922, in connect
    raise exc
OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'localhost' ([Errno 101] Network is unreachable)")

When I ssh back to the box and check the running processes the Cloud SQL proxy is no longer running.

pgrep cloud_sql_proxy
# No longer return a process id

My application is a script so its doing a lot of SQL work so maybe its overloading the proxy? How can I figure out the issue?

[hermantai]

You probably need to provide the output of the cloud sql proxy for someone to help out.

Instead of "nohup ./cloud_sql_proxy -dir=/cloudsql --instances=my-project:us-central1:my-sql-instance=tcp:3307 &"

Run "nohup ./cloud_sql_proxy -dir=/cloudsql --instances=my-project:us-central1:my-sql-instance=tcp:3307 &> /some-path-to-save-the-log.txt &"

[sthomp]

Thanks Ill give that a try. It seems to be related to the number of connections. If I lower my connection pool things seem to be more stable:

create_engine(...., poolclass=QueuePool, pool_size=1)

[Carrotman42]

Thanks for the bug report!

It would be very helpful for debugging's sake to grab the logs. Note, though, that logs are printed out on stderr, so you'll have to do something more like this:

nohup ./cloud_sql_proxy -dir=/cloudsql --instances=my-project:us-central1:my-sql-instance=tcp:3307 2> /some-path-to-save-the-log.txt &

[sthomp]

I tried to reproduce this issue and Im not seeing the problem right now. Ill keep my initialization script as you mentioned so that Ill be capturing log output. Next time the issue arises hopefully Ill have a log to report. Feel free to close this issue for now.

[Carrotman42]

Definitely reopen the issue if you notice it happening again.