snowflakedb / snowflake-connector-python

Snowflake Connector for Python
https://pypi.python.org/pypi/snowflake-connector-python/
Apache License 2.0
601 stars 474 forks source link

SNOW-249518: can't catch snowflake.connector.errors.ProgrammingError #575

Closed zevaverbach closed 3 years ago

zevaverbach commented 3 years ago

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using (python --version)? 3.6.8

  2. What operating system and processor architecture are you using (python -c 'import platform; print(platform.platform())')? Linux-3.10.0-1160.6.1.el7.x86_64-x86_64-with-redhat-7.9-Maipo

  3. What are the component versions in the environment (pip freeze)?

alembic==1.2.1
apache-airflow==1.10.9
apispec==1.3.3
appdirs==1.4.3
argcomplete==1.11.1
asn1crypto==0.24.0
astroid==2.4.2
attrs==19.3.0
azure-common==1.1.23
azure-devops==6.0.0b2
azure-storage-blob==2.1.0
azure-storage-common==2.1.0
Babel==2.7.0
backcall==0.2.0
bcrypt==3.1.7
bench-it==1.0.1
black==19.3b0
blessings==1.7
bokeh==1.3.4
boto3==1.9.242
botocore==1.12.242
bpython==0.19
brotlipy==0.7.0
cached-property==1.5.1
cattrs==0.9.0
certifi==2020.6.20
cffi==1.14.0
cfgv==3.2.0
chardet==3.0.4
Click==7.0
cloudpickle==1.2.2
colorama==0.4.1
colorlog==4.0.2
commonmark==0.9.1
conda==4.8.3
conda-package-handling==1.7.0
configparser==3.5.3
convertdate==2.3.0
cron-descriptor==1.2.24
croniter==0.3.30
cryptography==2.9.2
curtsies==0.3.4
cx-Oracle==7.2.3
dask==2.5.0
dataclasses==0.7
decorator==4.4.2
defusedxml==0.6.0
dictdiffer==0.8.1
diff-cover==2.6.1
dill==0.2.9
distlib==0.3.0
distributed==2.5.1
docutils==0.15.2
dumb-init==1.2.2
entrypoints==0.3
environs==8.0.0
et-xmlfile==1.0.1
filelock==3.0.12
flake8==3.7.8
flaky==3.7.0
Flask==1.1.1
Flask-Admin==1.5.4
Flask-AppBuilder==2.3.1
Flask-Babel==1.0.0
Flask-Bcrypt==0.7.1
Flask-Caching==1.3.3
Flask-JWT-Extended==3.23.0
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.1
flask-swagger==0.2.13
Flask-WTF==0.14.2
fsspec==0.5.1
funcsigs==1.0.0
future==0.16.0
gprof2dot==2019.11.30
graphviz==0.13.2
greenlet==0.4.17
gunicorn==19.9.0
HeapDict==1.0.1
holidays==0.10.3
ibm-db==3.0.1
ibm-db-sa==0.3.5
identify==1.4.30
idna @ file:///tmp/build/80754af9/idna_1593446292537/work
ijson==2.5.1
importlib-metadata==1.6.0
importlib-resources==2.0.1
inflect==4.1.0
iniconfig==1.0.1
ipython==7.16.1
ipython-genutils==0.2.0
iso8601==0.1.12
isodate==0.6.0
isort==5.5.0
itsdangerous==1.1.0
jdcal==1.4.1
jedi==0.17.2
Jinja2==2.10.1
jinja2-pluralize==0.3.0
jmespath==0.9.4
jplephem==2.14
json-merge-patch==0.2
jsonschema==3.0.2
korean-lunar-calendar==0.2.1
lazy-object-proxy==1.4.2
ldap3==2.8.1
locket==0.2.0
lockfile==0.12.2
lunardate==0.2.0
Mako==1.1.0
Markdown==2.6.11
MarkupSafe==1.1.1
marshmallow==2.19.5
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.19.0
mccabe==0.6.1
more-itertools==8.4.0
msgpack==0.6.2
msrest==0.6.18
neovim==0.3.1
nodeenv==1.5.0
numpy==1.17.2
oauthlib==3.1.0
openpyxl==3.0.5
ordereddict==1.1
oyaml==1.0
packaging==19.2
pandas==0.25.1
parso==0.7.1
partd==1.0.0
pendulum==1.4.4
pexpect==4.8.0
pickleshare==0.7.5
Pillow==6.2.0
pipenv==2020.6.2
pluggy==0.13.1
pre-commit==2.7.1
prison==0.1.3
prometheus-client==0.8.0
prompt-toolkit==3.0.8
psutil==5.6.3
psycopg2-binary==2.8.3
ptyprocess==0.6.0
pure-sasl==0.6.2
pure-transport==0.1.4
py==1.9.0
pyarrow==0.14.1
pyasn1==0.4.8
pyCalverter==1.6.1
pycodestyle==2.5.0
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pycryptodome==3.9.7
pycryptodomex==3.9.0
pyflakes==2.1.1
Pygments==2.7.1
PyHive==0.6.1.dev0
PyJWT==1.7.1
pylint==2.6.0
pyluach==1.1.0
PyMeeus==0.3.7
pymssql==2.1.4
PyMySQL==0.9.3
pynvim==0.4.2
pyodbc==4.0.27
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1594392929924/work
pyparsing==2.4.2
pyrsistent==0.15.4
PySocks==1.7.1
pytest==6.0.1
pytest-profiling==1.7.0
python-daemon==2.1.2
python-dateutil==2.8.0
python-dotenv==0.14.0
python-editor==1.0.4
python3-openid==3.1.0
pytz==2019.2
pytzdata==2019.3
PyYAML==5.1.2
requests @ file:///tmp/build/80754af9/requests_1592841827918/work
requests-oauthlib==1.3.0
rich==7.1.0
ruamel-yaml==0.15.87
s3transfer==0.2.1
sasl==0.2.1
selenium==3.141.0
setproctitle==1.1.10
sgp4==2.12
six==1.15.0
skyfield==1.26
skyfield-data==1.1.0
snowflake-connector-python==2.0.1
snowflake-sqlalchemy==1.1.15
sortedcontainers==2.1.0
sqlacodegen==2.3.0
SQLAlchemy==1.3.8
SQLAlchemy-JSONField==0.9.0
SQLAlchemy-Utils==0.36.3
sqlfluff==0.3.6
sqlparse==0.3.1
tabulate==0.8.5
tblib==1.4.0
tenacity==4.12.0
teradata==15.10.0.21
teradatasql==16.20.0.60
termcolor==1.1.0
text-unidecode==1.2
thrift==0.11.0
thrift-sasl==0.3.0
toml==0.10.0
toolz==0.10.0
tornado==6.0.3
tqdm @ file:///tmp/build/80754af9/tqdm_1593446365756/work
traitlets==4.3.3
typing==3.7.4.1
typing-extensions==3.7.4.1
tzlocal==1.5.1
unicodecsv==0.14.1
urllib3==1.25.9
virtualenv==20.0.25
virtualenv-clone==0.5.4
wcwidth==0.2.5
Werkzeug==0.16.0
wordsegment==1.3.1
workalendar==10.3.0
wrapt==1.12.1
WTForms==2.2.1
zict==1.0.0
zipp==3.1.0
zope.deprecation==4.4.0
  1. What did you do? If possible, provide a recipe for reproducing the error. A complete runnable program is good.
    
    ...
    from snowflake.connector import ProgrammingError
    # I also tried "from snowflake.connector.errors import ProgrammingError"

... for view in views: try: e.execute(f"select * from {view} limit 1") except ProgrammingError as e: print(str(e)) missing_views.append(view) except Exception as e: print(str(e)) continue


5. What did you expect to see?
I expect the first `except` to catch any Snowflake `ProgrammingError`s.

6. What did you see instead?
They aren't caught in the first `except`, only in the second "catchall" `except`.

7. Can you set logging to DEBUG and collect the logs?

... [2020-12-17 14:41:58,078] {connection.py:846} DEBUG - sql=[ROLLBACK], sequence_id=[295], is_file_transfer=[None] [2020-12-17 14:41:58,078] {network.py:940} DEBUG - Active requests sessions: 1, idle: 0 [2020-12-17 14:41:58,078] {network.py:639} DEBUG - remaining request timeout: None, retry cnt: 1 [2020-12-17 14:41:58,078] {network.py:778} DEBUG - socket timeout: 60 [2020-12-17 14:41:58,396] {connectionpool.py:383} DEBUG - "POST /queries/v1/query-request?requestId=&request_guid=HTTP/1.1" 200 995 [2020-12-17 14:41:58,396] {network.py:808} DEBUG - SUCCESS [2020-12-17 14:41:58,397] {network.py:953} DEBUG - Active requests sessions: 0, idle: 1 [2020-12-17 14:41:58,397] {network.py:536} DEBUG - ret[code] = None, after post request [2020-12-17 14:41:58,397] {cursor.py:526} DEBUG - sfqid: [2020-12-17 14:41:58,397] {cursor.py:528} INFO - query execution done [2020-12-17 14:41:58,397] {cursor.py:530} DEBUG - SUCCESS [2020-12-17 14:41:58,397] {cursor.py:534} DEBUG - PUT OR GET: None

(snowflake.connector.errors.ProgrammingError) 002057 (42601): : SQL compilation error: View definition for '' declared X column(s), but view query produces Y column(s). [SQL: select * from limit 1] (Background on this error at: http://sqlalche.me/e/f405)

[2020-12-17 14:41:58,398] {connection.py:561} DEBUG - cursor [2020-12-17 14:41:58,398] {cursor.py:443} DEBUG - executing SQL/command [2020-12-17 14:41:58,398] {connection.py:1039} DEBUG - parameters: {} [2020-12-17 14:41:58,398] {cursor.py:466} DEBUG - binding: [select from limit 1] with input=[{}], processed=[{}] [2020-12-17 14:41:58,398] {cursor.py:502} INFO - query: [select from limit 1] [2020-12-17 14:41:58,398] {connection.py:1068} DEBUG - sequence counter: 296 [2020-12-17 14:41:58,398] {cursor.py:323} DEBUG - running query [select * from limit 1] [2020-12-17 14:41:58,399] {cursor.py:332} DEBUG - is_file_transfer: False ...



The above error is only caught and printed out via the bare `Exception`, not the expected `except ProgrammingError`.
sfc-gh-mkeller commented 3 years ago

Hey, @zevaverbach It looks like you are using our SQLAlchemy library https://github.com/snowflakedb/snowflake-sqlalchemy Please note that SQLAlchemy wraps the exceptions raised by the underlying connectors in its own Exception. See for an example: https://github.com/snowflakedb/snowflake-sqlalchemy/blob/6615f51444c539d56ade2be08c542dd4827a70a8/snowdialect.py#L344

zevaverbach commented 3 years ago

@sfc-gh-mkeller thanks for your response, this was spot-on. In your opinion is it problematic that the exception doesn't identify its wrapper (it shows as snowflake.connector.errors.ProgrammingError as seen in the log)? If so, can you help me understand where/how this is happening? Maybe I can make a PR to fix it (here or in snowflake-sqlalchemy).

zevaverbach commented 2 years ago

Since I just found my way back here via a Google search, I'm going to explicitly add the solution here.

Instead of using

from snowflake.connector.errors import ProgrammingError
...
except SnowflakeProgrammingError:
    ....

you need to do

from sqlalchemy import exc as sa_exc
...
except sa_exc.ProgrammingError:
    ...
rchua-at-synagie commented 2 years ago

Will this be handled in the future? Kind of unintuitive. Encountered this problem today too.

zevaverbach commented 2 years ago

@sfc-gh-mkeller hey, do you have any thoughts about my question above? I've just arrived here for the third time from a Google search, and it seems like a few other wanderers have had the same confusion.