python ibmcloudsql library has the wrong dependency on pandas

powellquiring commented 2 years ago

I need to write two defects:

I think this repo is the right place for the second one: ibmcloudsql is not expressing the dependency requirement correctly for pandas. Please help me find the right place to file the first issue.

[ ] Watson studio pre installed software has a dependency problem with ibmcloudsql
[ ] ibmcloudsql is not expressing the dependency requirement correctly for pandas

First problem: watson studio has conflicting dependencies. Expected: ibmcloudsql would install without errors. Actual: error message produced.

If you tell me where to open this issue I will do so:

New notebook

!pip freeze

absl-py @ file:///tmp/build/80754af9/absl-py_1600297518631/work
aiohttp @ file:///tmp/build/80754af9/aiohttp_1614360992924/work
appdirs==1.4.4
argon2-cffi @ file:///tmp/build/80754af9/argon2-cffi_1613037097816/work
astropy @ file:///tmp/build/80754af9/astropy_1617745353437/work
astunparse==1.6.3
async-generator @ file:///home/ktietz/src/ci/async_generator_1611927993394/work
async-timeout==3.0.1
attrs @ file:///tmp/build/80754af9/attrs_1620827162558/work
autoai-libs==1.12.13
autovizwidget @ file:///tmp/build/80754af9/autovizwidget_1613246391658/work
av==8.0.2
azure-core==1.15.0
azure-cosmos==4.2.0
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
beautifulsoup4 @ file:///home/linux1/recipes/ci/beautifulsoup4_1610988766420/work
biopython @ file:///tmp/build/80754af9/biopython_1632748063151/work
black==19.10b0
bleach @ file:///tmp/build/80754af9/bleach_1612211392645/work
blinker==1.4
bokeh @ file:///tmp/build/80754af9/bokeh_1620779595936/work
boto3 @ file:///tmp/build/80754af9/boto3_1617902877650/work
botocore @ file:///tmp/build/80754af9/botocore_1623076139559/work
brotlipy==0.7.0
ca-data-connector @ file:///tmp/ws-build/install/ca-data-connector-python-11.2.0.tgz
cachetools @ file:///tmp/build/80754af9/cachetools_1619597386817/work
cassandra-driver==3.25.0
certifi==2021.10.8
cffi @ file:///tmp/build/80754af9/cffi_1613246945912/work
chardet @ file:///tmp/build/80754af9/chardet_1605303185383/work
click @ file:///home/linux1/recipes/ci/click_1610990599742/work
cloudpickle @ file:///Users/ktietz/demo/mc3/conda-bld/cloudpickle_1629142150447/work
coverage @ file:///tmp/build/80754af9/coverage_1614613670853/work
cplex==20.1.0.1
cryptography @ file:///tmp/build/80754af9/cryptography_1616769286105/work
cx-Oracle @ file:///tmp/build/80754af9/cx_oracle_1622750149598/work
cycler==0.10.0
Cython @ file:///tmp/build/80754af9/cython_1636035786592/work
cytoolz==0.11.0
dask @ file:///tmp/build/80754af9/dask-core_1623085854073/work
debugpy @ file:///tmp/build/80754af9/debugpy_1629222698064/work
decision-optimization-client==1.0.0
decorator @ file:///tmp/build/80754af9/decorator_1621259047763/work
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
docplex==2.20.204
docutils @ file:///tmp/build/80754af9/docutils_1613157663989/work
entrypoints==0.3
et-xmlfile==1.1.0
flatbuffers @ file:///tmp/build/80754af9/python-flatbuffers_1614345733764/work
fsspec @ file:///tmp/build/80754af9/fsspec_1617959894824/work
future==0.18.2
gast==0.3.3
gensim==3.8.3
geographiclib==1.50
geomet==0.2.1.post1
geopy==2.1.0
gmpy2==2.0.8
google-auth @ file:///tmp/build/80754af9/google-auth_1604100403705/work
google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1617120569401/work
google-pasta @ file:///Users/ktietz/demo/mc3/conda-bld/google-pasta_1630577991354/work
graphviz==0.16
greenery==3.3.2
greenlet @ file:///tmp/build/80754af9/greenlet_1628887725296/work
grpcio @ file:///tmp/build/80754af9/grpcio_1613730115948/work
h5py==2.10.0
hdijupyterutils @ file:///tmp/build/80754af9/hdijupyterutils_1613415755648/work
html5lib @ file:///Users/ktietz/demo/mc3/conda-bld/html5lib_1629144453894/work
hyperopt==0.2.5
ibm-cloud-sdk-core==3.10.1
ibm-cos-sdk==2.7.0
ibm-cos-sdk-core==2.7.0
ibm-cos-sdk-s3transfer==2.7.0
ibm-db==3.0.4
ibm-db-sa==0.3.7
ibm-watson-machine-learning==1.0.173
ibm-watson-openscale==3.0.12
ibm-watson-studio-lib @ file:///tmp/ws-build/install/ibm-watson-studio-lib-py-3.0.7.tgz
idna==2.8
imageio @ file:///tmp/build/80754af9/imageio_1617700267927/work
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1617874469820/work
importlib-resources @ file:///tmp/build/80754af9/importlib_resources_1625135880749/work
iniconfig @ file:///home/linux1/recipes/ci/iniconfig_1610983019677/work
ipykernel @ file:///tmp/build/80754af9/ipykernel_1633545412716/work/dist/ipykernel-6.4.1-py3-none-any.whl
ipython @ file:///tmp/build/80754af9/ipython_1632141392005/work
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
ipywidgets @ file:///tmp/build/80754af9/ipywidgets_1610481889018/work
itc-utils @ file:///tmp/ws-build/install/itc-library-python-0.1.9.tgz
JayDeBeApi==1.2.3
jdcal @ file:///Users/ktietz/demo/mc3/conda-bld/jdcal_1630584345063/work
jedi @ file:///tmp/build/80754af9/jedi_1606932564285/work
Jinja2 @ file:///tmp/build/80754af9/jinja2_1621238361758/work
jmespath @ file:///Users/ktietz/demo/mc3/conda-bld/jmespath_1630583964805/work
joblib @ file:///tmp/build/80754af9/joblib_1601912903842/work
JPype1==1.3.0
jsonref==0.2
jsonschema @ file:///Users/ktietz/demo/mc3/conda-bld/jsonschema_1630511932244/work
jsonsubschema==0.0.5
jupyter-client @ file:///tmp/build/80754af9/jupyter_client_1616770841739/work
jupyter-core @ file:///tmp/build/80754af9/jupyter_core_1612213311222/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab-widgets @ file:///tmp/build/80754af9/jupyterlab_widgets_1609884341231/work
Keras-Preprocessing @ file:///tmp/build/80754af9/keras-preprocessing_1612283640596/work
kiwisolver @ file:///tmp/build/80754af9/kiwisolver_1612282420641/work
lale==0.5.5
lightgbm==3.2.1
locket==0.2.1
lomond==0.3.3
lxml @ file:///tmp/build/80754af9/lxml_1616443220220/work
Markdown==3.1.1
MarkupSafe @ file:///tmp/build/80754af9/markupsafe_1621528148836/work
matplotlib @ file:///tmp/build/80754af9/matplotlib-suite_1613407855456/work
matplotlib-inline @ file:///tmp/build/80754af9/matplotlib-inline_1628242447089/work
mistune==0.8.4
mock @ file:///tmp/build/80754af9/mock_1607622725907/work
more-itertools @ file:///tmp/build/80754af9/more-itertools_1622818384463/work
mpmath==1.2.1
multidict @ file:///tmp/build/80754af9/multidict_1607367757617/work
mypy-extensions==0.4.3
mysql-connector-python==8.0.25
nbclient @ file:///tmp/build/80754af9/nbclient_1614364831625/work
nbconvert @ file:///tmp/build/80754af9/nbconvert_1601914830498/work
nbformat @ file:///tmp/build/80754af9/nbformat_1617383369282/work
nest-asyncio @ file:///tmp/build/80754af9/nest-asyncio_1613680548246/work
networkx==2.3
nltk @ file:///tmp/build/80754af9/nltk_1621347441292/work
notebook @ file:///tmp/build/80754af9/notebook_1621528346532/work
numexpr @ file:///tmp/build/80754af9/numexpr_1618856526911/work
numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1603479643886/work
oauthlib @ file:///tmp/build/80754af9/oauthlib_1623060228408/work
olefile @ file:///Users/ktietz/demo/mc3/conda-bld/olefile_1629805411829/work
onnx==1.6.0
openpyxl @ file:///tmp/build/80754af9/openpyxl_1615411699337/work
opt-einsum==3.1.0
packaging @ file:///tmp/build/80754af9/packaging_1611952188834/work
pandas==1.2.4
pandocfilters @ file:///tmp/build/80754af9/pandocfilters_1605120460739/work
parso==0.7.0
partd @ file:///tmp/build/80754af9/partd_1618000087440/work
pathspec==0.7.0
patsy==0.5.1
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow==8.4.0
plotly @ file:///tmp/build/80754af9/plotly_1610565580232/work
pluggy @ file:///tmp/build/80754af9/pluggy_1615976321666/work
portion==2.1.6
project-lib @ file:///tmp/ws-build/install/project-lib-py-2.0.4.tgz
prometheus-client @ file:///tmp/build/80754af9/prometheus_client_1623189609245/work
prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1633440160888/work
protobuf==3.11.2
psycopg2 @ file:///tmp/build/80754af9/psycopg2_1612298147424/work
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py @ file:///tmp/build/80754af9/py_1607971587848/work
pyarrow==3.0.0
pyasn1 @ file:///Users/ktietz/demo/mc3/conda-bld/pyasn1_1629708007385/work
pyasn1-modules==0.2.8
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pycryptodome @ file:///tmp/build/80754af9/pycryptodome_1613764578245/work
pyerfa @ file:///tmp/build/80754af9/pyerfa_1613849062021/work
Pygments @ file:///tmp/build/80754af9/pygments_1621606182707/work
PyJWT @ file:///tmp/build/80754af9/pyjwt_1619651636675/work
pykerberos==1.2.1
pymssql==2.1.5
pyodbc===4.0.0-unsupported
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
pyrsistent @ file:///tmp/build/80754af9/pyrsistent_1600141720057/work
PySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/work
pytest==6.2.3
python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/work
pytz @ file:///tmp/build/80754af9/pytz_1612215392582/work
PyWavelets @ file:///tmp/build/80754af9/pywavelets_1601658317819/work
PyYAML==5.4.1
pyzmq==20.0.0
regex @ file:///tmp/build/80754af9/regex_1617569202463/work
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-kerberos==0.12.0
requests-oauthlib==1.3.0
retrying @ file:///Users/ktietz/demo/mc3/conda-bld/retrying_1629465456590/work
rsa @ file:///tmp/build/80754af9/rsa_1614366226499/work
s3transfer @ file:///tmp/build/80754af9/s3transfer_1616183147336/work
scikit-image==0.16.2
scikit-learn @ file:///tmp/build/80754af9/scikit-learn_1598376899566/work
scipy==1.4.1
seaborn @ file:///tmp/build/80754af9/seaborn_1608578541026/work
Send2Trash @ file:///tmp/build/80754af9/send2trash_1607525499227/work
sentencepiece==0.1.91
six @ file:///tmp/build/80754af9/six_1605205327372/work
smart-open==5.1.0
snapml==1.7.6
snowballstemmer @ file:///tmp/build/80754af9/snowballstemmer_1611258885636/work
sortedcontainers @ file:///tmp/build/80754af9/sortedcontainers_1606865132123/work
soupsieve @ file:///tmp/build/80754af9/soupsieve_1616183228191/work
sparkmagic @ file:///tmp/build/80754af9/sparkmagic_1614023370530/work/sparkmagic
SQLAlchemy @ file:///tmp/build/80754af9/sqlalchemy_1626941010864/work
statsmodels @ file:///tmp/build/80754af9/statsmodels_1614023746358/work
sympy @ file:///tmp/build/80754af9/sympy_1618252284338/work
tables==3.6.1
tabulate==0.8.9
tensorboard @ file:///opt/conda/conda-bld/tensorboard_1637064385358/work/tmp_pip_dir
tensorboard-plugin-wit==1.6.0
tensorflow==2.4.4
tensorflow-estimator @ file:///opt/conda/conda-bld/tensorflow-estimator_1637128688012/work/tensorflow_estimator_pkg/tensorflow_estimator-2.4.0-py2.py3-none-any.whl
termcolor==1.1.0
terminado==0.9.4
testpath @ file:///home/ktietz/src/ci/testpath_1611930608132/work
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
toolz @ file:///home/linux1/recipes/ci/toolz_1610987900194/work
torch==1.7.1
torchtext==0.8.0a0+0f911ec
torchvision==0.8.2
tornado @ file:///tmp/build/80754af9/tornado_1606942300299/work
tqdm @ file:///tmp/build/80754af9/tqdm_1615925068909/work
traitlets @ file:///home/ktietz/src/ci/traitlets_1611929699868/work
typed-ast @ file:///tmp/build/80754af9/typed-ast_1610484547928/work
typing-extensions @ file:///tmp/build/80754af9/typing_extensions_1611751222202/work
urllib3 @ file:///tmp/build/80754af9/urllib3_1625084269274/work
wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work
webencodings==0.5.1
Werkzeug @ file:///tmp/build/80754af9/werkzeug_1631818832573/work
widgetsnbextension==3.5.1
wrapt==1.12.1
xgboost==1.3.3
xlrd @ file:///tmp/build/80754af9/xlrd_1608072521494/work
XlsxWriter @ file:///tmp/build/80754af9/xlsxwriter_1617224712951/work
yarl @ file:///tmp/build/80754af9/yarl_1606939922162/work
zipp @ file:///tmp/build/80754af9/zipp_1615904174917/work

!pip install ibmcloudsql

Collecting ibmcloudsql
  Downloading ibmcloudsql-0.5.0.tar.gz (54 kB)
     |████████████████████████████████| 54 kB 2.5 MB/s  eta 0:00:01
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: pyarrow in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from ibmcloudsql) (3.0.0)
Collecting isodate
  Downloading isodate-0.6.0-py2.py3-none-any.whl (45 kB)
     |████████████████████████████████| 45 kB 5.7 MB/s  eta 0:00:01
Requirement already satisfied: python-dateutil in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from ibmcloudsql) (2.8.1)
Requirement already satisfied: numpy in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from ibmcloudsql) (1.19.2)
Requirement already satisfied: packaging in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from ibmcloudsql) (20.9)
Requirement already satisfied: typing-extensions in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from ibmcloudsql) (3.7.4.3)
Requirement already satisfied: requests>=2.2.0 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from ibmcloudsql) (2.25.1)
Collecting sqlparse>=0.4.2
  Downloading sqlparse-0.4.2-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 2.3 MB/s  eta 0:00:01
Collecting pre-commit
  Downloading pre_commit-2.16.0-py2.py3-none-any.whl (191 kB)
     |████████████████████████████████| 191 kB 11.8 MB/s eta 0:00:01
Collecting ibm-cos-sdk-core==2.10.0
  Downloading ibm-cos-sdk-core-2.10.0.tar.gz (888 kB)
     |████████████████████████████████| 888 kB 62.4 MB/s eta 0:00:01
Collecting ibm-cos-sdk==2.10.0
  Downloading ibm-cos-sdk-2.10.0.tar.gz (52 kB)
     |████████████████████████████████| 52 kB 2.3 MB/s  eta 0:00:01
Requirement already satisfied: pandas>=1.1.0 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from ibmcloudsql) (1.2.4)
Requirement already satisfied: importlib-metadata in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from ibmcloudsql) (3.10.0)
Collecting backoff==1.10.0
  Downloading backoff-1.10.0-py2.py3-none-any.whl (31 kB)
Collecting deprecated
  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
Collecting ibm-cos-sdk-s3transfer==2.10.0
  Downloading ibm-cos-sdk-s3transfer-2.10.0.tar.gz (133 kB)
     |████████████████████████████████| 133 kB 64.3 MB/s eta 0:00:01
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from ibm-cos-sdk==2.10.0->ibmcloudsql) (0.10.0)
Requirement already satisfied: pytz>=2017.3 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from pandas>=1.1.0->ibmcloudsql) (2021.1)
Requirement already satisfied: six>=1.5 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from python-dateutil->ibmcloudsql) (1.15.0)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from requests>=2.2.0->ibmcloudsql) (2.8)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from requests>=2.2.0->ibmcloudsql) (3.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from requests>=2.2.0->ibmcloudsql) (1.26.6)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from requests>=2.2.0->ibmcloudsql) (2021.10.8)
Requirement already satisfied: wrapt<2,>=1.10 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from deprecated->ibmcloudsql) (1.12.1)
Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from importlib-metadata->ibmcloudsql) (3.4.1)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from packaging->ibmcloudsql) (2.4.7)
Collecting virtualenv>=20.0.8
  Downloading virtualenv-20.10.0-py2.py3-none-any.whl (5.6 MB)
     |████████████████████████████████| 5.6 MB 31.6 MB/s eta 0:00:01
Collecting cfgv>=2.0.0
  Downloading cfgv-3.3.1-py2.py3-none-any.whl (7.3 kB)
Requirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from pre-commit->ibmcloudsql) (5.4.1)
Collecting nodeenv>=0.11.1
  Downloading nodeenv-1.6.0-py2.py3-none-any.whl (21 kB)
Collecting identify>=1.0.0
  Downloading identify-2.4.0-py2.py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 12.6 MB/s eta 0:00:01
Requirement already satisfied: toml in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from pre-commit->ibmcloudsql) (0.10.2)
Collecting filelock<4,>=3.2
  Downloading filelock-3.4.0-py3-none-any.whl (9.8 kB)
Collecting platformdirs<3,>=2
  Downloading platformdirs-2.4.0-py3-none-any.whl (14 kB)
Collecting backports.entry-points-selectable>=1.0.4
  Downloading backports.entry_points_selectable-1.1.1-py2.py3-none-any.whl (6.2 kB)
Collecting distlib<1,>=0.3.1
  Downloading distlib-0.3.4-py2.py3-none-any.whl (461 kB)
     |████████████████████████████████| 461 kB 67.9 MB/s eta 0:00:01
Building wheels for collected packages: ibmcloudsql, ibm-cos-sdk, ibm-cos-sdk-core, ibm-cos-sdk-s3transfer
  Building wheel for ibmcloudsql (PEP 517) ... done
  Created wheel for ibmcloudsql: filename=ibmcloudsql-0.5.0-py3-none-any.whl size=56366 sha256=935d96bb6642cc4bac8ecbcc7d9f5fb1a6972c79e7f1cba8bc4ace598b227188
  Stored in directory: /tmp/wsuser/.cache/pip/wheels/9a/67/be/eaf3911422a6dd44c150d38621af3ee0fec8f2680dd8d12300
  Building wheel for ibm-cos-sdk (setup.py) ... done
  Created wheel for ibm-cos-sdk: filename=ibm_cos_sdk-2.10.0-py2.py3-none-any.whl size=73425 sha256=7dde3f88c259e1e2e7763fb28724b47a7bb655be615102b83f9294bc9d7c6f45
  Stored in directory: /tmp/wsuser/.cache/pip/wheels/f3/a1/2e/0cffd7ccec0cb6963ec77d8473d28a7e79f3d0e4ecbf273588
  Building wheel for ibm-cos-sdk-core (setup.py) ... done
  Created wheel for ibm-cos-sdk-core: filename=ibm_cos_sdk_core-2.10.0-py2.py3-none-any.whl size=526045 sha256=c114c092187a8951eea1a8ff207412eaa0815a4f3079026abbf4c29b2751b3f9
  Stored in directory: /tmp/wsuser/.cache/pip/wheels/ce/df/1e/1da07def7155e0dd5cb2d93925ac6c0026042ecd3d3411ed9e
  Building wheel for ibm-cos-sdk-s3transfer (setup.py) ... done
  Created wheel for ibm-cos-sdk-s3transfer: filename=ibm_cos_sdk_s3transfer-2.10.0-py2.py3-none-any.whl size=88636 sha256=835171d6fac17e63a753c77c2605bc6a6e215c58916d3943fb9f7eebcd89406b
  Stored in directory: /tmp/wsuser/.cache/pip/wheels/ba/16/5d/43d4b1becec01bb67c90f2a5a87263b6f7b5a2226f45bda48b
Successfully built ibmcloudsql ibm-cos-sdk ibm-cos-sdk-core ibm-cos-sdk-s3transfer
Installing collected packages: platformdirs, ibm-cos-sdk-core, filelock, distlib, backports.entry-points-selectable, virtualenv, nodeenv, identify, ibm-cos-sdk-s3transfer, cfgv, sqlparse, pre-commit, isodate, ibm-cos-sdk, deprecated, backoff, ibmcloudsql
  Attempting uninstall: ibm-cos-sdk-core
    Found existing installation: ibm-cos-sdk-core 2.7.0
    Uninstalling ibm-cos-sdk-core-2.7.0:
      Successfully uninstalled ibm-cos-sdk-core-2.7.0
  Attempting uninstall: ibm-cos-sdk-s3transfer
    Found existing installation: ibm-cos-sdk-s3transfer 2.7.0
    Uninstalling ibm-cos-sdk-s3transfer-2.7.0:
      Successfully uninstalled ibm-cos-sdk-s3transfer-2.7.0
  Attempting uninstall: ibm-cos-sdk
    Found existing installation: ibm-cos-sdk 2.7.0
    Uninstalling ibm-cos-sdk-2.7.0:
      Successfully uninstalled ibm-cos-sdk-2.7.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ibm-watson-machine-learning 1.0.173 requires ibm-cos-sdk==2.7.*, but you have ibm-cos-sdk 2.10.0 which is incompatible.
Successfully installed backoff-1.10.0 backports.entry-points-selectable-1.1.1 cfgv-3.3.1 deprecated-1.2.13 distlib-0.3.4 filelock-3.4.0 ibm-cos-sdk-2.10.0 ibm-cos-sdk-core-2.10.0 ibm-cos-sdk-s3transfer-2.10.0 ibmcloudsql-0.5.0 identify-2.4.0 isodate-0.6.0 nodeenv-1.6.0 platformdirs-2.4.0 pre-commit-2.16.0 sqlparse-0.4.2 virtualenv-20.10.0

Second problem, dependency on pandas not expressed. Restart Kernel

Cell:

import ibmcloudsql
import getpass
cloud_api_key = getpass.getpass('Enter your IBM Cloud API Key')
sql_crn = 'crn:v1:bluemix:public:sql-query:us-south:a/713c783d9a507a53135fe6793c37cc74:7edc87d9-595d-47d7-b3a1-e7f6c9f86de0::'

sql_cos_endpoint = 'cos://us-south/data-lake-pfq'

sqlClient = ibmcloudsql.SQLQuery(cloud_api_key, sql_crn, sql_cos_endpoint + '/accidents')
data_source = sql_cos_endpoint + "/Traffic_Collision_Data_from_2010_to_Present.csv"
query = """
WITH location AS (
    SELECT
        id,
        cast(split(coordinates, ',')[0] as float) as latitude,
        cast(split(coordinates, ',')[1] as float) as longitude
    FROM (SELECT
            `Dr Number` as id,
            regexp_replace(Location, '[()]', '') as coordinates
        FROM {0}
    )
)
SELECT
    d.`Dr Number` as id,
    d.`Date Occurred` as date,
    d.`Time Occurred` AS time,
    d.`Area Name` AS area,
    d.`Victim Age` AS age,
    d.`Victim Sex` AS sex,
    l.latitude,
    l.longitude
FROM {0} AS d
    JOIN
    location AS l
    ON l.id = d.`Dr Number`
WHERE
    d.`Time Occurred` >= 1700 AND
    d.`Time Occurred` <= 2000 AND
    d.`Victim Age` >= 20 AND
    d.`Victim Age` <= 35 AND
    l.latitude != 0.0000 AND
    l.latitude != 0.0000
""".format(data_source)
print(query)

traffic_location = sqlClient.run_sql(query)
print(traffic_location)

Results:

...
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/wsuser/ipykernel_122/2163707681.py in <module>
     42 print(query)
     43 
---> 44 traffic_location = sqlClient.run_sql(query)
     45 print(traffic_location)

/opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages/ibmcloudsql/SQLQuery.py in run_sql(self, sql_text, pagesize)
   1665                 pprint.pprint(details)
   1666                 raise e
-> 1667         return self.get_result(job_id)
   1668 
   1669     def run(self, pagesize=None, get_result=False):

/opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages/ibmcloudsql/SQLQuery.py in get_result(self, jobId, pagenumber)
    942                         body.__iter__ = types.MethodType(self.__iter__, body)
    943 
--> 944                     partition_df = pd.read_csv(body, on_bad_lines='skip')
    945 
    946                 elif result_format == "parquet":

TypeError: read_csv() got an unexpected keyword argument 'on_bad_lines'

Upgrade pandas: Cell contents:

!pip install pandas==1.3.4

Results

Collecting pandas==1.3.4
  Downloading pandas-1.3.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
     |████████████████████████████████| 11.5 MB 14.7 MB/s eta 0:00:01
Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from pandas==1.3.4) (2.8.1)
Requirement already satisfied: pytz>=2017.3 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from pandas==1.3.4) (2021.1)
Requirement already satisfied: numpy>=1.17.3 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from pandas==1.3.4) (1.19.2)
Requirement already satisfied: six>=1.5 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas==1.3.4) (1.15.0)
Installing collected packages: pandas
  Attempting uninstall: pandas
    Found existing installation: pandas 1.2.4
    Uninstalling pandas-1.2.4:
      Successfully uninstalled pandas-1.2.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sparkmagic 0.18.0 requires nose, which is not installed.
hdijupyterutils 0.18.0 requires jupyter>=1, which is not installed.
hdijupyterutils 0.18.0 requires nose, which is not installed.
ibm-watson-machine-learning 1.0.173 requires ibm-cos-sdk==2.7.*, but you have ibm-cos-sdk 2.10.0 which is incompatible.
ibm-watson-machine-learning 1.0.173 requires pandas<1.3.0,>=0.24.2, but you have pandas 1.3.4 which is incompatible.
autoai-libs 1.12.13 requires pandas<1.3,>=0.24.2, but you have pandas 1.3.4 which is incompatible.
Successfully installed pandas-1.3.4

Now replay the cell using ibmcloudsql played earlier:

cell contents

traffic_location = sqlClient.run_sql(query)
print(traffic_location)

Results:

...
    l.latitude != 0.0000

              id        date  time         area  age sex  latitude  longitude
0      111521708  12/03/2011  1800  N Hollywood   28   F   34.1867  -118.4137
1      111516183  08/17/2011  1925  N Hollywood   21   F   34.1867  -118.4052
2      111517569  09/13/2011  1825  N Hollywood   23   M   34.1577  -118.3615
3      111519647  10/24/2011  1930  N Hollywood   27   M   34.1577  -118.3855
4      111521454  11/28/2011  1730  N Hollywood   33   F   34.1763  -118.3965
...          ...         ...   ...          ...  ...  ..       ...        ...
42591  171224507  10/07/2017  1740  77th Street   29   M   33.9646  -118.2918
42592  171219904  08/07/2017  1900  77th Street   24   M   33.9932  -118.3171
42593  171219971  08/08/2017  1925  77th Street   30   M   33.9643  -118.2871
42594  171227417  11/17/2017  1800  77th Street   33   F   33.9890  -118.3294
42595  171222354  09/01/2017  1715  77th Street   24   F   33.9890  -118.3308

torsstei commented 2 years ago

@powellquiring I am not sure if I fully grasp the issue. ibmcloudsql does express its prerequisite to pandas and it is also installed as you can see in the output of pip install ibmcloudsql. It the issue seems to be rather with ibm-watson-machine-learning, which is already installed and apparently doesn't like the pandas version that is installed (which ibmcloudsql doesn't actually control entirely. It only has a pandas minimum version that it prereqs: https://github.com/IBM-Cloud/sql-query-clients/blob/master/Python/setup.py#L50) and which is perfectly within the range of by ibm-watson-machine-learning required versions. Not sure why you installed pandas==1.3.4 version explicitly, which is incompatible with ibm-watson-machine-learning requirements.

The other problem seems to be that ibm-watson-machine-learning seems to have a hard requirement on ibm-cos-sdk==2.7.* and ibmcloudsql requires 2.10.0. I think the first approach should be to check with ibm-watson-machine-learning why they have hard prerequisite on older ibm-cos-sdk version and ask them to lift this.

powellquiring commented 2 years ago

I installed 1.3.4 because of the exception mentioned:

...
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/wsuser/ipykernel_122/2163707681.py in <module>
     42 print(query)
     43 
---> 44 traffic_location = sqlClient.run_sql(query)
     45 print(traffic_location)

/opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages/ibmcloudsql/SQLQuery.py in run_sql(self, sql_text, pagesize)
   1665                 pprint.pprint(details)
   1666                 raise e
-> 1667         return self.get_result(job_id)
   1668 
   1669     def run(self, pagesize=None, get_result=False):

/opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages/ibmcloudsql/SQLQuery.py in get_result(self, jobId, pagenumber)
    942                         body.__iter__ = types.MethodType(self.__iter__, body)
    943 
--> 944                     partition_df = pd.read_csv(body, on_bad_lines='skip')
    945 
    946                 elif result_format == "parquet":

TypeError: read_csv() got an unexpected keyword argument 'on_bad_lines'

torsstei commented 2 years ago

OK @powellquiring . I added toleration of pandas versions that don't support 'on_bad_lines' (https://github.com/IBM-Cloud/sql-query-clients/blob/master/Python/ibmcloudsql/SQLQuery.py#L944). Published as ibmcloudsql v 0.5.1. Hope this will help you to avoid the conflict with the ibm-watson-machine-learning prerequisite limitations.

powellquiring commented 2 years ago

Is there a test that verifies ibmcloudsql with the set of prerequisite module versions that are allowed?

alexlang74 commented 2 years ago

I tried this today in PROD. There is no pandas conflict anymore, but ibmcloudsql requires a new numpy version which leads to conflicts. From the install stack:

Collecting ibmcloudsql
  Downloading ibmcloudsql-0.5.9.tar.gz (57 kB)
  ...
Collecting numpy>=1.22.2
   Downloading numpy-1.23.1-cp39....

ERROR:...
scipy 1.7.3 requires numpy<1.23.0,>=1.16.5, but you have numpy 1.23.1 which is incompatible.
numba 0.54.1 requires numpy<1.21,>=1.17, but you have numpy 1.23.1 which is incompatible.
autoai-ts-libs 1.1.9 requires numpy<1.23,>=1.19.2; python_version >= "3.9", but you have numpy 1.23.1 which is incompatible.
autoai-libs 1.13.2 requires numpy<1.22,>=1.19.2; python_version >= "3.9", but you have numpy 1.23.1 which is incompatible.

alexlang74 commented 2 years ago

@torsstei is there a way you can relax the numpy requirement for ibmcloudsql ? In Q4 this year, we will provide new Notebook images that contain numpy 1.23, but for the existing ones, we stay at 1.20...

torsstei commented 2 years ago

@alexlang74 OK, I just ran some tests with numpy 1.20.3 and ibmcloudsql seems to work fine. So I have relaxed the prerequisite to 1.20.3 and published ibmcloudsql 0.5.10.

alexlang74 commented 2 years ago

@torsstei Fast as a shark! Just tried, and no more pip errors whatsoever!

alexlang74 commented 2 years ago

@powellquiring Can you try for yourself and close the issue?

powellquiring commented 2 years ago

I trust you

IBM-Cloud / sql-query-clients

python ibmcloudsql library has the wrong dependency on pandas #123