Open tswast opened 7 months ago
Many tests are failing in the presubmit session.
nox > python -m pip freeze aiohttp==3.9.1 aiosignal==1.3.1 appdirs==1.4.4 argcomplete==3.2.1 asttokens==2.4.1 asyncmock==0.4.2 atpublic==4.0 attrs==23.2.0 bidict==0.22.1 # Editable Git install with no remote (bigframes==0.19.2) -e /tmpfs/src/github/python-bigquery-dataframes black==23.12.1 bleach==6.1.0 cachetools==5.3.2 certifi==2023.11.17 cffi==1.16.0 cfgv==3.4.0 charset-normalizer==3.3.2 click==8.0.4 click-plugins==1.1.1 cligj==0.7.2 cloudpickle==3.0.0 colorlog==6.8.0 comm==0.2.1 coverage==7.4.0 cryptography==41.0.7 db-dtypes==1.2.0 decorator==5.1.1 distlib==0.3.8 docstring-inheritance==2.1.2 docutils==0.20.1 et-xmlfile==1.1.0 exceptiongroup==1.2.0 execnet==2.0.2 executing==2.0.1 filelock==3.13.1 fiona==1.9.5 frozenlist==1.4.1 fsspec==2023.12.2 gcp-docuploader==0.6.5 gcp-releasetool==1.17.0 gcsfs==2023.12.2.post1 geopandas==0.14.2 google-api-core==1.34.0 google-auth==2.26.2 google-auth-oauthlib==1.2.0 google-cloud-bigquery @ git+[https://github.com/googleapis/python-bigquery.git@1f96439b3dbd27f11be5e2af84f290ec6094d0a4](https://www.google.com/url?q=https://github.com/googleapis/python-bigquery.git@1f96439b3dbd27f11be5e2af84f290ec6094d0a4&sa=D) google-cloud-bigquery-connection==1.14.0 -e git+[https://github.com/googleapis/python-bigquery-storage.git@fb0d6f07d4b7582cf01ec98fee3081ec34995603#egg=google_cloud_bigquery_storage](https://www.google.com/url?q=https://github.com/googleapis/python-bigquery-storage.git@fb0d6f07d4b7582cf01ec98fee3081ec34995603%23egg%3Dgoogle_cloud_bigquery_storage&sa=D) google-cloud-core==2.4.1 google-cloud-functions==1.14.0 google-cloud-iam==2.13.0 google-cloud-resource-manager==1.11.0 google-cloud-storage==2.14.0 google-cloud-testutils==1.4.0 google-crc32c==1.5.0 google-resumable-media==2.7.0 googleapis-common-protos==1.62.0 greenlet==3.0.3 grpc-google-iam-v1==0.13.0 grpcio==1.60.0 grpcio-status==1.48.2 humanize==4.9.0 ibis-framework==7.1.0 identify==2.5.33 idna==3.6 importlib-metadata==7.0.1 importlib-resources==6.1.1 iniconfig==2.0.0 ipython==8.20.0 ipywidgets==8.1.1 jaraco.classes==3.3.0 jedi==0.19.1 jeepney==0.8.0 Jinja2==3.1.3 joblib==1.3.2 jupyterlab-widgets==3.0.9 keyring==24.3.0 markdown-it-py==3.0.0 MarkupSafe==2.1.4 matplotlib-inline==0.1.6 mdurl==0.1.2 mock==5.1.0 more-itertools==10.2.0 multidict==6.0.4 multipledispatch==1.0.0 mypy-extensions==1.0.0 nh3==0.2.15 nodeenv==1.8.0 nox==2023.4.22 numpy==1.26.3 oauthlib==3.2.2 openpyxl==3.1.2 packaging==23.2 pandas==2.2.0 pandas-gbq==0.20.0 parso==0.8.3 parsy==2.1 pathspec==0.12.1 pexpect==4.9.0 pins==0.8.4 pkginfo==1.9.6 platformdirs==4.1.0 pluggy==1.3.0 pooch==1.8.0 pre-commit==3.6.0 prompt-toolkit==3.0.43 proto-plus==1.23.0 protobuf==3.20.3 ptyprocess==0.7.0 pure-eval==0.2.2 pyarrow==14.0.2 pyarrow-hotfix==0.6 pyasn1==0.5.1 pyasn1-modules==0.3.0 pycparser==2.21 pydata-google-auth==1.8.2 Pygments==2.17.2 PyJWT==2.8.0 pyperclip==1.8.2 pyproj==3.6.1 pytest==7.4.4 pytest-asyncio==0.23.3 pytest-cov==4.1.0 pytest-mock==3.12.0 pytest-retry==1.6.1 pytest-timeout==2.2.0 pytest-xdist==3.5.0 python-dateutil==2.8.2 pytz==2023.3.post1 PyYAML==6.0.1 readme-renderer==42.0 requests==2.31.0 requests-oauthlib==1.3.1 requests-toolbelt==1.0.0 rfc3986==2.0.0 rich==13.7.0 rsa==4.9 scikit-learn==1.4.0 scipy==1.12.0 SecretStorage==3.3.3 shapely==2.0.2 six==1.16.0 SQLAlchemy==2.0.25 sqlglot==19.9.0 stack-data==0.6.3 tabulate==0.9.0 threadpoolctl==3.2.0 tomli==2.0.1 toolz==0.12.0 tqdm==4.66.1 traitlets==5.14.1 twine==4.0.2 typing_extensions==4.9.0 tzdata==2023.4 urllib3==2.1.0 virtualenv==20.25.0 wcwidth==0.2.13 webencodings==0.5.1 widgetsnbextension==4.0.9 xarray==2024.1.0 xxhash==3.4.1 yarl==1.9.4 zipp==3.17.0 ... def test_read_gbq_after_session_start_must_comply_with_default_location( scalars_pandas_df_index, test_data_tables, test_data_tables_tokyo, dataset_id_permanent_tokyo, read_method, query_prefix, ): # Form query as a table name or a SQL depending on the test scenario query_tokyo = test_data_tables_tokyo["scalars"] query = test_data_tables["scalars"] if query_prefix: query_tokyo = f"{query_prefix} {query_tokyo}" query = f"{query_prefix} {query}" # Initially there is no location set in the bigquery options assert not bpd.options.bigquery.location # Starting user journey with anything other than read_gbq*, such as # read_pandas would bind the session to default location US > df = bpd.read_pandas(scalars_pandas_df_index) [tests/system/small/test_pandas_options.py:143](https://cs.corp.google.com/piper///depot/google3/tests/system/small/test_pandas_options.py?l=143): _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [bigframes/pandas/__init__.py:570](https://cs.corp.google.com/piper///depot/google3/bigframes/pandas/__init__.py?l=570): in read_pandas return global_session.with_default_session( [bigframes/core/global_session.py:65](https://cs.corp.google.com/piper///depot/google3/bigframes/core/global_session.py?l=65): in with_default_session return func(get_global_session(), *args, **kwargs) [bigframes/session/__init__.py:849](https://cs.corp.google.com/piper///depot/google3/bigframes/session/__init__.py?l=849): in read_pandas return self._read_pandas(pandas_dataframe, "read_pandas") [bigframes/session/__init__.py:889](https://cs.corp.google.com/piper///depot/google3/bigframes/session/__init__.py?l=889): in _read_pandas load_job = self.bqclient.load_table_from_dataframe( [.nox/system_prerelease/lib/python3.11/site-packages/google/cloud/bigquery/client.py:2713](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/google/cloud/bigquery/client.py?l=2713): in load_table_from_dataframe dataframe.to_parquet( [.nox/system_prerelease/lib/python3.11/site-packages/pandas/util/_decorators.py:333](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/util/_decorators.py?l=333): in wrapper return func(*args, **kwargs) [.nox/system_prerelease/lib/python3.11/site-packages/pandas/core/frame.py:3101](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/core/frame.py?l=3101): in to_parquet return to_parquet( [.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py:480](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py?l=480): in to_parquet impl.write( [.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py:190](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py?l=190): in write table = self.api.Table.from_pandas(df, **from_pandas_kwargs) [pyarrow/table.pxi:3869](https://cs.corp.google.com/piper///depot/google3/pyarrow/table.pxi?l=3869): in pyarrow.lib.Table.from_pandas ??? [.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:613](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=613): in dataframe_to_arrays arrays = [convert_column(c, f) [.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:613](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=613): in <listcomp> arrays = [convert_column(c, f) [.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:600](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=600): in convert_column raise e [.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:594](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=594): in convert_column result = pa.array(col, type=type_, from_pandas=True, safe=safe) [pyarrow/array.pxi:340](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=340): in pyarrow.lib.array ??? [pyarrow/array.pxi:82](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=82): in pyarrow.lib._ndarray_to_array ??? [pyarrow/array.pxi:70](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=70): in pyarrow.lib._ndarray_to_type ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E pyarrow.lib.ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column geography_col with type geometry') [pyarrow/error.pxi:91](https://cs.corp.google.com/piper///depot/google3/pyarrow/error.pxi?l=91): ArrowTypeError ... =========================== short test summary info ============================ FAILED tests/system/small/test_dataframe.py::test_dataframe_bool_aggregates[all_axis0] FAILED tests/system/small/test_dataframe.py::test_dataframe_bool_aggregates[any_axis0] FAILED tests/system/small/test_dataframe.py::test_df_pivot[values2-int64_too-columns2] FAILED tests/system/small/test_groupby.py::test_dataframe_groupby_analytic[cumprod] FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq-on-table-name] FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq-on-sql] FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq_table-on-table-name] FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq_query-on-sql] FAILED tests/system/small/test_series.py::test_groupby_window_ops[cumprod] - ... FAILED tests/system/small/test_dataframe.py::test_to_pandas_downsampling_option_override FAILED tests/system/small/test_series.py::test_series_add_prefix - AssertionE... FAILED tests/system/small/test_series.py::test_series_add_suffix - AssertionE... FAILED tests/system/small/test_session.py::test_read_pandas - pyarrow.lib.Arr... FAILED tests/system/small/test_session.py::test_read_pandas_multi_index - pya... FAILED tests/system/small/test_session.py::test_read_pandas_rowid_exists_adds_suffix FAILED tests/system/small/test_series.py::test_string_astype_int - AssertionE... FAILED tests/system/small/test_session.py::test_read_pandas_tokyo - pyarrow.l... FAILED tests/system/small/test_session.py::test_read_csv_local_default_engine[default_sep] FAILED tests/system/small/test_session.py::test_read_csv_gcs_default_engine FAILED tests/system/small/test_session.py::test_read_csv_local_default_engine[custom_sep] 20 failed, 1599 passed, 21 skipped, 7 xfailed, 804 warnings in 468.88s (0:07:48) nox > Command py.test --quiet -n=20 --timeout=600 --junitxml=small_prerelease_3.11_sponge_log.xml --cov=bigframes --cov=tests/system/small --cov-append --cov-config=.coveragerc --cov-report=term-missing --cov-fail-under=0 tests/system/small failed with exit code 1 nox > Session system_prerelease failed. nox > Ran multiple sessions: nox > * unit_prerelease: success nox > * system_prerelease: failed
https://fusion2.corp.google.com/invocations/f5604d0d-929d-456f-ab59-b7e292081af2/targets/bigframes%2Fpresubmit%2Fe2e/log
The issue could be solved by a fix in python client change: https://github.com/googleapis/python-bigquery/pull/1796
Many tests are failing in the presubmit session.
https://fusion2.corp.google.com/invocations/f5604d0d-929d-456f-ab59-b7e292081af2/targets/bigframes%2Fpresubmit%2Fe2e/log