bigquery.datalab-migration.samples_test: test_client_library_load_table_from_dataframe failed

[flaky-bot[bot]]

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and I will stop commenting.

---

commit: c3a65998973d7010d0f77edb84034f05ed40adeb buildURL: Build Status, Sponge status: failed

Test output

Traceback (most recent call last):
  File "/workspace/bigquery/datalab-migration/samples_test.py", line 409, in test_client_library_load_table_from_dataframe
    load_job = client.load_table_from_dataframe(dataframe, table_ref)
  File "/workspace/bigquery/datalab-migration/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 2333, in load_table_from_dataframe
    parquet_compression=parquet_compression,
  File "/workspace/bigquery/datalab-migration/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 472, in dataframe_to_parquet
    arrow_table = dataframe_to_arrow(dataframe, bq_schema)
  File "/workspace/bigquery/datalab-migration/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 436, in dataframe_to_arrow
    bq_to_arrow_array(get_column_or_index(dataframe, bq_field.name), bq_field)
  File "/workspace/bigquery/datalab-migration/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 218, in bq_to_arrow_array
    return pyarrow.Array.from_pandas(series, type=arrow_type)
  File "pyarrow/array.pxi", line 887, in pyarrow.lib.Array.from_pandas
  File "pyarrow/array.pxi", line 292, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 107, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: Did not pass numpy.dtype object

[flaky-bot[bot]]

Looks like this issue is flaky. :worried:

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (c3a65998973d7010d0f77edb84034f05ed40adeb), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[shollyman]

Same as #5313, this broke with pyarrow3.

[tswast]

Strange that it's failing in the DataFrame to arrow conversion. I'll have to dig in deeper to see why our unit tests wouldn't have caught this. Perhaps we aren't testing with enough data types

[tswast]

Just to verify, this failure did happen after https://github.com/GoogleCloudPlatform/python-docs-samples/pull/5296 was merged.

[tswast]

Tests are passing locally on Mac. Will try on my Linux box next. Maybe it's a problem with a specific wheel?

[tswast]

Passes locally on linux with Python 3.8. I see the test failures are for Python 3.7, but my local 3.7 env is busted, so still looking into that.

[tswast]

Tests pass locally. I'm closing as can't reproduce. Will investigate further if flaky-bot reopens.

(nox) swast@tims-cloud.c ~/src/python-docs-samples/bigquery/datalab-migration% nox -s py-3.7 -r
nox > Running session py-3.7
nox > Re-using existing virtual environment at .nox/py-3-7.
nox > python -m pip install -r requirements.txt
nox > python -m pip install -r requirements-test.txt                                                                          nox > pytest --junitxml=sponge_log.xml                                                                                        ==================================================== test session starts =====================================================
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /usr/local/google/home/swast/src/python-docs-samples
/bigquery/datalab-migration/.nox/py-3-7/bin/python
cachedir: .pytest_cache
rootdir: /usr/local/google/home/swast/src/python-docs-samples, configfile: pytest.ini                                         collected 13 items
samples_test.py::test_datalab_query_magic PASSED                                                                       [  7%]
samples_test.py::test_client_library_query_magic PASSED                                                                [ 15%]
samples_test.py::test_datalab_query_magic_results_variable PASSED                                                      [ 23%]
samples_test.py::test_client_library_query_magic_results_variable PASSED                                               [ 30%]
samples_test.py::test_datalab_magic_parameterized_query PASSED                                                         [ 38%]
samples_test.py::test_client_library_magic_parameterized_query PASSED                                                  [ 46%]
samples_test.py::test_datalab_list_tables_magic PASSED                                                                 [ 53%]
samples_test.py::test_datalab_query PASSED                                                                             [ 61%]
samples_test.py::test_client_library_query PASSED                                                                      [ 69%]
samples_test.py::test_datalab_load_table_from_gcs_csv PASSED                                                           [ 76%]
samples_test.py::test_client_library_load_table_from_gcs_csv PASSED                                                    [ 84%]
samples_test.py::test_datalab_load_table_from_dataframe PASSED                                                         [ 92%]
samples_test.py::test_client_library_load_table_from_dataframe PASSED                                                  [100%]

====================================================== warnings summary ======================================================
.nox/py-3-7/lib/python3.7/site-packages/future/standard_library/__init__.py:65
  /usr/local/google/home/swast/src/python-docs-samples/bigquery/datalab-migration/.nox/py-3-7/lib/python3.7/site-packages/future/standard_library/__init__.py:65: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

bigquery/datalab-migration/samples_test.py::test_datalab_query_magic
bigquery/datalab-migration/samples_test.py::test_datalab_query_magic_results_variable
bigquery/datalab-migration/samples_test.py::test_datalab_magic_parameterized_query
  /usr/local/google/home/swast/src/python-docs-samples/bigquery/datalab-migration/.nox/py-3-7/lib/python3.7/site-packages/google/datalab/utils/commands/_utils.py:377: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    config = yaml.load(content)

bigquery/datalab-migration/samples_test.py::test_client_library_query_magic
bigquery/datalab-migration/samples_test.py::test_client_library_query_magic_results_variable
bigquery/datalab-migration/samples_test.py::test_client_library_magic_parameterized_query
  /usr/local/google/home/swast/src/python-docs-samples/bigquery/datalab-migration/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/bigquery/job/query.py:1327: UserWarning: A progress bar was requested, but there was an error loading the tqdm library. Please install tqdm to use the progress bar functionality.
    query_result = wait_for_query(self, progress_bar_type)

bigquery/datalab-migration/samples_test.py::test_client_library_query_magic
bigquery/datalab-migration/samples_test.py::test_client_library_query_magic_results_variable
bigquery/datalab-migration/samples_test.py::test_client_library_magic_parameterized_query
  /usr/local/google/home/swast/src/python-docs-samples/bigquery/datalab-migration/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/bigquery/table.py:1770: UserWarning: A progress bar was requested, but there was an error loading the tqdm library. Please install tqdm to use the progress bar functionality.
    create_bqstorage_client=create_bqstorage_client,

bigquery/datalab-migration/samples_test.py::test_datalab_magic_parameterized_query
  /usr/local/google/home/swast/src/python-docs-samples/bigquery/datalab-migration/.nox/py-3-7/lib/python3.7/site-packages/google/datalab/utils/commands/_utils.py:332: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    config = yaml.load(config)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
----- generated xml file: /usr/local/google/home/swast/src/python-docs-samples/bigquery/datalab-migration/sponge_log.xml -----
========================================= 13 passed, 11 warnings in 74.03s (0:01:14) =========================================
nox > Session py-3.7 was successful.