search

DataBiosphere / azul

Metadata indexer and query service used for AnVIL, HCA, LungMAP, and CGP
Apache License 2.0
7 stars 2 forks source link

ValueError with version 3.14.0 of google-cloud-bigquery #6709

Open achave11-ucsc opened 3 days ago

achave11-ucsc commented 3 days ago

Causing indexer.test_tdr.TestTDRHCAPlugin.test_subgraph_stitching to error.

Error
Traceback (most recent call last):
  File "/Users/achave11/Pycharm/Azul/azul.sc/test/indexer/test_tdr.py", line 341, in test_subgraph_stitching
    self._test_fetch_bundle(bundle, load_tables=True)
  File "/Users/achave11/Pycharm/Azul/azul.sc/test/indexer/test_tdr.py", line 353, in _test_fetch_bundle
    emulated_bundle = plugin.fetch_bundle(test_bundle.fqid)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/achave11/Pycharm/Azul/azul.sc/src/azul/plugins/repository/tdr.py", line 197, in fetch_bundle
    bundle = self._emulate_bundle(bundle_fqid)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/achave11/Pycharm/Azul/azul.sc/src/azul/plugins/repository/tdr_hca/__init__.py", line 323, in _emulate_bundle
    entities, root_entities, links_jsons = self._stitch_bundles(bundle)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/achave11/Pycharm/Azul/azul.sc/src/azul/plugins/repository/tdr_hca/__init__.py", line 394, in _stitch_bundles
    upstream = self._find_upstream_bundles(source, all_dangling_inputs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/achave11/Pycharm/Azul/azul.sc/src/azul/plugins/repository/tdr_hca/__init__.py", line 511, in _find_upstream_bundles
    for row in rows:
  File "/Users/achave11/Pycharm/Azul/azul.sc/.venv/lib/python3.11/site-packages/google/api_core/page_iterator.py", line 209, in _items_iter
    for item in page:
  File "/Users/achave11/Pycharm/Azul/azul.sc/.venv/lib/python3.11/site-packages/google/api_core/page_iterator.py", line 131, in __next__
    result = self._item_to_value(self._parent, item)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/achave11/Pycharm/Azul/azul.sc/.venv/lib/python3.11/site-packages/google/cloud/bigquery/table.py", line 2962, in _item_to_row
    _helpers._row_tuple_from_json(resource, iterator.schema),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/achave11/Pycharm/Azul/azul.sc/.venv/lib/python3.11/site-packages/google/cloud/bigquery/_helpers.py", line 300, in _row_tuple_from_json
    row_data.append(_field_from_json(cell["v"], field))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/achave11/Pycharm/Azul/azul.sc/.venv/lib/python3.11/site-packages/google/cloud/bigquery/_helpers.py", line 276, in _field_from_json
    return converter(resource, field)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/achave11/Pycharm/Azul/azul.sc/.venv/lib/python3.11/site-packages/google/cloud/bigquery/_helpers.py", line 142, in _timestamp_from_json
    return _datetime_from_microseconds(int(value))
                                       ^^^^^^^^^^
ValueError: invalid literal for int() with base 10: '1597094666.174274'

At a glance, and because of the Azul source code site … https://github.com/DataBiosphere/azul/blob/efe016f5b3ccbe53f369cfd0444e4494a50c3eee/src/azul/plugins/repository/tdr_hca/__init__.py#L340

… it didn't appear to be a trivial fix.

achave11-ucsc commented 3 days ago

Limiting the updated google-cloud-bigquery version in #6511.