search

Octan-Labs / pipeline

1 stars 0 forks source link

Cannot index ETH #233

Open tuannm1006 opened 9 months ago

tuannm1006 commented 9 months ago

Indexer cannot insert data to table with eth_ prefix.

[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO - 2023-11-15 00:11:22,553 - root [INFO] - Exporting with ClickHouseItemExporter
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO - Traceback (most recent call last):
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     return _run_code(code, main_globals, None,
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     exec(code, run_globals)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/ethereum-etl/ethereumetl/__main__.py", line 26, in <module>
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     cli()
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     return self.main(*args, **kwargs)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     rv = self.invoke(ctx)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     return ctx.invoke(self.callback, **ctx.params)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     return __callback(*args, **kwargs)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/ethereum-etl/ethereumetl/cli/export_all.py", line 58, in export_all
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     export_all_common(get_partitions(start, end, partition_batch_size, provider_uri, worker_job_index, first_worker_partition_index, partition_to_hour),
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/ethereum-etl/ethereumetl/jobs/export_all_common.py", line 78, in export_all_common
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     streamer_adapter.export_all(start_block, end_block)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/ethereum-etl/ethereumetl/streaming/eth_streamer_adapter.py", line 59, in export_all
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     self._calculate_offset_and_export(sort_by(enriched_transactions, ('block_number', 'transaction_index')))
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/ethereum-etl/ethereumetl/streaming/eth_streamer_adapter.py", line 145, in _calculate_offset_and_export
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     self.item_exporter.export_items(items)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/ethereum-etl/blockchainetl/jobs/exporters/clickhouse_item_exporter.py", line 78, in export_items
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     self.connection.insert(table, data=table_data, column_names=self.tables[table].column_names,
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/site-packages/clickhouse_connect/driver/client.py", line 554, in insert
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     return self.data_insert(context)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 251, in data_insert
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     response = self._raw_request(block_gen, params, headers, error_handler=error_handler, server_wait=False)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 427, in _raw_request
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     error_handler(response)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 237, in error_handler
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     self._error_handler(resp)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -   File "/usr/local/lib/python3.9/site-packages/clickhouse_connect/driver/httpclient.py", line 353, in _error_handler
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -     raise OperationalError(err_str) if retried else DatabaseError(err_str) from None
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO - clickhouse_connect.driver.exceptions.DatabaseError: :HTTPDriver for https://oa172rbtqk.ap-southeast-1.aws.clickhouse.cloud:8443 returned response code 404)
[2023-11-15, 00:11:25 UTC] {pod_manager.py:367} INFO -  Code: 47. DB::Exception: Missing columns: 'contract' while processing query: 'SELECT address, contract FROM default.eth_L2S_reputation_score WHERE (snapshot_block_number = (identity(_CAST(0, 'Nullable(UInt64)')) AS _subquery1)) AND (contract = true)', required columns: 'address' 'snapshot_block_number' 'contract', maybe you meant: 'address' or 'snapshot_block_number'. (UNKNOWN_IDENTIFIER) (version 23.8.1.41669 (official build))