search

aperture-data / aperturedb-python

ApertureDB Python Client
Other
7 stars 3 forks source link

cli ingestion failing /dist-packages/aperturedb/cli/transact.py:100 #506

Open luisremis opened 1 month ago

luisremis commented 1 month ago

Seing on faces: https://benchmarks.aperturedata.io/files/adb-performance-evaluation/2024-10-12/celeba-ingest/1728710824_faces_prod_gcp/

Total queries executed: 1
Avg Query time (s): 1.143507480621338
Query time std: 0.0
Avg Query Throughput (q/s): 0.8745023683243756
Overall insertion throughput (element/s): 0.8735228869734146
Total inserted elements: 1
Total successful commands: 1
=================================================
[05:26:31] No transformer applied                                  ingest.py:213
           <aperturedb.Connector.Connector object at               ingest.py:215
           0x7fed99884550>                                                      
sample_count 4

Progress: 100.00% - ETA(s): 0.00
============ ApertureDB Loader Stats ============
Total time (s): 1.160581350326538
Total queries executed: 1
Avg Query time (s): 1.1589436531066895
Query time std: 0.0
Avg Query Throughput (q/s): 0.8628547188807483
Overall insertion throughput (element/s): 3.446548575741434
Total inserted elements: 4
Total successful commands: 4
=================================================
[05:26:37] No transformer applied                                  ingest.py:213
           <aperturedb.Connector.Connector object at               ingest.py:215
           0x7f602b0f4550>                                                      
sample_count 2

Progress: 100.00% - ETA(s): 0.00
============ ApertureDB Loader Stats ============
Total time (s): 1.1496906280517578
Total queries executed: 1
Avg Query time (s): 1.1485238075256348
Query time std: 0.0
Avg Query Throughput (q/s): 0.8706828656468057
Overall insertion throughput (element/s): 1.739598419958558
Total inserted elements: 2
Total successful commands: 6
=================================================
[05:26:43] No transformer applied                                  ingest.py:213
           <aperturedb.Connector.Connector object at               ingest.py:215
           0x7f5735ea4550>                                                      
sample_count 3

Progress: 100.00% - ETA(s): 0.00
============ ApertureDB Loader Stats ============
Total time (s): 1.1463963985443115
Total queries executed: 1
Avg Query time (s): 1.1451160907745361
Query time std: 0.0
Avg Query Throughput (q/s): 0.8732739047650774
Overall insertion throughput (element/s): 2.616895869360184
Total inserted elements: 3
Total successful commands: 9
=================================================
Executing query [{'FindImage': {'constraints': {'Blurry': ['!=', None]}, 'results': {'count': True}}}] for celebA.csv
Expected count [celebA.csv]: 202599, Actual count: 202468
Executing query [{'FindDescriptor': {'set': 'ViT-B/16', 'results': {'count': True}}}] for celebA.csv_clip_pytorch_embeddings_metadata.adb.csv
Expected count [celebA.csv_clip_pytorch_embeddings_metadata.adb.csv]: 202468, Actual count: 202468
Executing query [{'FindDescriptor': {'set': 'ViT-B/16', 'results': {'count': True}}}] for celebA.csv_clip_pytorch_embeddings_connection.adb.csv
Expected count [celebA.csv_clip_pytorch_embeddings_connection.adb.csv]: 202468, Actual count: 202468
Executing query [{'FindDescriptor': {'set': 'facenet_pytorch_embeddings', 'results': {'count': True}}}] for celebA.csv_facenet_pytorch_embeddings_metadata.adb.csv
Expected count [celebA.csv_facenet_pytorch_embeddings_metadata.adb.csv]: 202468, Actual count: 204015
Executing query [{'FindDescriptor': {'set': 'facenet_pytorch_embeddings', 'results': {'count': True}}}] for celebA.csv_facenet_pytorch_embeddings_connection.adb.csv
Expected count [celebA.csv_facenet_pytorch_embeddings_connection.adb.csv]: 202468, Actual count: 204015
Executing query [{'FindImage': {'constraints': {'Blurry': ['==', None]}, 'results': {'count': True}}}] for hqimages.adb.csv
Expected count [hqimages.adb.csv]: 1547, Actual count: 1547
Executing query [{'FindPolygon': {'results': {'count': True}}}] for hqpolygons.adb.csv
Expected count [hqpolygons.adb.csv]: 17697, Actual count: 17697
Executing query [{'FindBoundingBox': {'results': {'count': True}}}] for hqbboxes.adb.csv
Expected count [hqbboxes.adb.csv]: 17697, Actual count: 17697
Results:[False, True, True, False, False, True, True, True]
Creating read-only user
fuse not found for this env. This is not critical for adb to continue.
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/lib/python3.10/dist-packages/aperturedb/cli/transact.py:100 in    │
│ from_json_file                                                               │
│                                                                              │
│    97 │   │   transaction = json.loads(inputstream.read())                   │
│    98 │   │   old_argv = sys.argv[1:]                                        │
│    99 │   │   sys.argv[1:] = [output_path]                                   │
│ ❱ 100 │   │   output_types[output_type](client, transaction, output_path=out │
│   101 │   │   sys.argv[1:] = old_argv                                        │
│   102                                                                        │
│                                                                              │
│ /usr/local/lib/python3.10/dist-packages/aperturedb/cli/transact.py:40 in     │
│ dump_to_stdout                                                               │
│                                                                              │
│    37 def dump_to_stdout(client: Connector, transaction: dict, **kwargs):    │
│    38 │   from aperturedb.CommonLibrary import execute_query                 │
│    39 │                                                                      │
│ ❱  40 │   result, response, blobs = execute_query(                           │
│    41 │   │   client=client,                                                 │
│    42 │   │   q=transaction,                                                 │
│    43 │   │   blobs=])                                                      │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: execute_query() got an unexpected keyword argument 'q'
App Done.
Results available at: https://benchmarks.aperturedata.io/files/adb-performance-evaluation/2024-10-12/celeba-ingest/1728710824_faces_prod_gcp/
Dashboard: https://faces.datasets.gcp.cloud.aperturedata.io/grafana/d/mPHHiqbnk/aperturedb-connectivity-status?from=1728708585000&to=1728710820000&var-job_filter=job%7C%3D%7Caperturedb&var-pod_ip=All&var-node_filter=pod_ip%7C%3D~%7C&orgId=1&refresh=5s
Error with app: celeba-ingest. Exiting with error code: 1
Stream closed EOF for deployments/faces-ingest-28811805-fjs9q (faces-ingest)
gsaluja9 commented 3 hours ago

@luisremis . if you change from using kwargs to postitional args, this will resolve. Is this a problem still?