Open MaximVetlugin opened 2 years ago
Parallelism :
I believe currently these requests are single threaded in Egeria itself. If the query is across multiple repos an enterprise connector could implement this in parallel - but our current one does not. In terms of xtdb itself, behaviour will depend on how that itself is configured .
Overall Performance:
How much data are you retrieving? Can you use a more tightly scoped call? Would indexing help (there is work underway to add an index - but for searches vs explicit gets)? Beyond that I think capturing profile information would help to understand any hotspots
Just chiming in to mention that XTDB happily supports independent queries running with multiple threads & cores, but within the scope of a single query execution is blocking and single-threaded.
Multiple threads can coordinate and execute related queries in parallel using a stable (consistent) tx-id basis, so complex parallel designs can be implemented on top, but note that a db
should be created per thread.
Hi, Please, help me to understand where is a problem with performance. Thank you!
I send this request to Egeria
{{baseURL}}/servers/{{server}}/open-metadata/repository-services/users/admin/enterprise/instances/entities/from-entity/{guid}/by-neighborhood?level=2
body { "class" : "EntityNeighborhoodFindRequest", "limitResultsByStatus" : [ "ACTIVE" ], "relationshipTypeGUIDs" : [ "f00df00d-0000-0000-0000-000000000000", "babebabe-0000-0000-0000-000000000000", "e222e222-0000-0000-0000-000000000000" ] }
(we are using our Entity types)
Entity example in responce
It takes from 30 sec to 6 min.
And every time only one cpu core uses!![image](https://user-images.githubusercontent.com/36268863/175898444-d9304622-444d-45b2-8e0b-64f1bb8050c1.png)
Local repository config
{ "class": "Connection", "connectorType": { "class": "ConnectorType", "connectorProviderClassName": "org.odpi.egeria.connectors.juxt.xtdb.repositoryconnector.XtdbOMRSRepositoryConnectorProvider" }, "configurationProperties": { "xtdbConfigEDN": "{:xtdb.rocksdb/block-cache {:xtdb/module xtdb.rocksdb/->lru-block-cache :cache-size 1073741824}:xtdb/index-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store :block-cache :xtdb.rocksdb/block-cache :db-dir \"data/servers/{{server}}/rdb-index\"}} :kafka-config {:xtdb/module xtdb.kafka/->kafka-config :bootstrap-servers \"{{kafka-cluster}}\"} :xtdb/tx-log {:xtdb/module xtdb.kafka/->tx-log :kafka-config :kafka-config :tx-topic-opts {:topic-name \"{{tx-topic-name}}\" :replication-factor {{tx-topic-replication-factor}}} :poll-wait-duration \"PT0.050S\"} :xtdb.lucene/lucene-store {:db-dir \"data/servers/{{server}}/lucene\" :indexer {:xtdb/module xtdb.lucene.egeria/->egeria-indexer} :analyzer {:xtdb/module xtdb.lucene.egeria/->ci-analyzer}} :xtdb/document-store {:xtdb/module xtdb.jdbc/->document-store :connection-pool {:dialect {:xtdb/module xtdb.jdbc.psql/->dialect} :pool-opts {:maximumPoolSize 10} :db-spec {:jdbcUrl \"jdbc:postgresql://{{PostgreSQL}}/{{PostgreSQLDBName}}?user={{PostgreSQLLogin}}&password={{PostgreSQLPassword}}&ApplicationName={{PostgreSQLApplicationName}}\"}}:poll-sleep-duration \"PT1S\"}}", "luceneRegexes": true, "syncIndex": true } }
JVM parameters /bin/java -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=9101:/opt/jmx_exporter/config.yaml -XX:MaxDirectMemorySize=1024m -Xms32g -Xmx32g -Dloader.path=/opt/platform/lib -jar /opt/platform/server-chassis-spring.jar