search

unipop-graph / unipop

Data Integration Graph
Apache License 2.0
204 stars 35 forks source link

JDBC - smart table union filter. #49

Closed GurRonen closed 8 years ago

GurRonen commented 8 years ago

In order to improve performance when querying SQL databases, we can filter out tables from the union by analysis of which tables conform to the PredicatesHolder and its 'must-have' fields, as any tables that do not have those fields can be cut out.

GurRonen commented 8 years ago

in the search implementation, as seen below currently the query is sent out to all tables, after a union is done between them. this union can be mitigated if we filter out by schema the tables that do not conform to the predicate holder.

Iterator<Condition> conditions = this.predicatesTranslator.translate(allPredicates).iterator();
        Iterator<String> tables = schemas.stream().map(RowSchema::getTable).iterator();
        String firstTable = tables.next();
        SelectWhereStep step = createSqlQuery(columnsToRetrieve, firstTable);

        tables.forEachRemaining(table -> step
                .unionAll(createSqlQuery(columnsToRetrieve, table)));
rmagen commented 8 years ago

Closing as this issue isn't relevant anymore.