Why view created without WHERE condition, when load from AQL query?

[EbiousVi]

Why view created without WHERE condition, when load from AQL query?

We use:

• arangodb-spark-datasource-3.2_2.12
• spark 3.3.0
• scala 2.12
• Java 8

            Map<String, String> opts = new HashMap<>();
            opts.put("query", "for user in users return user");

            spark
                    .read()
                    .format("com.arangodb.spark")
                    .options(opts)
                    .load()
                    .where("`active` == false")
                    .createOrReplaceTempView("v");

            Dataset<Row> sql = spark.sql("select `active` from v");
            sql.show();// returns data without where condition `active` == false

Thanks!

[rashtao]

Thanks for reporting, I verified that this is a bug in the connector.

Note that the expected behavior in this case would be applying the filter on the Spark side and not pushing it down to ArangoDB. Therefore all documents returned by the query will be transferred. As stated in the documentation:

Predicate and projection pushdowns are only performed while reading an ArangoDB collection (set by the table configuration parameter). In case of a batch read from a custom query (set by the query configuration parameter), no pushdown optimizations are performed.

For this specific case, reading from the users collection would be a better choice (i.e. setting the option table: users). In this case the filter would be pushed down to ArangoDB.

[EbiousVi]

Sorry, maybe I phrased my question wrong.

Most likely, my problem is not related to pushdown, but that there is no way to apply a filter at the spark level. Spark sql does not filter data by the generated view from the AQL query. My AQL query is abstract, this behavior is manifested with any query.

not working example

working example, with .persist() it works as expected