search

sasha-polev / aerospark

Aerospike Spark Connector
Apache License 2.0
35 stars 38 forks source link

Limit functionality is missing on aeroapark connector. #7

Closed thefishermanhacker closed 8 years ago

thefishermanhacker commented 8 years ago

Hi, Is there away to use limit keyword on queries? If not, is there a feature request?

Thanks,

sasha-polev commented 8 years ago

Hi

You can use LIMIT in Spark SQL queries, the iteration will just stop and query will be terminated. Data will never be read from Aerospike into Spark.

AQL (Aerospike SQL) does not use LIMIT clause, so you cant use it when declaring datasource

Hope that helps.

Thanks Sasha

thefishermanhacker commented 8 years ago

Thanks for the response. So I need to create: 1) application/driver that runs over Spark slave, which run aql queries and stored them in the cache. 2) run Spark SQL on my client which run query against the cached view.

Thanks,

sasha-polev commented 8 years ago

Well, not really.

it really depends if you use Spark SQL or pure Spark, but everything is defined in the driver and then executed by workers. Its just you first define initial RDD/dataframe with AQL and then can filter it down with SQL or take N statements. Only required number of rows will be actually read from Aerospike.

Thanks Sasha

On 9 November 2015 at 21:29, thefishermanhacker notifications@github.com wrote:

Thanks for the response. So I need to create: 1) application/driver that runs over Spark slave, which run aql queries and stored them in the cache. 2) run Spark SQL on my client which run query against the cached view.

Thanks,

— Reply to this email directly or view it on GitHub https://github.com/sasha-polev/aerospark/issues/7#issuecomment-155104440 .

This message contains confidential information. If you are not the intended recipient, please notify the sender and delete the message immediately. One Point Consulting Ltd is a limited liability company registered in England and Wales (registered number 05516457) and whose registered office is at Business Environment, 1 Olympic Way, Wembley, London, HA90NP, UK.

thefishermanhacker commented 8 years ago

Cool, Thanks for your help! I managed to do so!

Sasi