Open tlnagy opened 4 years ago
The tabular query commands like @select
expect a stream of named tuples, but here you are getting a stream of SQLite.Row
elements instead. If you convert to a DataFrame
first then it will iterate named tuples, so that is why it works in that case.
Two ideas: you could either try to convert these Row
instances into named tuples somehow. I don't know if they have that ability built in, but maybe something like @map(NamedTuple(_))
or @map({_...})
might work? I haven't tried that.
You could also give https://github.com/queryverse/QuerySQLite.jl a shot. That is an alternative Query.jl backend that translates your Query.jl query into SQL. You would write something like this:
import QuerySQLite
db = QuerySQLite.Database(db)
db.fxm |> @select(-:c)
And in theory that should then run a SQL statement that never even loads the c
column from the database at all.
I'm not sure, though, whether this works with the -:c
construction, @bramtayl would know.
I haven't done any performance testing on QuerySQL at all, but it's definitely worth a shot! If you don't see performance improvements (highly possible), open an issue and I can try to see what I can do (it might just be a matter of optimizing the generated SQL code a bit).
I have a very large table in a SQL database that I'm working with and it would be great to ignore some columns before loading, but currently I can't drop columns using
@select
:It works fine if I first convert to a DataFrame