search

eakmanrq / sqlframe

Turning PySpark Into a Universal DataFrame API
https://sqlframe.readthedocs.io/en/stable/
MIT License
191 stars 3 forks source link

"lit" command on pyspark doesn't seem to be supported #19

Closed souravagasti closed 2 months ago

souravagasti commented 2 months ago

Getting an error with the following code:

image

eakmanrq commented 2 months ago

Lit does work you just need to make sure to import from SQLFrame path instead of PySpark.

from sqlframe.standalone import StandaloneSession
from sqlframe.standalone.functions import lit

.....

When using SQLFrame you no longer import from pyspark but instead import from sqlframe. I will update docs this evening to make this clearer.

eakmanrq commented 2 months ago

PR: https://github.com/eakmanrq/sqlframe/pull/20

1harshit1 commented 2 months ago

I see a strange behaviour,

image

The code works fine with line 21, but does not work with line 20. (It is the same step in different order).

image
eakmanrq commented 2 months ago

I think if you replace df.Name + df.Age to col("name") + col("age") then it will work. Will look into fix tonight.

eakmanrq commented 2 months ago

@1harshit1 This should be fixed in 1.1.3. If you have further issues please open a new issue and I would be happy to take a look. Thanks for reporting this!