nanne-aben
commented
1 year ago Ah, yes, that would be great!
Sorry for my late response btw, I had misconfigured my email notification settings for this repo.
Right now, we only have one open task. Consider the following code:
from typedspark import transform_to_schema
from typedspark import register_schema_to_dataset
from typedspark import Column, Schema, create_partially_filled_dataset
from pyspark.sql.types import IntegerType, StringType
from pyspark.sql import SparkSession
spark = SparkSession.Builder().getOrCreate()
class Person(Schema):
id: Column[IntegerType]
name: Column[StringType]
age: Column[IntegerType]
class Job(Schema):
id: Column[IntegerType]
salary: Column[IntegerType]
df_a = create_partially_filled_dataset(spark, Person, {Person.id: [1, 2, 3]})
df_b = create_partially_filled_dataset(spark, Job, {Job.id: [1, 2, 3]})
person = register_schema_to_dataset(df_a, Person)
job = register_schema_to_dataset(df_b, Job)
class PersonWithJob(Person, Job):
pass
(
transform_to_schema(
df_a
.join(df_b, person.id == job.id),
PersonWithJob
)
.show()
)This currently yields an AnalysisException. The problem here is that transform_to_schema() tries to subset df_a.join(df_b, person.id == job.id) to all columns defined in PersonWithJob, but it then finds that the id column is not uniquely defined. And it's indeed unclear whether it should be PersonWithJob.id = person.id or PersonWithJob.id = job.id.
The current solution is to drop one of the columns.
(
transform_to_schema(
df_a
.join(df_b, person.id == job.id)
.drop(person.id),
PersonWithJob
)
.show()
)That works, but it has some downsides:
- It's unintuitive, people keep running into this problem
- It assumes that the column names stay the same (and if they don't, someone has to debug this awful error message)
I think the solution is two-fold.
First: can we detect ourselves whether any of the columns from the provided schema are ambiguous? And if so, raise a more informative error message? (Rather than the generic AnalysisException?
Second: can we define a more intuitive solution, for example:
(
transform_to_schema(
df_a
.join(df_b, person.id == job.id),
PersonWithJob,
{
PersonWithJob.id: job.id,
}
)
.show()
)It would be great if we could use something like this to signal that we want PersonWithJob.id = job.id rather than PersonWithJob.id = person.id.
Would you be interested in working on (one of) these? Lemme know if you have any questions!
mohitCodepy
I would like to contribute on this organisation, If you have any bugs or want the implementation of the feature then I'm available.
Thank you