Closed mrericrichter closed 4 months ago
I checked in PySpark and this is different behaviour so I will add it but put it behind an option
Fixed in build 16 - but you will need to set this option for it to work:
spark.Conf.Set("spark.connect.dotnet.validatethiscallcolumnname", "true");
See: https://github.com/GoEddie/spark-connect-dotnet/blob/main/docs/options.md
The
public Column this[string name]
method of theDataFrame
class should throw an exception if the given columnname
is not part of the DataFrame. The exception message should include the name of the missing column.This allows errors in the user's Spark code to be found right where they occur. Otherwise, errors about missing columns could pop up at very late stages and it might be very difficult to find the error.