Open chris-bishop-tfs opened 2 months ago
mode='overwrite' drops and replaces data/tables, which can lead to schema changes without warning.
mode='overwrite'
Worth considering issuing overwrite as a truncate=True by default to force the check
truncate=True
https://stackoverflow.com/questions/49650790/how-to-overwrite-data-with-pysparks-jdbc-without-losing-schema
Add option here https://github.com/chris-bishop-tfs/data_io/blob/main/data_io/connection.py#L484
chris-bishop-tfs
commented
2 months ago chris-bishop-tfs
commented
2 months ago
mode='overwrite'drops and replaces data/tables, which can lead to schema changes without warning.Worth considering issuing overwrite as a
truncate=Trueby default to force the checkhttps://stackoverflow.com/questions/49650790/how-to-overwrite-data-with-pysparks-jdbc-without-losing-schema