1- In Activity 2 all columns are upper case so all the joins needs to be adjusted to be upper case. because different from spark sql which is not case sensitive, Glue join is case sensitive.
2- Using drop_fields after a join as it is will drop the original column not the one renamed with period(.) in front of it, in order to drop the renamed column, you need to write it as follows: .drop_fields(['.CUST_ID'])
1- In Activity 2 all columns are upper case so all the joins needs to be adjusted to be upper case. because different from spark sql which is not case sensitive, Glue join is case sensitive.
2- Using drop_fields after a join as it is will drop the original column not the one renamed with period(.) in front of it, in order to drop the renamed column, you need to write it as follows: .drop_fields(['
.CUST_ID'])