hive_metastore_migration.py fails with AttributeError: 'str' object has no attribute '_jdf'

[jobennin]

Testing the HMS migration script with spark-submit command fails with: AttributeError: 'str' object has no attribute '_jdf'

which is triggered by the call: id_type = df.get_schema_type(id_col)

If I change the call to: id_type = get_schema_type(df, id_col) I get past the error but expose other df related errors in other functions.

This is tested on:

"emr-5.31.0" "Hadoop":"2.10.0" "Hive":"2.3.7" "Spark":"2.4.6"

Full stack trace: Traceback (most recent call last): File "/home/hadoop/hive_metastore_migration.py", line 1525, in main() File "/home/hadoop/hive_metastore_migration.py", line 1519, in main etl_from_metastore(sc, sql_context, db_prefix, table_prefix, hive_metastore, options) etl_from_metastore(sc, sql_context, db_prefix, table_prefix, hive_metastore, options) etl_from_metastore(sc, sql_context, db_prefix, table_prefix, hive_metastore, options) File "/home/hadoop/hive_metastore_migration.py", line 1414, in etl_from_metastore etl_from_metastore(sc, sql_context, db_prefix, table_prefix, hive_metastore, options) File "/home/hadoop/hive_metastore_migration.py", line 1414, in etl_from_metastore File "/home/hadoop/hive_metastore_migration.py", line 1414, in etl_from_metastore .transform(hive_metastore) .transform(hive_metastore) .transform(hive_metastore) File "/home/hadoop/hive_metastore_migration.py", line 753, in transform ms_database_params=hive_metastore.ms_database_params) File "/home/hadoop/hive_metastore_migration.py", line 734, in transform_databases dbs_with_params = self.join_with_params(df=ms_dbs, df_params=ms_database_params, id_col='DB_ID') File "/home/hadoop/hive_metastore_migration.py", line 336, in join_with_params df_params_map = self.transform_params(params_df=df_params, id_col=id_col) File "/home/hadoop/hive_metastore_migration.py", line 314, in transform_params return self.kv_pair_to_map(params_df, id_col, key, value, 'parameters') File "/home/hadoop/hive_metastore_migration.py", line 326, in kv_pair_to_map id_type = df.get_schema_type(id_col) File "/home/hadoop/hive_metastore_migration.py", line 199, in get_schema_type return df.select(column_name).schema.fields[0].dataType File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1327, in select AttributeError: 'str' object has no attribute '_jdf'

I have also tried with EMR v6.5 with Spark v3.1.2. Same error. I thought it might be Spark version issue. What Spark version has this script been successful with? EMR version? I launch the spark-submit per the readme with the --jdbc* options changed as needed.

[Dearkano]

same issue here