Need a get_content_list method for connections

[ninjapapa]

Client code:

my_conn = smvApp.get_connection_by_name('my_hdfs')
csv_files = my_conn.get_content_list(pattern=r".*\.csv", ignore_case=True)

[ninjapapa]

Since get_content_list have to use spark context, if might be better to make it a SmvApp method.

[AliTajeldin]

• I think the method should probably stay on the conn instance. From user experience, they are acting on a connection.
• Because other connections will not have a concept of pattern and case, we should probably define the base get_content_list as a function with keyword args **kw so that other connection types can define other filter params.

[ninjapapa]

• Agree
• Why hive connection does not need pattern? Could also have, right?

[ninjapapa]

Or maybe pattern should be an attribute of that connection....

[AliTajeldin]

probably should have said "might not have" instead of "will not have". Hive will have pattern but other future connection types may not have that concept.

[AliTajeldin]

pattern should probably stay with the get_content_list. cleaner than splitting the content filter criterion between two places (in constructor and in getting content list). Also, we may want to use pattern to filter out large connection lists as user type in (rather than our usual case of getting all items and filtering the in-memory list)

[ninjapapa]

Will get everything without the pattern match.

[ninjapapa]

Fo JDBC, the ways to get a list of tables are different for different DBs.

• MySql
• mssql
• derby
• postgresql
• Oracle