search

nv-morpheus / Morpheus

Morpheus SDK
Apache License 2.0
311 stars 120 forks source link

[FEA]: Add support of writing to databricks delta table using Module. #1482

Open spatil44 opened 5 months ago

spatil44 commented 5 months ago

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

At present, Morpheus only allows writing to Databricks delta tables through a stage-based method. Unfortunately, this capability isn't available when constructing an inference pipeline using a modular approach. To address this limitation, implementing a 'Databricks Writer Module' would enable direct writing of results to the Delta Lake.

Describe your ideal solution

def _data_bricks_delta_lake_writer(builder: mrc.Builder): module_config = builder.get_current_module_config()

Additional context

I have already created "databricks_writer_module" in consultation with @drobison00 . Will raise pull request shortly.

Code of Conduct

jarmak-nv commented 5 months ago

Hi @spatil44!

Thanks for submitting this issue - our team has been notified and we'll get back to you as soon as we can! In the mean time, feel free to add any relevant information to this issue.