introduce new source detail option eventhub.accessKeySecretName

databrickslabs / dlt-meta

This is metadata driven DLT based framework for bronze/silver pipelines

Other

125 stars 54 forks source link

introduce new source detail option eventhub.accessKeySecretName #14

Closed msdotnetclr closed 8 months ago

msdotnetclr commented 9 months ago

Introduce a new source_details option "eventhub.accessKeySecretName" to store the name of the secret holding the actual Event Hub access key (Shared Access Policy) value, keep "eventhub.accessKeyName" to store the name of the access key (Shared Access Policy).

This allows the code to use "eventhub.accessKeySecretName" to get the actual access key value, and use "eventhub.accessKeyName" to get the access key name, thereby construct eh_shared_key_value properly, regardless if "eventhub.accessKeySecretName" and "eventhub.accessKeyName" are the same.

msdotnetclr commented 9 months ago

Link this to issue #13

codecov[bot] commented 9 months ago

Codecov Report

Merging #14 (ecbbe81) into main (28ddde6) will increase coverage by 0.06%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #14      +/-   ##
==========================================
+ Coverage   84.93%   85.00%   +0.06%     
==========================================
  Files           7        7              
  Lines         737      740       +3     
  Branches      127      128       +1     
==========================================
+ Hits          626      629       +3     
  Misses         61       61              
  Partials       50       50

Flag	Coverage Δ
unittests	`85.00% <100.00%> (+0.06%)`	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
src/pipeline_readers.py	`92.20% <100.00%> (+0.31%)`	:arrow_up:

msdotnetclr commented 9 months ago

Pushed a few more commits to address the following:

Backward compatibility (so existing event hubs pipelines that are already working won't break when the changes are deployed)
Unit test and integration test
Documentation

Please review.

rportilla-databricks commented 9 months ago

3 updates that we will need to make according to the repo owner:

1 Remove oryx-build-commands.txt file 2) Update the changelog (suggested update)

[v.0.0.4] - 2023-10-09

Added

Functionality to introduce an new option for event hub configuration. Namely a source_details option 'eventhub.accessKeySecretName' to properly construct the eh_shared_key_value properly. Without this option, there were errors while connecting to the event hub service (linked to issue-13 - https://github.com/databrickslabs/dlt-meta/issues/13)

3) Increment version to 0.0.4 inside setup.py

msdotnetclr commented 9 months ago

Thanks @rportilla-databricks. The suggested changes have been committed.