search

andyweaves / dlt-pii-firewall

Demo of using Delta Live Tables to identify potential PII data
Apache License 2.0
8 stars 4 forks source link

event_log table #2

Open sheridan06 opened 1 year ago

sheridan06 commented 1 year ago

Perhaps this isn't so much of an "issue" as it is me just missing something.

In the 01_observability DLT notebook, cmd2 is building the event_logs table by reading the {STORAGE_PATH}/system/events location as delta. This notebook returns an error b/c there is nothing in that path to read and create the table.

So either I've missed a step of uploading a file to that DBFS location, or I've missed a step that would generate the Delta files in that location.

Any insight is greatly appreciated

andyweaves commented 1 year ago

Hey, I can only reproduce this if the Storage location doesn't match the STORAGE_PATH param that you pass in (in the current DLT UI you do this as a key/value pair in the Advanced section of the Pipeline settings page).

If you just want to try the DLT pipeline out, you could also remove the 01_observability notebook and just run 02_detect_and_redact_pii.

I only just committed a code change to fix a bug that caused an error processing the generated data, so make sure you do a pull before you re-run!

maryr94 commented 2 months ago

I also am having this same issue :/ storage path/location are the same...