The current metadata exporter only support to be run under the production folder. This PR updates the directory traverse logic to make it runnable under teams folder or on single file.
Before the change:
python3 ~/.local/bin/run.py --mode=metadata-export --input-root-path production
After the change:
python3 ~/.local/bin/run.py --mode=metadata-export --input-root-path production
python3 ~/.local/bin/run.py --mode=metadata-export --input-root-path production/joins/ml_infra
python3 ~/.local/bin/run.py --mode=metadata-export --input-root-path production/joins
python3 ~/.local/bin/run.py --mode=metadata-export --input-root-path production/joins/ml_infra/test_config.v1
Will all be supported.
Why / Goal
Test Plan
[ ] Added Unit Tests
[ ] Covered by existing CI
[x] Integration tested
Tested by local run metadata exporter
Summary
The current metadata exporter only support to be run under the production folder. This PR updates the directory traverse logic to make it runnable under teams folder or on single file.
Before the change: python3 ~/.local/bin/run.py --mode=metadata-export --input-root-path production
After the change: python3 ~/.local/bin/run.py --mode=metadata-export --input-root-path production python3 ~/.local/bin/run.py --mode=metadata-export --input-root-path production/joins/ml_infra python3 ~/.local/bin/run.py --mode=metadata-export --input-root-path production/joins python3 ~/.local/bin/run.py --mode=metadata-export --input-root-path production/joins/ml_infra/test_config.v1
Will all be supported.
Why / Goal
Test Plan
Checklist
Reviewers
@hzding621 @SophieYu41