Closed soumyendra98 closed 6 months ago
Fix for ais etl init - missing --comm-type argument
Fixes in - https://github.com/NVIDIA/aistore/blob/main/docs/etl.md Inline ETL example Step 5
ais etl init spec --from-file md5_spec.yaml --name etl-md5 → ais etl init spec --from-file md5_spec.yaml --name etl-md5 --comm-type hpull ETL[etl-md5]: job "etl-tKiJJphmR"
Offline ETL example
cat code.py → cat > code.py
cat deps.txt → cat > deps.txt
ais etl init code --name etl-torchvision --from-file code.py --deps-file deps.txt --runtime python3.11v2 → ais etl init code --name etl-torchvision --from-file code.py --deps-file deps.txt --runtime python3.11v2 --comm-type hpull ETL[etl-torchvision]: job "etl-SnLbvA7gz"
ETL name specifications
ais etl init code --name=etl-md5 --from-file=code.py --runtime=python3 --deps-file=deps.txt → ais etl init code --name=etl-md5 --from-file=code.py --runtime=python3 --deps-file=deps.txt --comm-type hpull
ais etl init spec --name=etl-md5 --from-file=spec.yaml → ais etl init spec --name=etl-md5 --from-file=spec.yaml --comm-type hpull
Fixes in - https://github.com/NVIDIA/aistore/blob/main/docs/cli/etl.md Init ETL with Code Example
ais etl init code --from-file=code.py --runtime=python3.11v2 --name=transformer-md5 → ais etl init code --from-file=code.py --runtime=python3.11v2 --name=transformer-md5 --comm-type hpull
ais etl init code --name=etl-md5 --from-file=code.py --runtime=python3.11v2 --chunk-size=32768 --before=before --after=after → ais etl init code --name=etl-md5 --from-file=code.py --runtime=python3.11v2 --chunk-size=32768 --before=before --after=after --comm-type hpull
Fixes in - https://github.com/NVIDIA/aistore/blob/main/docs/tutorials/etl/compute_md5.md Simplified flow
Fixes in - https://github.com/NVIDIA/aistore/blob/main/docs/tutorials/etl/etl_imagenet_pytorch.md Transform Dataset
ais etl init code --from-file=code.py --deps-file=deps.txt --runtime=python3.11v2 --name="pytorch-transformer" → ais etl init code --from-file=code.py --deps-file=deps.txt --runtime=python3.11v2 --name="pytorch-transformer" --comm-type hpull
merged this into main
Fix for ais etl init - missing --comm-type argument
Fixes in - https://github.com/NVIDIA/aistore/blob/main/docs/etl.md Inline ETL example Step 5
ais etl init spec --from-file md5_spec.yaml --name etl-md5 → ais etl init spec --from-file md5_spec.yaml --name etl-md5 --comm-type hpull ETL[etl-md5]: job "etl-tKiJJphmR"
Offline ETL example
cat code.py → cat > code.py
cat deps.txt → cat > deps.txt
ais etl init code --name etl-torchvision --from-file code.py --deps-file deps.txt --runtime python3.11v2 → ais etl init code --name etl-torchvision --from-file code.py --deps-file deps.txt --runtime python3.11v2 --comm-type hpull ETL[etl-torchvision]: job "etl-SnLbvA7gz"
ETL name specifications
ais etl init code --name=etl-md5 --from-file=code.py --runtime=python3 --deps-file=deps.txt → ais etl init code --name=etl-md5 --from-file=code.py --runtime=python3 --deps-file=deps.txt --comm-type hpull
ais etl init spec --name=etl-md5 --from-file=spec.yaml → ais etl init spec --name=etl-md5 --from-file=spec.yaml --comm-type hpull
Fixes in - https://github.com/NVIDIA/aistore/blob/main/docs/cli/etl.md Init ETL with Code Example
ais etl init code --from-file=code.py --runtime=python3.11v2 --name=transformer-md5 → ais etl init code --from-file=code.py --runtime=python3.11v2 --name=transformer-md5 --comm-type hpull
ais etl init code --name=etl-md5 --from-file=code.py --runtime=python3.11v2 --chunk-size=32768 --before=before --after=after → ais etl init code --name=etl-md5 --from-file=code.py --runtime=python3.11v2 --chunk-size=32768 --before=before --after=after --comm-type hpull
Fixes in - https://github.com/NVIDIA/aistore/blob/main/docs/tutorials/etl/compute_md5.md Simplified flow
ais etl init code --from-file=code.py --runtime=python3.11v2 --name=transformer-md5 → ais etl init code --from-file=code.py --runtime=python3.11v2 --name=transformer-md5 --comm-type hpull
Fixes in - https://github.com/NVIDIA/aistore/blob/main/docs/tutorials/etl/etl_imagenet_pytorch.md Transform Dataset
ais etl init code --from-file=code.py --deps-file=deps.txt --runtime=python3.11v2 --name="pytorch-transformer" → ais etl init code --from-file=code.py --deps-file=deps.txt --runtime=python3.11v2 --name="pytorch-transformer" --comm-type hpull