ploomber / soopervisor

☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.
https://soopervisor.readthedocs.io
Apache License 2.0
45 stars 18 forks source link

Warn if files are too big #81

Closed Wxl19980214 closed 2 years ago

Wxl19980214 commented 2 years ago

Describe your changes

Added a check for big files and issue warnings

Issue ticket number and link

Closes #58

Checklist before requesting a review

Wxl19980214 commented 2 years ago

Passed local test. But I haven't try it out for real. There's something wrong the tutorial and I cannot run it.

edublancas commented 2 years ago

hey. yeah, people are having issues with the soopervisor tutorial and we haven't fixed them.

but you can test without following the full tutorial, check this out https://soopervisor.readthedocs.io/en/latest/cookbook/kubernetes.html

Wxl19980214 commented 2 years ago

Tested, ready for review

idomic commented 2 years ago

@Wxl19980214 one of the tests is failing: test_copy_warn_if_file_too_big

Wxl19980214 commented 2 years ago

checking

Wxl19980214 commented 2 years ago

It's windows thing. The directory format is dir\file for windows but dir/file for linux and macOs. I will just remove it

Wxl19980214 commented 2 years ago

Ready to run workflow

edublancas commented 2 years ago

approved running workflow

idomic commented 2 years ago

LGTM, nice work!