search

IBM / data-prep-kit

Open source project for data preparation of LLM application builders
https://ibm.github.io/data-prep-kit/
Apache License 2.0
236 stars 122 forks source link

[Bug] text_encoder failing kfp_ray test #678

Open daw3rd opened 2 weeks ago

daw3rd commented 2 weeks ago

Search before asking

Component

Transforms/Other

What happened + What you expected to happen

With the recent PR to properly enabled kfp workflow tests (PR https://github.com/IBM/data-prep-kit/pull/672) on transforms, we found that repo_level_ordering was not passing (see log here. To get the PR to pass, text_encoder/kfp_ray/Makefile was renamed to Makefile.disable-cicd.

Reproduction script

To reproduce

cd transforms/language/text_encoder/kfp_ray git mv Makefile.disable-cicd Makefile git commit -s -a -m "restore kfp_ray test for text_encoder" git push

Make a PR and watch the cicd

Anything else

 make[5]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
rm -rf data-processing-lib-ray
rm -rf data-processing-lib-python
rm -rf shared_workflow_support_lib
rm -rf workflow_support_lib
make[4]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
make[3]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
make[3]: Entering directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
kind load docker-image quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest --name=dataprep
Image: "quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" with ID "sha256:000fdabe3bf3f95eabb3f5aef098fed7e57e047bda1046df696ecf299eaeb7d0" not yet present on node "dataprep-control-plane", loading...
Image: "quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" with ID "sha256:000fdabe3bf3f95eabb3f5aef098fed7e57e047bda1046df696ecf299eaeb7d0" not yet present on node "dataprep-worker", loading...
Image: "quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" with ID "sha256:000fdabe3bf3f95eabb3f5aef098fed7e57e047bda1046df696ecf299eaeb7d0" not yet present on node "dataprep-worker2", loading...
ERROR: command "docker save -o /tmp/images-tar359392491/images.tar quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" failed with error: exit status 1

Command Output: write /tmp/images-tar359392491/.docker_temp_4286091745: no space left on device
make[3]: *** [Makefile:57: kind-load-image] Error 1
make[3]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
make[2]: *** [/home/runner/work/data-prep-kit/data-prep-kit/transforms/language/text_encoder/kfp_ray/../../../..//transforms/.make.workflows:45: .workflows.test-pipeline] Error 2
make[2]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/language/text_encoder/kfp_ray'
make[1]: *** [Makefile:45: workflow-test] Error 2
make[1]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/language/text_encoder/kfp_ray'
make: *** [Makefile:64: workflow-test] Error 2
make: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/language/text_encoder'
Error: Process completed with exit code 2.

OS

Ubuntu

Python

3.10.x

Are you willing to submit a PR?

revit13 commented 2 weeks ago

It is added to the kfp black list in https://github.com/IBM/data-prep-kit/pull/684. Additionally, the kfp_ray/Makefile has been restored to enable local testing and to verify the workflow build in the CI/CD pipeline.