Closed shahrokhDaijavad closed 4 months ago
This is about testing the scalability of kfpV1 on a real and large cluster like fmc-preprocessing.
Learned from Yuan-Chi and Hamid about the on-going fmc-preprocessing cluster runs our team is conducting. Then, I submitted an ededup run on that cluster with a datasize of about 600 files that ran successfully in about one hour.
Search before asking
Component
Library/kfp
Feature
This is testing KFPv1 on a real cluster like fmc-preprocessing that can handle a large number of datasets
Are you willing to submit a PR?