search

opendatahub-io / ilab-on-ocp

12 stars 22 forks source link

Optimize data transfers within pipeline #47

Open sallyom opened 1 month ago

sallyom commented 1 month ago

S3 should only be utilized to store results - functional data back & forth should be done via PVCs as much as possible.

leseb commented 1 month ago

If this is just to pick the best score, we could also read the job logs and read the scores from here, even less back and forth.

tumido commented 3 days ago

@sallyom is this solved by us sharing data via PVCs now? Can we close this as solved via https://github.com/opendatahub-io/ilab-on-ocp/pull/155