search

SAME-Project / same-project

https://sameproject.ml/
Apache License 2.0
19 stars 8 forks source link

module imports no longer seem to persist across steps #130

Closed lukemarsden closed 2 years ago

lukemarsden commented 2 years ago

This notebook: https://github.com/SAME-Project/SAME-samples/blob/multi-stage-pipeline/03-road-signs/roadsigns.ipynb

Now fails like this:

Screenshot 2022-05-09 at 14 12 35

(latest main as of today)

Possibly a regression due to adding the exploding variables support?

lukemarsden commented 2 years ago

now fails with this:

Screenshot 2022-05-09 at 16 35 56
Bubblyworld commented 2 years ago

Ah, crap, sorry - I'll be back at my laptop in about 2 hours to fix this. I think it might just be a matter of changing the order of the memory/dill exploding variable checks in kubeflow steps.jinja template. Got a bit overconfident before leaving...

On Mon, 9 May 2022, 17:37 lukemarsden, @.***> wrote:

now fails with this:

[image: Screenshot 2022-05-09 at 16 35 56] https://user-images.githubusercontent.com/264658/167445804-c8d4579c-502b-4487-95a3-aa1d98c65ea5.png

— Reply to this email directly, view it on GitHub https://github.com/SAME-Project/same-project/issues/130#issuecomment-1121262405, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPNHP2E266UQJXZOAH5XC3VJEWNHANCNFSM5VOH46DQ . You are receiving this because you were assigned.Message ID: @.***>

Bubblyworld commented 2 years ago

Looks like it's correctly replacing the CSV reader with an exploding variable, but then trying to check the mem size of the exploding variable (which subsequently explodes as designed). So we should do mem check first.

On Mon, 9 May 2022, 17:47 Guy Paterson-Jones, @.***> wrote:

Ah, crap, sorry - I'll be back at my laptop in about 2 hours to fix this. I think it might just be a matter of changing the order of the memory/dill exploding variable checks in kubeflow steps.jinja template. Got a bit overconfident before leaving...

On Mon, 9 May 2022, 17:37 lukemarsden, @.***> wrote:

now fails with this:

[image: Screenshot 2022-05-09 at 16 35 56] https://user-images.githubusercontent.com/264658/167445804-c8d4579c-502b-4487-95a3-aa1d98c65ea5.png

— Reply to this email directly, view it on GitHub https://github.com/SAME-Project/same-project/issues/130#issuecomment-1121262405, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPNHP2E266UQJXZOAH5XC3VJEWNHANCNFSM5VOH46DQ . You are receiving this because you were assigned.Message ID: @.***>

lukemarsden commented 2 years ago

All good! No rush sir! Enjoy your evening 😄