search

nebari-dev / nebari-workflow-controller

A WIP kubernetes admission controller to control what workflows can be deployed by which users
BSD 3-Clause "New" or "Revised" License
0 stars 2 forks source link

Allow workflows/cronworkflows to be resubmitted #18

Open iameskild opened 1 year ago

iameskild commented 1 year ago

Currently there is a known limitation which prevents us from resubmitting workflows/cronworkflows. As such, this is preventing the Argo-Jupyter-Scheduler from being able to Resume paused cron-workflows and prevents cron-workflows from being updated (ie to run on a new schedule).

dharhas commented 1 year ago

@iameskild When you have a moment, can you add details or point to an upstream issue that defines why this is? i.e. is this a limitation of Argo or an implementation issue etc.

iameskild commented 1 year ago

@dharhas as far as a I can tell, this is a limitation of the Nebari-Workflow-Controller ( https://github.com/nebari-dev/nebari-workflow-controller#known-limitations ). Resubmitting workflows fail in general when using NWC.

dharhas commented 1 year ago

Yes. But there are no details on why.

iameskild commented 1 year ago

I'm not sure why this is a limitation on the NWC side. @Adam-D-Lewis if you have more information, do you mind chiming in?

Adam-D-Lewis commented 1 year ago

Originally, I didn't support resubmission because I wasn't sure how to get the username from the creator-preferred-username label, but I think Eskild added a way to get the username from the creator-preferred-username, but I need to check that his solution applies here.

Adam-D-Lewis commented 1 year ago

So I checked and unfortunately, the creator-preferred-username set by argo just strips off unsupported characters like @ and ! in the username so getting the creator on a resubmitted workflow is not possible.

It looks like there is a fix that was just committed 2 weeks ago and might be in the latest version of Argo Workflows if we want to upgrade (It should be in the v3.5 release of argo worfklows). With the fix we could just look at the creator label instead of the creator-preferred-username. https://github.com/argoproj/argo-workflows/issues/11414

ericdatakelly commented 4 months ago

I'm working on the doc for jupyter-scheduler and followed the warning to this issue. I see that argo-workflows is up to v3.5.6. Will this be fixed soon or should I leave the warning unmodified? https://www.nebari.dev/docs/tutorials/jupyter-scheduler#submit-jupyter-notebooks-to-run-on-a-schedule

Adam-D-Lewis commented 4 months ago

@ericdatakelly Updating Argo workflows is desired, but not scheduled at the moment. We would accept a PR doing so, but we would need to update Jupyter scheduler and argo-workflow-controller as well before merging.