ibnesayeed commented 6 years ago

Description

Both DockerHub as well as the open source implementation of the Docker Registry support setting up webhooks for CI/CD pipeline. We can leverage this feature to automate the deployment of updated services. Currently, if we utilize such webhooks to automate stack/service deployments using scripts, Portainer will lose control on those stacks.

Here is one workflow I can imagine (in stacks with the swarm mode enabled, but can be utilized in plain services too):

When a stack is created, it registers one or more services based on the stackfile
Services corresponding to images that are not explicitly tagged with SHA digest in the stackfile should give an option to enable webhook based updates
Each service should generate a pingback URL (something like https://portainer.example.com/pingback/<RANDOM_TOKEN>) when webhook based updates are enabled for the service
The RANDOM_TOKEN can be generated on demand or can optionally be read from a well-defined service label present in the stackfile (such as portainer.pingback.token)
These pingback URLs can be called without authentication, their security depends on the randomness and length of the token
These pingback URLs can then be added in the corresponding images in corresponding registries (such as DockerHub) so that every time an associated image is built/pushed in the registry, Portainer gets notified
When Portainer receives a notification:
- Extract the token and find corresponding entity (such as the service/stack)
- Pull the image with tag of the service/stack (not the implicitly resolved SHA based tag)
- Update image reference of the service with new SHA, if necessary, in the stack record
- Update instances of the service using defined update policy
- While the registry might notify on every build, an update is only triggered if the corresponding tag is updated
If there are more types of tokens in future, we might want to make it a first class entity and place it in the sidebar for token management where we can list the token itself, its type, its association, how often it gets pinged, and when it was last ping, with the option to delete existing tokens

deviantony commented 6 years ago

Sorry for the late answer. I believe that the way you describe the feature is a bit too technical here, actually I did not even understand the use case behind it. Could you give me more functional details about what you'd like to achieve?

ibnesayeed commented 6 years ago

The goal of this request is to be able to automatically update running services managed by Portainer when their corresponding image is updated (built or pushed) in the registry (such as DockerHub).

My first post describes one potential approach to implement it in a secure way. It utilizes webhooks the same way automated builds in DockerHub are notified by associated GitHub/Bitbucket to trigger a new build. DockerHub also has the option to setup webhooks that will ping any URLs specified in it when an image is updated. We can leverage that to implement this feature.

Please let me know if you need more explanation on specific parts.

ncresswell commented 6 years ago

Oh so an integrated “watchtower” experience...

Rgds,

Neil Cresswell

On 26/02/2018, at 3:25 PM, Sawood Alam notifications@github.com<mailto:notifications@github.com> wrote:

The goal of this request is to be able to automatically update running services managed by Portainer when their corresponding image is updated (built or pushed) in the registry (such as DockerHub).

My first post describes one potential approach to implement it in secure way. It utilizes webhooks the same way automated builds in DockerHub are notified by associated GitHub/Bitbucket to trigger a new build. DockerHub also has the option to setup webhooks that will ping any URLs specified in it when an image is updated. We can leverage that to implement this feature.

Please let me know if you need more explanation on specific parts.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_portainer_portainer_issues_1663-23issuecomment-2D368372154&d=DwMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=1nvAiOGea4FI3fcdhFf-8_qK4mfCCBx6tCL6UNPoFQA&s=Ys81d7Tm_31bM7JtEmThReyWJautlF-iTVe1x233E8Y&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AWGrlSK983MIrpcawUH3aKts9Myil3INks5tYhXwgaJpZM4SMts1&d=DwMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=1nvAiOGea4FI3fcdhFf-8_qK4mfCCBx6tCL6UNPoFQA&s=9VEw6zb4ZqgbyM9_OyBEe-lrh0LA1pcWim-2El90RAE&e=.

deviantony commented 6 years ago

ibnesayeed commented 6 years ago

Yes, it is related to #1304, but outlines an implementation mechanism that utilizes the webhook (or in other words notification or pingback) feature of registries or other CI/CD systems rather than using periodic polling that can be wasteful or delayed.

dehy commented 6 years ago

It's like Docker Cloud auto redeploy functionnality, and I can only 👍 . As Docker Cloud is closing, many people will seek for an alternative... and they are now. See last paragraph here https://docs.docker.com/docker-cloud/migration/#what-stays-the-same

zhangruhong commented 6 years ago

I need this feature!

Cinezaster commented 6 years ago

We also like to see this feature.

Our current setup with docker cloud:

We push our project on github
Docker hub automatically builds a new image
Docker cloud redeployed our images if this was configured that way

We used this setup for our development. We have a dev branch that automatically builds and redeploys on git push When we merge to a staging this also automatically builds and redeploys When we merge the staging to production (master) it automatically gets build but we

With the current portainer we need to pull the latest image on the image page and after that we need to go to the service and update the service.

ibnesayeed commented 6 years ago

I came back to check for any progress on this request. I was surprised to see the number of thumbs ups to the first post, but no one seems to be interested in working on this. While I have an architectural understanding of how this feature can possibly be implemented (as outlined in the first post), I am not familiar with Portainer code base and for the next few months some other priorities are keeping my hands tied. Though, I would really like if someone could take care of it soon. I will be more than happy to discuss implementation details if necessary.

ncresswell commented 6 years ago

I have actually assigned this to one of our developers to create a functional spec and give me an estimate for development effort.

Watch this space..

From: Sawood Alam notifications@github.com Sent: Wednesday, 27 June 2018 12:39 p.m. To: portainer/portainer portainer@noreply.github.com Cc: Neil Cresswell neil@cresswell.net.nz; Comment comment@noreply.github.com Subject: Re: [portainer/portainer] [Feature Request] Automate deployment using registry webhooks (#1663)

I came back to check for any progress on this request. I was surprised to see the number of thumbs ups to the first post, but no one seems to be interested in working on this. While I have an architectural understanding of how this feature can possibly be implemented (as outlined in the first post), I am not familiar with Portainer code base and for the next few months some other priorities are keeping my hands tied. Though, I would really like if someone could take care of it soon. I will be more than happy to discuss implementation details if necessary.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_portainer_portainer_issues_1663-23issuecomment-2D400506423&d=DwMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=s5UOJA0LMQhYo4cJoPJMCabVMZ7GXSrYR8W6DLyAWvk&s=uKaL_Difg8k_Uy1LtTzNyAXVR3tVsxJ_oYWf7EN4kHA&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AWGrlXbQhPQW3bKQTmCXKNq6unPoUY5bks5uAtQGgaJpZM4SMts1&d=DwMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=s5UOJA0LMQhYo4cJoPJMCabVMZ7GXSrYR8W6DLyAWvk&s=O6uss9ua_UOTYRVDl6pNSsJgeeCx_4f3_4cWV4YCJo0&e=.

deviantony commented 6 years ago

FYI here is the first sketch of the specs:

Service redeploy feature

Portainer can expose a new webhook API that can be used to trigger an update of specific services.

Notes:

this spec only describe how to associate a webhook to a Swarm service (for Swarm stacks, each service of the stack could be updated independently via the service-details view). We could potentially add the same behavior to containers but we should start with services first.
it can be interesting to play with https://github.com/imjosh/docker-deploy-webhook to get some extra insights.

Portainer UI (frontend)

In the service-creation/service-details views, a new switch "Create re-deploy webhook" is added.

When enabled, upon successful service creation a HTTP POST request is sent to /api/webhooks (authenticated user policy) with the following payload:

serviceId: docker service identifier of the service,
endpointId: endpoint identifier where the service is located

Portainer API (backend)

The previous request will create a new webhook object:

id: webhook identifier,
serviceId: docker service identifier,
endpointId: endpoint identifier where the service is located

The webhook can then be used by sending a HTTP POST request to api/webhook/:id (public access policy in order to be reached from external services such as DockerHub)

It requires no payload.

Note: as the /api/webhook/:id endpoint is publicly exposed, the webhook identifier format should be in the form of a randomly generated string. We might want to add an extra query parameter that can be matched for access restriction.

When receiving a POST request at /api/webhook/:id, the API will send a request to the specified endpoint to retrieve details about the service associated to the webhook (using the service identifier). It will extract information about the image used in the service details, retrieve any registry authentication details if required (check in database) and send a service update request.

Note: In order to execute Docker requests from the backend, we will probably need to embed the Docker SDK inside the API to allow the backend to act as a Docker client.

Cinezaster commented 6 years ago

In Docker hub you have the ability to create a Webhook https://docs.docker.com/docker-hub/webhooks/ that triggers a post call whenever an image is build or a tag is added. So we could trigger an image pull and redeploy in portainer.

deviantony commented 6 years ago

@Cinezaster that's the goal, yes

Cinezaster commented 6 years ago

@deviantony we never use the service-creation view to create our service but instead us a stackfile to create our service, would it be possible to add the webhook later via the service view?

deviantony commented 6 years ago

Yes, you'll be able to go the service-details view to create a re-deploy webhook as well, I'll update the specs.

Cinezaster commented 6 years ago

Any news on this request?

deviantony commented 6 years ago

Yup, it's still in our priority list, I'll assign someone on this soon.

olljanat commented 6 years ago

FYI. Not fully same thing but can be useful for some users. I just published my Portainer extension for VSTS: https://marketplace.visualstudio.com/items?itemName=OlliJanatuinen.portainer-deploy

deviantony commented 6 years ago

Anybody keen to try the implementation for this? It's available via portainer/portainer:feat1663-add-webhook (Linux amd64 build only).

Any feedback welcome !

Cinezaster commented 6 years ago

@deviantony we will try it and test it @appsaloon

Strum355 commented 6 years ago

Is this only for swarm services only or for independent containers as well?

deviantony commented 6 years ago

@Strum355 only for Swarm services right now.

Strum355 commented 6 years ago

Ah no worries, is there a rough eta on when we could see this for containers?

deviantony commented 6 years ago

No ETA yet

ibnesayeed commented 6 years ago

Thanks for pushing this one forward. I just deployed it and now exploring around. I will provide more feedback in the PR #2161.

Should we also plan to generalize this webhook functionality for other entities (as I briefly described in an earlier comment) and make it a top-class entity to show up in the sidebar menu? An immediate extension that I can think of is webhooks for images that would allow us to connect directly from a source repo (such as one on GitHub or BitBucket) and build an image directly (which is made possible by #1667) when the code changes. Optionally, we can also update services/containers when an associated image is updated locally.

ibnesayeed commented 6 years ago

If we do decide to make webhooks for more than one entities, we might want to namespace URLs as https://portainer.example.com/api/webhooks/services/12345... and https://portainer.example.com/api/webhooks/images/12345... instead of https://portainer.example.com/api/webhooks/12345....

Though it is not necessary as the hook IDs can be unique across the system and their associated type can be stored in a table in the form of polymorphic association.

JoJordens commented 6 years ago

@Cinezaster and I tried it out with the webhooks hub.docker provides. And it works, our services are automatically being updated after each push to a repository. But the tags of the images that were updated are all <none>, both in the image list in portainer as well as when I look at docker images. This doesn't interfere with functionality however. We have multiple services using the same image with different tags, and they're all still using the correct image after the update.

ncresswell commented 6 years ago

Once development is done, we need someone to document /blog how to link portainer with dockerhub webhooks as you have done, feel like being a volunteer for that?

Rgds,

Neil Cresswell

On 27/08/2018, at 4:37 PM, Jo Jordens notifications@github.com<mailto:notifications@github.com> wrote:

@Cinezasterhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Cinezaster&d=DwMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=Ppw5k5PG1o8Dc0s32b3QQRfKWrl25HEvMc5Jx0-8vrY&s=TqOWOtwxUrb9DhaLtaZGfZ_4zCYlwaTMfGzIvtX0QtE&e= and I tried it out with the webhooks hub.docker provides. And it works, our services are automatically being updated after each push to a repository. But the tags of the images that were updated are all , both in the image list in portainer as well as when I look at docker images. This doesn't interfere with functionality however. We have multiple services using the same image with different tags, and they're all still using the correct image after the update.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_portainer_portainer_issues_1663-23issuecomment-2D416172189&d=DwMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=Ppw5k5PG1o8Dc0s32b3QQRfKWrl25HEvMc5Jx0-8vrY&s=0UKZwz1EI0L8JYdLqrswGx6dpZ0DT9QlxD_aXH-N6vE&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AWGrlVGGU2TyMZz2ErAk-5FseDNYgftzE5ks5uU73VgaJpZM4SMts1&d=DwMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=Ppw5k5PG1o8Dc0s32b3QQRfKWrl25HEvMc5Jx0-8vrY&s=ofElX8u3XOItT169JEYAQ0UO8f6moEEEoBWaOqBJCMU&e=.

deviantony commented 6 years ago

Any feedback from the users that were able to test this yet?

Cinezaster commented 6 years ago

@deviantony, @JoJordens tested the feature, his only comment : tags of the images are <none> after an update. @ncresswell We're going to write a blog post this or next week about how to use this feature.

ibnesayeed commented 6 years ago

I have also been testing this and provided feedback on both the functionality and code.

Should we namespace webhook URIs (by including services in the URI to extend the functionality to other aspects such as images in future)?
I think the <none> tag issue is present in the stack functionality. When a stack is deployed/updated without first updating the image explicitly, the image that is pulled does not get tagged properly. I have been seeing this issue for a long time.
Since we are not utilizing any payload from the webhook pingback, should we also make it work on a GET request and not just on a POST request? Ideally, webhooks should be a PUT request, but existing systems send it via POST, so having it work on POST is important. However, supporting GET will be more convenient to trigger an update from the command line.

deviantony commented 6 years ago

Thanks @Cinezaster

Also thanks @ibnesayeed but I was expecting UI/UX feedback :-)

Regarding your technical points:

Should we namespace webhook URIs (by including services in the URI to extend the functionality to other aspects such as images in future)?

I don't think so, but we'll extend the Webhook object to support a new Type field.

I think the tag issue is present in the stack functionality. When a stack is deployed/updated without first updating the image explicitly, the image that is pulled does not get tagged properly. I have been seeing this issue for a long time.

Yup, this is a known issue.

Since we are not utilizing any payload from the webhook pingback, should we also make it work on a GET request and not just on a POST request? Ideally, webhooks should be a PUT request, but existing systems send it via POST, so having it work on POST is important. However, supporting GET will be more convenient to trigger an update from thee command line.

As most of the systems are sending POST requests, I believe that we should keep it this way. While a GET request would be more convenient via the CLI, it's not that much of a hassle to actually send a POST request so I don't think we'll want to support GET requests as well.

ibnesayeed commented 6 years ago

On the UI side, I initially thought we might want to add the toggle switch in the tabular listing of services, be it on the main services page or under a specific stack. However, I think adding just the switch is insufficient. So if we do so, we will have to have more UI elements to expose the webhook URI (possibly in the drawer that shows up when a service row is clicked).

ibnesayeed commented 6 years ago

I don't think so, but we'll extend the Webhook object to support a new Type field.

As long as future extensions won't break, I am OK. I just pointed it out so that we can make it future-proof.

Yup, this is a known issue.

Do we have a ticket to track this issue?

As most of the systems are sending POST requests, I believe that we should keep it this way. While a GET request would be more convenient via the CLI, it's not that much of a hassle to actually send a POST request so I don't think we'll want to support GET requests as well.

I'm cool with this. I am not a big fan of having too many ways to do the same thing, but I thought it might be useful for some.

deviantony commented 6 years ago

@ibnesayeed see https://github.com/portainer/portainer/issues/1486

ibnesayeed commented 6 years ago

@kendrickm, I pulled the latest portainer:feat1663-add-webhook image a few minutes ago after your recent commits and redeployed the stack on my server. Now, my existing webhooks are broken. All the services where I have enabled webhooks are now appearing to have disabled associated webhooks. If I try to send a POST request to a webhook that was created earlier, I get a 500 response code and if I send a request to a non-existing random one, I get a 502. I feel like we can improve status codes here, depending on the situation.

I think the reason has to do with the newly added webhook type field which is not populated for existing tokens. If so, is there a way to patch the database on my running instance as I would not like to recreate all those tokens and update records in DockerHub for all associated repos.

$ curl -X POST -i https://portainer.example.com/api/webhooks/<created-earlier-but-now-hidden-token>
HTTP/1.1 500 Internal Server Error
Content-Length: 102
Content-Type: application/json
Date: Thu, 30 Aug 2018 21:23:15 GMT
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block

{"err":"Unsupported webhook type","details":"Webhooks for this resource are not currently supported"}

$ curl -X POST -i https://portainer.example.com/api/webhooks/<non-existing-random-token>
HTTP/1.1 502 Bad Gateway
Date: Thu, 30 Aug 2018 21:23:21 GMT
Content-Length: 11
Content-Type: text/plain; charset=utf-8

Bad Gateway

ibnesayeed commented 6 years ago

I was able to fix my deployment by manually editing the Bolt database file using https://github.com/br0xen/boltbrowser.

kendrickm commented 6 years ago

@ibnesayeed Yeah there was a bug around error handling causing it to segfault vs return an actual error. Are you using a webserver proxy in front of it?

ibnesayeed commented 6 years ago

Are you using a webserver proxy in front of it?

Yes, I am using Traefik as a reverse proxy in front of my Portainer instance and everything else.

ibnesayeed commented 6 years ago

While I am very happy with this feature being in the pipe of getting merged, I was thinking will it be useful to have the ability to optionally customize the token? I can think of some places where one might have some naming scheme for these tokens to automate batch processing without essentially querying all services and their associated tokens. On the UI side, we can add a button next to the Copy link button that says, "Customize token" or "Vanity token". We can perhaps have both a randomly generated token that has an optional custom alias.

deviantony commented 6 years ago

@ibnesayeed we can open a new enhancement request later on for this.

ibnesayeed commented 6 years ago

@ibnesayeed we can open a new enhancement request later on for this.

Yes, I agree. I didn't mean to stall the corresponding PR, but to share my thoughts.

deviantony commented 6 years ago

@ibnesayeed all good, feel free to open a new feature request for this evolution.

Strum355 commented 6 years ago

Im assuming a new enhancement request should be made to have automated deployment for individual containers?

deviantony commented 6 years ago

Yes @Strum355

ibnesayeed commented 6 years ago

Opened #2236 to track the custom token feature.

goalbased commented 5 years ago

Does this support Windows?

olljanat commented 5 years ago

@goalbased I cannot see any reason why it would not work on Windows. Can you test and let use know?

goalbased commented 5 years ago

I can't find the Service tab any idea? Do I need Swarm or something first?

olljanat commented 5 years ago

@goalbased It only exists when Swarm mode is enabled and that is because of how Docker it selves works.

Plz join to Portainer Slack and ask these offtopic questions on there.

portainer / portainer

[Feature Request] Automate deployment using registry webhooks #1663

Service redeploy feature

Portainer UI (frontend)

Portainer API (backend)