406 error from getServiceDefinition()

tomservo3428 commented 5 years ago

When I attempt to send your example declaration in 1a.AS3-EXAMPLE-Basic_L4_LB.json from my GHE repo to my BIG-IP, I get the following error:

info: [GheListener - ERROR] - getServiceDefinition(): {"code":406,"message":" "}

I don't get any other information, despite debug being enabled, and no issues were created in the repo for reference. Any thoughts @npearce ?

npearce commented 5 years ago

Can you confirm that the BIG-IP can reach Github Enterprise? Try with curl, e.g:

curl -v -H "Authorization: token {TOKEN}" https://your_ghe_dns_name/user

Am currently on my cell so unable to verify that command at this very moment. Will be back on-line soon.

tomservo3428 commented 5 years ago

The F5 can reach GHE. I created a new webhook, and while the configuration is correct - I get the little, green check mark - the delivery history shows that the setup payload couldn't be delivered. I have debug enabled, but I didn't see an entry in the restnoded log for that event so I'm not sure why it failed.

npearce commented 5 years ago

Ok, let me try to recreate. Can you provide me the following versions:

BIG-IP:
AS3:
Github Enterprise:
Github Webhook Server:

Something that might be worth a try, grab one of the example service definitions from the ‘develop’ branch.

tomservo3428 commented 5 years ago

Thanks for looking into this. Here is the information I have:

BIG-IP: 13.1.1 Build 0.47.4 Engineering Hotfix AS3: 3.7.0 Build 7 Github Enterprise: 2.13.1 Github Webhook Server: I'm not sure where to find this

I'm not sure if this is relevant, but I'm not a GHE admin, however I'm an owner in my Github org.

Also, I did see messages like this in the restnoded log:

[LoaderWorker] unsupported module file extension '/var/config/rest/iapps/f5-appsvcs/nodejs/codeCache.json', skipping... [LoaderWorker] unsupported module file extension '/var/config/rest/iapps/f5-appsvcs/nodejs/fortunes.json', skipping... LoaderWorker] unsupported module file extension '/var/config/rest/iapps/f5-appsvcs/nodejs/ltmPolicySpec.json', skipping... [LoaderWorker] unsupported module file extension '/var/config/rest/iapps/f5-appsvcs/nodejs/package-lock.json', skipping... [LoaderWorker] unsupported module file extension '/var/config/rest/iapps/f5-appsvcs/nodejs/package.json', skipping... [LoaderWorker] unsupported module file extension '/var/config/rest/iapps/f5-appsvcs/nodejs/paths.json', skipping... [LoaderWorker] unsupported module file extension '/var/config/rest/iapps/f5-appsvcs/nodejs/properties.json', skipping...

npearce commented 5 years ago

Ok, thanks for the data. The version of the Gitbub Webhook server is part of the binary file name, e.g.: https://github.com/f5devcentral/CaC-Github_Webhook_Server/tree/v0.2-release/DIST the webhook server version is v0.2.0.

Can you try one of the examples in the develop branch? You can get them there: https://github.com/f5devcentral/CaC-Github_Webhook_Server/tree/develop/EXAMPLES

tomservo3428 commented 5 years ago

Still no dice. I used service_def1.json and the following message popped up in the restnoded log:

Thu, 20 Dec 2018 20:29:10 GMT - info: [GheListener - ERROR] - getServiceDefinition(): {"code":406,"message":" "}

The webhook payload delivery history isn't much help either: "We couldn’t deliver this payload: OK"

Per the Github webhook documentation, I was able to successfully ping the webhook from my BIG-IP with the following command:

curl -X POST \ https://github.my_ghe.com/api/v3/repos/my_user_id/my_repo_name/hooks/{webhook ID}/pings \ -H 'Authorization: Basic MyAuthToken=' \ -H 'cache-control: no-cache'

The response code should be "204 No Content"

npearce commented 5 years ago

The webhook is reaching the BIG-IP and the BIG-IP is attempting to call back to Github Enterprise and fetch the newly committed service definition. The '406' code is being sent back Github Enterprise.

Unfortunately, goolging "Github Enterprise 406" returns a lot of different things... but a common theme was authentication related... so

Did you create and Auth Token for the BIG-IP? I noticed in the example above you are using Basic Auth, however the webhook server is expecting an Auth Token.

For my environment I created a BIG-IP user (just using the hostname), I added that BIG-IP user to the repo as a collaborator, and then I created an Auth Token for that user.

You can then test this with something like:

curl -v -k -X GET https://ip-172-31-1-200.us-west-1.compute.internal/api/v3/ -H 'Authorization: Token {your_auth_token}'

tomservo3428 commented 5 years ago

I used Postman to send the POST that was in the documentation so that I could ping the webhook and I used Basic Auth to do so. Everything checkout in Postman, so I copied the curl command that Postman generated to the BIG-IP CLI to see if I could ping the webhook from there, hence the Basic Auth.

I did create a token in GHE for my account and used that in the configuration settings in the webhook server. The webhook uses my BIG-IP creds, so I figured everything lined up appropriately. I wondered if it was a token issue, so I created a new token with my account and put that in the webhook server config, but still no joy.

I can start from scratch and do everything over, but I've done that a few times already with no success.

npearce commented 5 years ago

@CaptainBlasteroid - I've pushed version 0.3.0, which has some builtin auth validation. Every time you POST settings to /ghe_settings it will phone-home to github.com/github enterprise and try create a Github Issue using the auth settings.

https://github.com/f5devcentral/CaC-Github_Webhook_Server/tree/v0.3-release

My config looks like this:

{
    "config": {
        "ghe_base_url":"https://ip-172-31-1-200.us-west-1.compute.internal/api/v3",
        "repository": "NCaC/ip-172-31-1-20.us-west-1.compute.internal",
        "ghe_access_token": "16fc0a0fd2fb769e4ea873a53699190541289ac6",
        "max_queue_length": 10,
        "debug": false
    }
}

tomservo3428 commented 5 years ago

Two things after my testing this morning:

For the repository path, are you using the UserID/F5-Repo-Name?
After I installed the new package using the curl command, I attempted to set the configuration properties but I was getting 404 errors. When I sent a GET to https://my.f5.com/mgmt/shared/n8 to verify that the endpoints were available, the following error was returned: "Public URI path not registered. Please see /var/log/restjavad.0.log and /var/log/restnoded/restnoded.log for details." However, GET requests to https://my.f5.com/mgmt/shared are good.

Thoughts?

Just in case you'd like to see it, this is the full message returned from the request: { "code": 404, "message": "Public URI path not registered. Please see /var/log/restjavad.0.log and /var/log/restnoded/restnoded.log for details.", "referer": "10.10.10.10", "restOperationId": 37511810, "errorStack": [ "com.f5.rest.common.RestWorkerUriNotFoundException: Public URI path not registered. Please see /var/log/restjavad.0.log and /var/log/restnoded/restnoded.log for details.", "at com.f5.rest.workers.ForwarderPassThroughWorker.cloneAndForwardRequest(ForwarderPassThroughWorker.java:572)", "at com.f5.rest.workers.ForwarderPassThroughWorker.access$000(ForwarderPassThroughWorker.java:44)", "at com.f5.rest.workers.ForwarderPassThroughWorker$1.completed(ForwarderPassThroughWorker.java:314)", "at com.f5.rest.workers.ForwarderPassThroughWorker$1.completed(ForwarderPassThroughWorker.java:311)", "at com.f5.rest.workers.EvaluatePermissions$2.completed(EvaluatePermissions.java:191)", "at com.f5.rest.workers.EvaluatePermissions$2.completed(EvaluatePermissions.java:186)", "at com.f5.rest.workers.RolesWorker$10.completed(RolesWorker.java:959)", "at com.f5.rest.workers.RolesWorker$10.completed(RolesWorker.java:954)", "at com.f5.rest.workers.Cache$1.completed(Cache.java:151)", "at com.f5.rest.workers.TmosRoleCache$1.completed(TmosRoleCache.java:69)", "at com.f5.rest.workers.TmosRoleCache$1.completed(TmosRoleCache.java:64)", "at com.f5.rest.common.RestOperation.complete(RestOperation.java:2411)", "at com.f5.rest.common.RestWorker$3.completed(RestWorker.java:735)", "at com.f5.rest.common.RestWorker$3.completed(RestWorker.java:724)", "at com.f5.rest.common.RestOperation.complete(RestOperation.java:2411)", "at com.f5.rest.tmos.shared.adapter.TmosRoleWorker$1.completed(TmosRoleWorker.java:65)", "at com.f5.rest.tmos.shared.adapter.TmosRoleWorker$1.completed(TmosRoleWorker.java:58)", "at com.f5.rest.tmos.shared.mcp.McpOperation.complete(McpOperation.java:377)", "at com.f5.rest.tmos.shared.mcp.McpRunnableTask.run(McpRunnableTask.java:47)", "at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)", "at java.util.concurrent.FutureTask.run(FutureTask.java:262)", "at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)", "at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)", "at java.lang.Thread.run(Thread.java:745)\n" ], "kind": ":resterrorresponse"

npearce commented 5 years ago

May I suggest a Zoom meeting to speed things along? If this is something you can accommodate, please email me at n.pearce@f5.com

tomservo3428 commented 5 years ago

Email sent :)

npearce commented 5 years ago

Currently using https://username:password@bigip_ip_addressin the GitHub webhook configuration. This can cause problems with strong passwords/special chars. Created #47 to add 'webhook auth token' feature to the BIG-IP Webhook Server.

Ok for me to close @CaptainBlasteroid ?

npearce / CaC-Github_Webhook_Server

406 error from getServiceDefinition() #42