Closed steventarver closed 8 years ago
scottdensmore
commented
8 years ago The problem is the ServerService postProcessBuildServerResponse line 168. When you get the response back from CreateServer, the self link is not valid until the blueprint completes. Since you try and add the public ip at the same time (which I would recommend we not do), you try and get the server info that doesn't exist. At some point we broke the self link.
push1st1k
commented
8 years ago @steventarver As you mentioned, this issue appeared in VA1, but for example in DE1 it works well. The possible solution will be to wait until ServerMetadata serverInfo = client.findServerByUuid(response.findServerUuid()); will return a valid response with server metadata. But in this case some server operations (create, clone, import) won't be completely asynchronous, because the SDK should wait some time to retrieve the server metadata. For example, if you would like to create a server and don't wait the result, now you have to wait until the API will return a server metadata. But if you are planning to operate with this server later and call .waitUntilComplete(), it will work the same way as before.
@scottdensmore Maybe you have an idea, why it worked well before and even now in other DCs (DE1)? Something was changed in API? It would be great, if the method findServerByUuid responded with server ID immediately and the method create returns this information. Actually, the public IP is added only after the server returns status 'active', not at the same time. I'm going to fix it with adding a logic, that will send requests to the endpoint until it will return server metadata, so it won't be failing. But the delay of getting a server by UUID should be fixed at server side.
PR #203 was created.
steventarver
commented
8 years ago Would deferring the ServerMetadata fetch until it is needed be a better solution? That allows retaining a fast, async api, but looks like more refactoring.
We are using this method (create) synchronously, so either solution works for us. And your fast fix is a blessing; saves us some manual maintenance because most creates() are failing.
steventarver
commented
8 years ago @push1st1k Clarifyiing my previous comment: If the blueprint queue is full, the number of requests exceeds the number of executors, you could be waiting minutes for the server meta data to be created.
It seems, that to cover all cases, one would have to wait to fetch and use ServerMetaData until the blueprint completes successfully.
steventarver
commented
8 years ago @push1st1k When will you be cutting a release with this fix? I need to schedule a release - this fixes production issues for us.
push1st1k
commented
8 years ago This process isn't so fast, but I'll do my best to release as soon as I can.
When creating a server, a database entry for the server is not made until the blueprint is started (brannon). There is no timing guarantee between the server create completion and the blueprint start (hades). The first line of ServerService.postProcessBuildServerResponse()
throws an exception when the create server api call completes before the blueprint is started. This has worked pretty well for a long time. Last week, something changed that increased the failure frequency; it is happening for us almost every time in VA1. Hades recommends that we poll blueprint status until the blueprint is complete. I see you are doing that in the waitUntilComplete() loop, but the above call happens before entering waitUntilComplete.
Including verbose details in case it helps you locate the fix. We are using the ServerService like this:
Which produces this stack trace