search

CloudOpting / cloudopting-manager

The cloudopting core manager
MIT License
10 stars 7 forks source link

DigitalOcean flow orchestration #386

Closed cpopsa closed 7 years ago

cpopsa commented 8 years ago

The code for DigitalOcean interaction needs to be integrated into activiti orchestration. We have two options:

  1. We map the existing flow for cloudstack to be used for DigitalOcean also. In this way, gateway 14 need to be updated to support both cloudstack (cloudId == 1) and DigitalOcean (cloudId == 4)
  2. We create a new subprocess for DigitalOcean, and integrate it into to main orchestration process.
gioppoluca commented 8 years ago

A the moment there are 2 subprocesses one for Azure and onother for vmware we need to add another for digital ocean and place there all the steps needed. Parts for CloudStack cannot be touched

xeviscc commented 7 years ago

@cpopsa @ildave @ciprianpavel @guidospadotto-profesia Please coordinate in order to solve this issue and to integrate it with the activiti BPMN. Contact @gioppoluca if help needed.

guidospadotto-profesia commented 7 years ago

Maybe @ildave was not clear enough: we have never touched the Orchestration Process since its very start, so we do not know how it works and how to change it. @LucaGioppo knows it.

2016-07-15 10:41 GMT+02:00 Xavier Cases Camats notifications@github.com:

@cpopsa https://github.com/cpopsa @ildave https://github.com/ildave @ciprianpavel https://github.com/ciprianpavel @guidospadotto-profesia https://github.com/guidospadotto-profesia Please coordinate in order to solve this issue and to integrate it with the activiti BPMN. Contact @gioppoluca https://github.com/gioppoluca if help needed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CloudOpting/cloudopting-manager/issues/386#issuecomment-232895498, or mute the thread https://github.com/notifications/unsubscribe-auth/ALnoZix63IpClos4lEE3atUUXS9uhQToks5qV0ewgaJpZM4JDFYD .

xeviscc commented 7 years ago

Noone from Profesia was on the meeting today, so you do not know, but you should know as we changed responsibility from Activiti long ago. Luca is doing other stuff, so we stated again that if you have doubts or need for help, ask Luca, but all related issues to Activiti are Profesia's responsibility. Not meaning that Teamnet cannot colaborate, but you have to coordinate.

guidospadotto-profesia commented 7 years ago

I see, I missed this information.

Thank you Xavier

2016-07-15 11:37 GMT+02:00 Xavier Cases Camats notifications@github.com:

Noone from Profesia was on the meeting today, so you do not know, but you should know as we changed responsibility from Activiti long ago. Luca is doing other stuff, so we stated again that if you have doubts or need for help, ask Luca, but all related issues to Activiti are Profesia's responsibility. Not meaning that Teamnet cannot colaborate, but you have to coordinate.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CloudOpting/cloudopting-manager/issues/386#issuecomment-232906919, or mute the thread https://github.com/notifications/unsubscribe-auth/ALnoZkkBSVO-9oa09TuRJ4JiymwvRPjRks5qV1TtgaJpZM4JDFYD .

cpopsa commented 7 years ago

DigitalOcean subprocess task's:

  1. Generate DigitalOcean VM (like DeployGenerateVm)
    • Input variables:
      • String customizationId (id of customization)
      • Long cloudAccountId (id of cloudaccount)
    • Output variables:
      • String vmId (id of VM droplet)
  2. Check DigitalOcean VM created (like DeployGenerateVm)
    • Input variables:
      • String vmId (id of VM droplet filled in step 1)
      • Long cloudAccountId (id of cloudaccount)
    • Output variables:
      • boolean vmCreated (if vm is created or not)
  3. Get DigitalOcean VM IP (like DeployCheckAcquireIp)
    • Input variables:
      • String vmId (id of VM droplet filled in step 1)
      • Long cloudAccountId (id of cloudaccount)
    • Output variables:
      • String ip (VM droplet IP)

Order of execution:

Subprocess entrypoint in the main process:

Subprocess exitpoint in the main process:

xeviscc commented 7 years ago

@ildave Could you give an update of the status of this task?

xeviscc commented 7 years ago

@cpopsa @ildave Could you give an update of the status of this task?

cpopsa commented 7 years ago

@ildave - Please let me know if you need some other info

ildave commented 7 years ago

I was able to start the deploy on DigitalOcean, but I get an exception when executing the DigitaloceaProvion.provisionVM() method:

Caused by: java.lang.RuntimeException: Ssh Key not found for fingerprint XXXXXX
    at eu.cloudopting.provision.digitalocean.DigitaloceanProvision.provisionVM(DigitaloceanProvision.java:42) ~[cloud-component-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]

The error is triggered by this line: Key sshKey = api.keyApi().get(request.getIdentity());

The jclouds library tries to call this URL:

[DEBUG] jclouds.headers - >> GET https://api.digitalocean.com/v2/account/keys/<<apikey>> HTTP/1.1 but it receives 404:

[DEBUG] org.jclouds.http.internal.JavaUrlHttpCommandExecutorService - Receiving response -306271885: HTTP/1.1 404 Not Found
[DEBUG] jclouds.headers - << HTTP/1.1 404 Not Found
[DEBUG] jclouds.headers - << Status: 404 Not Found

I have configured a new cloud account for digital ocean using the token I received from Xicota as api_key and secret_key, and using https://api.digitalocean.com/v2/ as the endpoint

@ciprianpavel and @cpopsa, is this configuration correct? You should provide us the precise informations to how to correctly configure the DigitalOcean account in order to be able to go on with the process.

gioppoluca commented 7 years ago

Please bring this talk by mail if writing credentials in clear

cpopsa commented 7 years ago

I made some changes, it should work now. I also modified DigitalOcean record in cloud_accouns. Please let me know if you still have problems.

cpopsa commented 7 years ago

In DigitaloceanProvisionTests.java you will find some examples of how to invoke the api for DigitalOcean provision.

ildave commented 7 years ago

I have tried again with the modified code, and this time I get a different error:

[DEBUG] jclouds.headers - >> POST https://api.digitalocean.com/v2/droplets HTTP/1.1
[DEBUG] jclouds.headers - >> Accept: application/json
[DEBUG] jclouds.headers - >> Authorization: Bearer XXXX
[DEBUG] jclouds.headers - >> Content-Type: application/json
[DEBUG] jclouds.headers - >> Content-Length: 738
[TRACE] sun.net.www.protocol.http.HttpURLConnection - KeepAlive stream retrieved from the cache, sun.net.www.protocol.https.HttpsClient(https://api.digitalocean.com/v2/images?type=distribution)
[DEBUG] sun.net.www.protocol.http.HttpURLConnection - sun.net.www.MessageHeader@5f40e5f08 pairs: {POST /v2/droplets HTTP/1.1: null}{Accept: application/json}{Authorization: Bearer XXXX}{User-Agent: jclouds/1.9.2 java/1.8.0_51}{Content-Type: application/json}{Host: api.digitalocean.com}{Connection: keep-alive}{Content-Length: 738}
[DEBUG] sun.net.www.protocol.http.HttpURLConnection - sun.net.www.MessageHeader@6826f4a318 pairs: {null: HTTP/1.1 403 Forbidden}{Date: Wed, 05 Oct 2016 15:35:17 GMT}{Content-Type: application/json; charset=utf-8}{Transfer-Encoding: chunked}{Connection: keep-alive}{Set-Cookie: __cfduid=de1c0c2167b22625cffd8f21cb73cfb081475681716; expires=Thu, 05-Oct-17 15:35:16 GMT; path=/; domain=.digitalocean.com; HttpOnly}{Status: 403 Forbidden}{X-Frame-Options: SAMEORIGIN}{X-XSS-Protection: 1; mode=block}{X-Content-Type-Options: nosniff}{RateLimit-Limit: 5000}{RateLimit-Remaining: 4880}{RateLimit-Reset: 1475684288}{Cache-Control: no-cache}{X-Request-Id: 43d2719f-f39f-4304-b686-b934bd048af1}{X-Runtime: 0.067079}{Server: cloudflare-nginx}{CF-RAY: 2ed1f14b3aea0e4e-MXP}
[DEBUG] org.jclouds.http.internal.JavaUrlHttpCommandExecutorService - Receiving response 548275143: HTTP/1.1 403 Forbidden
[DEBUG] jclouds.headers - << HTTP/1.1 403 Forbidden
[DEBUG] jclouds.headers - << Status: 403 Forbidden
[DEBUG] jclouds.headers - << Transfer-Encoding: chunked
[DEBUG] jclouds.headers - << CF-RAY: 2ed1f14b3aea0e4e-MXP
[DEBUG] jclouds.headers - << Server: cloudflare-nginx
[DEBUG] jclouds.headers - << RateLimit-Reset: 1475684288
[DEBUG] jclouds.headers - << X-Request-Id: 43d2719f-f39f-4304-b686-b934bd048af1
[DEBUG] jclouds.headers - << X-Runtime: 0.067079
[DEBUG] jclouds.headers - << X-Content-Type-Options: nosniff
[DEBUG] jclouds.headers - << Connection: keep-alive
[DEBUG] jclouds.headers - << RateLimit-Limit: 5000
[DEBUG] jclouds.headers - << Date: Wed, 05 Oct 2016 15:35:17 GMT
[DEBUG] jclouds.headers - << X-Frame-Options: SAMEORIGIN
[DEBUG] jclouds.headers - << RateLimit-Remaining: 4880
[DEBUG] jclouds.headers - << Cache-Control: no-cache
[DEBUG] jclouds.headers - << Set-Cookie: __cfduid=de1c0c2167b22625cffd8f21cb73cfb081475681716; expires=Thu, 05-Oct-17 15:35:16 GMT; path=/; domain=.digitalocean.com; HttpOnly
[DEBUG] jclouds.headers - << X-XSS-Protection: 1; mode=block
[DEBUG] jclouds.headers - << Content-Type: application/json; charset=utf-8
[DEBUG] jclouds.wire - << "{"id":"forbidden","message":"You have reached your droplet limit."}"
[ERROR] org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable - exception during job execution: Job 107556 failed
org.activiti.engine.ActivitiException: Job 107556 failed
[...]
at eu.cloudopting.provision.digitalocean.DigitaloceanProvision.provisionVM(DigitaloceanProvision.java:62) ~[cloud-component-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
[... more stack trace below...]

Also, at the moment I cannot see what really happens on the Digital Ocean platform: is it possible to have access to the DO console to check if the VMs are starting?

cpopsa commented 7 years ago

The error is [DEBUG] jclouds.wire - << "{"id":"forbidden","message":"You have reached your droplet limit."}". Apparently the DigitalOcean account has a constraint on number of created droplets. I runned the listAllDroplets() method from DigitaloceanProvisionTests, and find out that currently there are 25 vm's running, 24 of these were created in just one minute (18:34) :) Maybe you tried something and didn't work as you expected. You should delete those 24 droplets or Xavi will kill us :) . If you want I can delete them for you. Unfortunately I don't have the credentials for accessing DigitalOcean web console, but maybe listAllDroplets() method would help you.

ildave commented 7 years ago

I get an exception in the DigitaloceanProvision call, calling this line of code:

Key sshKey = api.keyApi().get(request.getIdentity());

[DEBUG] org.jclouds.rest.internal.InvokeHttpMethod - >> invoking key:get
[DEBUG] org.jclouds.http.internal.JavaUrlHttpCommandExecutorService - Sending request 1887309476: GET https://api.digitalocean.com/v2/account/keys/%20 HTTP/1.1
[DEBUG] jclouds.headers - >> GET https://api.digitalocean.com/v2/account/keys/%20 HTTP/1.1
[DEBUG] jclouds.headers - >> Accept: application/json
[DEBUG] jclouds.headers - >> Authorization: Bearer 1990101366ffe7513d52957a68a090ac453f0ff66bf1f2edfa4d736c97f9c0b2
[TRACE] sun.net.www.protocol.http.HttpURLConnection - KeepAlive stream retrieved from the cache, sun.net.www.protocol.https.HttpsClient(https://api.digitalocean.com/v2/images?type=distribution)
[DEBUG] sun.net.www.protocol.http.HttpURLConnection - sun.net.www.MessageHeader@1146dcf86 pairs: {GET /v2/account/keys/%20 HTTP/1.1: null}{Accept: application/json}{Authorization: Bearer 1990101366ffe7513d52957a68a090ac453f0ff66bf1f2edfa4d736c97f9c0b2}{User-Agent: jclouds/1.9.2 java/1.8.0_51}{Host: api.digitalocean.com}{Connection: keep-alive}
[DEBUG] sun.net.www.protocol.http.HttpURLConnection - sun.net.www.MessageHeader@d35c4e419 pairs: {null: HTTP/1.1 404 Not Found}{Date: Tue, 18 Oct 2016 14:25:44 GMT}{Content-Type: application/json; charset=utf-8}{Transfer-Encoding: chunked}{Connection: keep-alive}{Set-Cookie: __cfduid=d86ba6fd6874f360972566d7314a02cb91476800744; expires=Wed, 18-Oct-17 14:25:44 GMT; path=/; domain=.digitalocean.com; HttpOnly}{Cache-Control: no-cache}{Ratelimit-Limit: 5000}{Ratelimit-Remaining: 4976}{Ratelimit-Reset: 1476801862}{Status: 404 Not Found}{X-Content-Type-Options: nosniff}{X-Frame-Options: SAMEORIGIN}{X-Gateway: Edge Gateway}{X-Request-Id: 90e2ae54-898c-44f0-90c1-6fe3cb8a7613}{X-Runtime: 0.021637}{X-Xss-Protection: 1; mode=block}{Server: cloudflare-nginx}{CF-RAY: 2f3ca94c5cfe3d89-MXP}
[DEBUG] org.jclouds.http.internal.JavaUrlHttpCommandExecutorService - Receiving response 1887309476: HTTP/1.1 404 Not Found
[DEBUG] jclouds.headers - << HTTP/1.1 404 Not Found
[DEBUG] jclouds.headers - << Status: 404 Not Found
[DEBUG] jclouds.headers - << Transfer-Encoding: chunked
[DEBUG] jclouds.headers - << X-Gateway: Edge Gateway
[DEBUG] jclouds.headers - << CF-RAY: 2f3ca94c5cfe3d89-MXP
[DEBUG] jclouds.headers - << Server: cloudflare-nginx
[DEBUG] jclouds.headers - << X-Request-Id: 90e2ae54-898c-44f0-90c1-6fe3cb8a7613
[DEBUG] jclouds.headers - << X-Runtime: 0.021637
[DEBUG] jclouds.headers - << X-Content-Type-Options: nosniff
[DEBUG] jclouds.headers - << Ratelimit-Remaining: 4976
[DEBUG] jclouds.headers - << Connection: keep-alive
[DEBUG] jclouds.headers - << Ratelimit-Reset: 1476801862
[DEBUG] jclouds.headers - << Date: Tue, 18 Oct 2016 14:25:44 GMT
[DEBUG] jclouds.headers - << X-Frame-Options: SAMEORIGIN
[DEBUG] jclouds.headers - << Ratelimit-Limit: 5000
[DEBUG] jclouds.headers - << Cache-Control: no-cache
[DEBUG] jclouds.headers - << X-Xss-Protection: 1; mode=block
[DEBUG] jclouds.headers - << Set-Cookie: __cfduid=d86ba6fd6874f360972566d7314a02cb91476800744; expires=Wed, 18-Oct-17 14:25:44 GMT; path=/; domain=.digitalocean.com; HttpOnly
[DEBUG] jclouds.headers - << Content-Type: application/json; charset=utf-8
[DEBUG] jclouds.wire - << "{"id":"not_found","message":"The resource you were accessing could not be found."}"
[ERROR] org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable - exception during job execution: Job 172610 failed
org.activiti.engine.ActivitiException: Job 172610 failed
    at org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:110) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]
    at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]

The exception causes the process to get stuck and to restart, creating more and more VM.

@cpopsa , can you check?

cpopsa commented 7 years ago

The line Key sshKey = api.keyApi().get(request.getIdentity()); was removed in 05.10.2016 commit (message: Fix DigitalOcean provision). Apparently you have some problems in your local project. If you work in a different branch, you should merge from master.

ildave commented 7 years ago

Yes, I had something wrong in my project. I fixed it and I will do some more tests later.

Thanks

ildave commented 7 years ago

Ok, I run in another exception:

[ERROR] org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable - exception during job execution: Job 202550 failed
org.activiti.engine.ActivitiException: Job 202550 failed
    at org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:110) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]
    at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]
    at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]
    at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:47) ~[activiti-spring-5.19.0.2.jar:5.19.0.2]
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45) ~[activiti-spring-5.19.0.2.jar:5.19.0.2]
    at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]
    at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]
    at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]
    at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.handleMultipleJobs(ExecuteJobsRunnable.java:94) [activiti-engine-5.19.0.2.jar:5.19.0.2]
    at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:49) [activiti-engine-5.19.0.2.jar:5.19.0.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
Caused by: java.lang.NullPointerException: null
    at eu.cloudopting.provision.digitalocean.DigitaloceanProvision.provisionVM(DigitaloceanProvision.java:50) ~[cloud-component-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
    at eu.cloudopting.provision.digitalocean.DigitaloceanProvision.provisionVM(DigitaloceanProvision.java:30) ~[cloud-component-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
    at eu.cloudopting.cloud.CloudService.createVM(CloudService.java:134) ~[cloud-component-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
    at eu.cloudopting.bpmn.tasks.deploy.DeployGenerateDoVm.execute(DeployGenerateDoVm.java:45) ~[bpmn-component-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
    at org.activiti.engine.impl.delegate.JavaDelegateInvocation.invoke(JavaDelegateInvocation.java:34) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]
    at org.activiti.engine.impl.delegate.DelegateInvocation.proceed(DelegateInvocation.java:37) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]
    at org.activiti.engine.impl.delegate.DefaultDelegateInterceptor.handleInvocation(DefaultDelegateInterceptor.java:25) ~[activiti-engine-5.19.0.2.jar:5.19.0.2]
[...more stuff...]

The issue seems to be in this call in the provisionVM() method of the DigitaloceanProvision class:

Image image = getImage(api, region);

that returns null.

@cpopsa can you please check?

cpopsa commented 7 years ago

Fixed, please update your local project.

ildave commented 7 years ago

I was able to go further in the process but now I get a NullPointerException when getting the IP address of the VM:

java.lang.NullPointerException: null
    at eu.cloudopting.bpmn.tasks.deploy.DeployDoVmIp.execute(DeployDoVmIp.java:45)

The null object comes from this line in the DeployDoVmIp class:

JSONObject result = cloudService.getAssociatedIpinfo(cloudParams);

that in the end calls the getAcquiredIpinfo() method of the DigitaloceanProvision class.

Can you check this @cpopsa ?

Thanks!

ildave commented 7 years ago

Hello, any update on this?

cpopsa commented 7 years ago

You call getAssociatedIpinfo with param cloudAccountId = -1 which does not exists. A similar problem is in DeployDoVmCheck.java. You call cloudService.checkVM(4l, cloudtask). I don't think that the cloudAccountId should be hardcoded, please check. cloudService.checkVM(cloudAccountId, cloudtask) should be used instead.