Error Using co-parser on ec2 manifest

[twitherspoon]

I am trying to provision AWS resources through CompatibleOne. The following is the steps I have taken just to make sure I'm going through the right process. And below my install/configuration steps is the error I am receiving. In addition, to the error message I never see ec2procci when I run co-status and I also can't get the dashboard to come up.

Thanks in advance for your assistance. -Tony

1. Launched Ubuntu Server 12.04.3 LTS on AWS EC2 (ami-6aad335a)
2. Installed the following tools (git, autoconf, make, lib tool, libssl-dev, uuid, uuid-dev, python-dev, python-boto, mysql-server, mysql-client, libmysqlclient-dev, libldap-dev)
3. Cloned accords-platform from github (http://github.com/compatibleone/accords-platform.git)
4. Ran export CFLAGS="-g -ggdb -O0"
5. cd accords-platform
6. Commented out PostgreSQL sections in configure.ac
7. Added or uncommented the following in scripts/accords.ini resthost= export OCCISQLTYPE=MYSQL export OCCISQLHOST=127.0.0.1 export OCCISQLUSER= export OCCISQLPASSWD=
8. Ran ./autogen.sh
9. Ran ./configure
10. Ran make clean
11. Ran make
12. Ran make install
13. ldconfig
14. sudo ./pyaccords_config.sh
15. cd to back to home directory
16. Ran sudo accords-platform/scripts/co-install small-accords
17. Ran co-stop
18. Updated accounts.xml with my account (admin)
19. Edited config_ec2procci.xml with my info (see below)
20. Added ec2manifest.xml with my info (see below)
21. Ran accords-config
22. Ran co-start
23. Ran co-parser against my ec2-manifest.xml

Here's the error I received:

=====Error Message===== ubuntu@ip-10-254-157-17:~/small-accords$ sudo co-parser ec2-manifest.xml Loading TLS configuration from : security/testcpTls.xml Rest Client Request : GET /-/ HTTP/1.1 Rest Client Request : GET /publication/ HTTP/1.1 Rest Client Request : GET /publication/0b83d5e1-0b0c-4228-9e18-801bca69520e HTTP/1.1 Rest Client Request : GET /-/ HTTP/1.1 Rest Client Request : GET /user/ HTTP/1.1 Rest Client Request : GET /user/b949011b-pars-400c-b6a7-c3f8a17f0ee6 HTTP/1.1 Rest Client Request : GET /publication/ HTTP/1.1 Rest Client Request : GET /publication/efed1926-54ed-4104-a0d5-d2a7a6abb629 HTTP/1.1 Rest Client Request : POST /authorization/ HTTP/1.1 CORDS Request Parser Phase 1

cords: error(4): parse error ec2-manifest.xml.xml Rest Client Request : DELETE /authorization/785d6e44-7152-4510-a204-d80f966598be HTTP/1.1 =====Error Message=====

=====config_ec2procci.xml===== <?xml version="1.0" encoding="UTF8"?>

```

=====config_ec2procci.xml=====

=====ec2-manifest.xml===== <?xml version="1.0" encoding="UTF8"?>

``` ```

=====ec2-manifest.xml=====

[ijm667]

Hello Tony, I am sorry for the delay in this response. I have looked into the issue and a correction has been made to the Amazon EC2 PROCCI to be aligned with the backend DataBase management now in place. I have been able to provision correctly using the Amazon EC2 PROCCI. In order to make this work for you I recommend the following course of action: stop your accords platform if it is running using: "co-stop"update your sources from the github using: "git pull"position the environment variable "export STRUKTSQL=1"rebuild the accords platform using "./autogen.sh; ./configure; make; make install;"rebuild the amazon ec2 procci using "./pyaccords_config.sh"* check that the correct python-boto package has been installed for your version of python. The accords platform production tools require at least the version 2.7 of python to do their work properly so you should have at least the python2.7-boto package installed and the setup.py script must be launched using : "python setup.py install" from the boto installation directory *restart your accords platform uing "co-start"check that the amazon ec2 procci is operational using: "ps -ef | grep ec2procci"copy the file accords-platform/configuration/config-ec2procci.xml to your working directoryedit the config-ec2procci.xml file to provide all the information indicated from your AWS EC2 subscription account where name="accords"create and edit a text file named csp-delete-ec2config.txt to contain the cordscript instructions shown here: $list = ec2config.list(); foreach ( $list as $item ) { $item.delete(); }delete the current amazon procci subscription information by launching the cordscript program created above using: "co-command run csp-delete-ec2config.txt"parse in the new subscription information using: "co-command config-ec2procci --xsd" Provisioning may now be performed using manifests and service level agreements which indicate the provider type of nodes to be provisioned as "amazonEc2. The image system name must resolve to a valid AMI that is accessible from your AWS subscription account. the value of "ubuntu" should be enough to choose the first available ubuntu image. I hope this helps you to move forward and please don't hesitate to contact me if required. SincerelyJamie Marshall

Date: Fri, 14 Feb 2014 12:38:57 -0800 From: notifications@github.com To: accords-platform@noreply.github.com Subject: [accords-platform] Error Using co-parser on ec2 manifest (#32)

I am trying to provision AWS resources through CompatibleOne. The following is the steps I have taken just to make sure I'm going through the right process. And below my install/configuration steps is the error I am receiving. In addition, to the error message I never see ec2procci when I run co-status and I also can't get the dashboard to come up.

Thanks in advance for your assistance.

-Tony

Launched Ubuntu Server 12.04.3 LTS on AWS EC2 (ami-6aad335a) Installed the following tools (git, autoconf, make, lib tool, libssl-dev, uuid, uuid-dev, python-dev, python-boto, mysql-server, mysql-client, libmysqlclient-dev, libldap-dev) Cloned accords-platform from github (http://github.com/compatibleone/accords-platform.git) Ran export CFLAGS="-g -ggdb -O0" cd accords-platform Commented out PostgreSQL sections in configure.ac Added or uncommented the following in scripts/accords.ini resthost= export OCCISQLTYPE=MYSQL export OCCISQLHOST=127.0.0.1 export OCCISQLUSER= export OCCISQLPASSWD= Ran ./autogen.sh Ran ./configure Ran make clean Ran make Ran make install ldconfig sudo ./pyaccords_config.sh cd to back to home directory Ran sudo accords-platform/scripts/co-install small-accords Ran co-stop Updated accounts.xml with my account (admin) Edited config_ec2procci.xml with my info (see below) Added ec2manifest.xml with my info (see below) Ran accords-config Ran co-start Ran co-parser against my ec2-manifest.xml Here's the error I received:

=====Error Message=====

ubuntu@ip-10-254-157-17:~/small-accords$ sudo co-parser ec2-manifest.xml

Loading TLS configuration from : security/testcpTls.xml

Rest Client Request : GET /-/ HTTP/1.1

Rest Client Request : GET /publication/ HTTP/1.1

Rest Client Request : GET /publication/0b83d5e1-0b0c-4228-9e18-801bca69520e HTTP/1.1

Rest Client Request : GET /-/ HTTP/1.1

Rest Client Request : GET /user/ HTTP/1.1

Rest Client Request : GET /user/b949011b-pars-400c-b6a7-c3f8a17f0ee6 HTTP/1.1

Rest Client Request : GET /publication/ HTTP/1.1

Rest Client Request : GET /publication/efed1926-54ed-4104-a0d5-d2a7a6abb629 HTTP/1.1

Rest Client Request : POST /authorization/ HTTP/1.1

CORDS Request Parser Phase 1

cords: error(4): parse error ec2-manifest.xml.xml

Rest Client Request : DELETE /authorization/785d6e44-7152-4510-a204-d80f966598be HTTP/1.1

=====Error Message=====

=====config_ec2procci.xml=====

<?xml version="1.0" encoding="UTF8"?>

<ec2config

    name="admin" 

authenticate="/home/ubuntu/ec2-west.pem"
accesskey="<insert access key id here>"
secretkey="<insert aws secret key here>"
user="admin" 
password="<insert password here>"
description="specify the user account description here" 

host="http://www.amazonaws.com" 
location="us-west-2" 
version="v1.1" 

namespace="accords" 
agent="EC2PROCCI/1.0a"

/>

=====config_ec2procci.xml=====

=====ec2-manifest.xml=====

<?xml version="1.0" encoding="UTF8"?>

        <image name=":ami-6aad335a" agent="none">
                <system name="ami-6aad335a" />
        </image>
</node>

=====ec2-manifest.xml=====

— Reply to this email directly or view it on GitHub.

[twitherspoon]

Hi Jaime, What do you mean by running "python setup.py install" from the boto installation directory. I see a few setup.py scripts none of which are in a boto directory.

I haven't done any of the other steps, but still have a few additional questions:

1. Do we not have to install small-accords any longer? That's where I use to run co-stop and co-start commands.
2. You state copying config-ec2procci.xml to my working directory. Can that be any directory? And is that basically any place where I'm running co-stop and co-start commands
3. Lastly after I have edited the proper files and the ec2procci service is running what actions must I take to complete provisioning in AWS. I want to make sure I have everything.

[ijm667]

Hello Mike, first of all ... you need to install the package python-boto which is the official python API for Amazon EC2. The package needs to be the right one with respect to your version of python which should be 2.7 or better. You should be able to find the boto packages using "aptitude search boto"

secondly ... by working directory, I mean your small-accords directory, and all commands to start and stop the platform etc should be run whilst that directory is your current working directory. thirdly ... once the right packages have been installed, and the files have been edited, and you are sure that the Amazon EC2 procci is running correctly, then you need to perform the following steps: 1) create a manifest2) parse the manifest into the accords platform3) create a service level agreement4) parse the SLA into the accords platform5) broker an instance of service from that agreement6) start the instance of service. All of these operations (except the number 1 - manifest creation - ) can be easily performed using the accords platform dashboard tool which is an Apache PHP application. To be able to use the dashboard you must:First install the Apache2 package by "aptitude install apache2"Then create a symbolic link from the Apache2 document root directory to your small-accords-platform directory. This would be something like: "ln -s /home/c2/small-accords /var/www/small-accords"Then ensure that the apache2 user has the rights to access the directory and to be able to create files and execute scripts by either make the apache2 user the owner or by giving the chmod other rights to the direcftory and its files. Just follow your standard practice for LAMP applications. A manifest can be created using a standard text editor and for a single provisioned machine would look something like this:

In the manifest above you should notice: a) the node with the provider element set to any or set to amazonEc2b) the system element of the image element of the node which identifies the "Ubuntu 13.04 Server" AMIc) the compute, storage and network elements can be adapted to suit your needs. To perform the sequence of operations through the demonstration dashboard please access your Accords Platform Dashboard on your machine and Login. The standard user and password for the demonstration dashboard are "accords" and "platform" respectively. 2) Upload and Parse Manifest Click on the TAB marked SLA and then upload and parse your manifest into the Accords Platform using the Choose File input element and the associated upload and parse submit button. 3 and 4) Create and Parse SLA Return to the TAB marked SLA and create an Agreement by filling in the fields as below: Name: "the-name-of-your-agreement-with-no-spaces"Description: "the-description-of-your-agreement-with-no-spaces"Account: "accords"Manifest: "select the name of your manifest in the drop down list"Algorithm: "default"Scripted: "not-important-for-now"Provider: "amazonEc2"Zone: "not-important-for-now" Then scroll down to the bottom of the SLA page and click on the generate sla and parse button. 5) Broker an Instance of Service from SLA Click on the TAB marked Menu and then on Agreements then scroll down the page to localise your newly created agreement. Click on the ICON to the right marked Create Service Instance and wait till the operation has completed and the page changes to give the instance creation report followed by the Service Instance list. 6) Start the Instance of Service From the Service Instance list, which may be reached by clicking on the TAB marked Menu and then on Services , localise your newly created instance of service and then click on the start service button and wait till the page changes to give the start action report followed by the Service Instance list. I hope this helps you to reach your goal. SincerelyJamie Marshall

Date: Thu, 20 Feb 2014 11:55:19 -0800 From: notifications@github.com To: accords-platform@noreply.github.com CC: ijm667@hotmail.com Subject: Re: [accords-platform] Error Using co-parser on ec2 manifest (#32)

Hi Jaime,

What do you mean by running "python setup.py install" from the boto installation directory. I see a few setup.py scripts none of which are in a boto directory.

I haven't done any of the other steps, but still have a few additional questions:

1. Do we not have to install small-accords any longer? That's where I use to run co-stop and co-start commands.
2. You state copying config-ec2procci.xml to my working directory. Can that be any directory? And is that basically any place where I'm running co-stop and co-start commands
3. Lastly after I have edited the proper files and the ec2procci service is running what actions must I take to complete provisioning in AWS. I want to make sure I have everything.

— Reply to this email directly or view it on GitHub.

[twitherspoon]

Hi Jaime, When I run sudo co.start it hangs at "Starting accords security services \n CORDS Request Parser Phase 1" It's been hanging there for a good while 15 plus minutes.

I'll try a fresh image and see if that resolves it.

I'm using a Cannonical Ubuntu image from AWS, so I it already has boto installed

[twitherspoon]

Nope started a fresh build and it's stuck at CORDS Request Parser Phase 1

Any known issues running accords on an AWS instance?

I'll try this one more time in a VM on my Mac using VMware Fusion, but ultimately I wan to run this in AWS, so others can connect to it.

[twitherspoon]

I ran the install on a ubuntu vm on my Mac and it's stuck in the same place CORDS Request Parser Phase 1.

Only difference it hangs for a while and outputs parser error messages and moves to the next parsing tasks. For example here are a few of the error messages:

Cords Parser Error: * * * errors in ./plan_accounts.xml

Cords Parser Error: * * * errors in ./plan_metrics.xml * * *

Cords Parser Error: * * * errors in ./plan_coips.xml

[ijm667]

Tony, I have been having the same problems and I have got to the bottom of it. The parsing operation uses a schema for the control of the description of the document. This schema is located on the www.compatibleone.fr server which is a physical machine in a 24/24 data center.It is very unfortunate, this base machine, providing this 24/24 schema support has just been rellocated by the data center in which it currently hosted for data center policy reasons. Since they dont manage the domain name there has been an inconsistancy. I have now corrected this and the domain name www.compatibleone.fr is now pointing to the correct machine at 195.59.192.59 . I am very sorry for the inconvenience that this has caused. SincerelyJamie

Date: Mon, 24 Feb 2014 08:21:34 -0800 From: notifications@github.com To: accords-platform@noreply.github.com CC: ijm667@hotmail.com Subject: Re: [accords-platform] Error Using co-parser on ec2 manifest (#32)

Nope started a fresh build and it's stuck at CORDS Request Parser Phase 1

Any known issues running accords on an AWS instance?

I'll try this one more time in a VM on my Mac using VMware Fusion, but ultimately I wan to run this in AWS, so others can connect to it.

— Reply to this email directly or view it on GitHub.

[twitherspoon]

I think we are really close. Here's my latest error. What I did was within ec2config.xml I inputed my AWS Access Key, My Secret Key, and for authenticate I inserted the path to one of my aws key pair files that I initially created in the Amazon console. I think the issue is with the key pair I'm using and can't figure out why. Based on the error message below it seems like the application wants to import the key pair, but I'm using an already existing key pair.

Thanks in advance.

=====Error Message=====

File "/home/ubuntu/accords-platform/pyaccords/pygen/amazonEc2Act.py", line 28, in start resCateg = amazonEc2_start(accesskey,secretkey,zone,keypair,categoryAtr) File "/home/ubuntu/accords-platform/pyaccords/pygen/amazonEc2Action.py", line 93, in amazonEc2_start key = conn.import_key_pair(mykeyname,mykey) File "/usr/lib/python2.7/dist-packages/boto/ec2/connection.py", line 1821, in import_key_pair return self.get_object('ImportKeyPair', params, KeyPair, verb='POST') File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 916, in get_object raise self.ResponseError(response.status, response.reason, body) boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request <?xml version="1.0" encoding="UTF-8"?>

InvalidParameterValueValue (a ton of random letters and numbers) for parameter PublicKeyMaterial is invalid. Length exceeds maximum of 2048.cb3f6e55-fd38-444e-a1da-161268da9e27

[ijm667]

Hello Tony, you are in fact very close now. I have had a look at the configuration of my EC2 account and in fact the authorization field of the ec2 configuration subscription record points to the PUB file, of 381 bytes long, containing my SSL KEY, which starts by:
ssh-rsa AAAAB3NzaC1y .................... I imagine that your file is simply the PEM file, of 1692 bytes long(ish), that was returned after the key creation operation and starts with: -----BEGIN RSA PRIVATE KEY-----MIIEowIB ................ Information is given here on the AWS web site http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingUserCerts.html#Using_UploadCertificate relating to the conversion of a private key to produce a certificate and in particular the operations indicated: Create the User Signing CertificateYou can now create a user signing certificate.To create a user signing certificateUse the openssl req command and the following syntaxopenssl req -new -x509 -nodes -sha1 -days 365 -key private-key.pem -outform PEM > certificate.pem when you have this certificate the full pathname to the filename in which is contained must be used as the value of the authorization field of the ec2 configuration. This should allow the provisioning operation to proceed. Subscription information of this kind is very specific from provider to provider. I hope this information helps you get it operational for you at last. SincerelyJamie

Date: Tue, 25 Feb 2014 11:55:55 -0800 From: notifications@github.com To: accords-platform@noreply.github.com CC: ijm667@hotmail.com Subject: Re: [accords-platform] Error Using co-parser on ec2 manifest (#32)

I think we are really close. Here's my latest error. What I did was within ec2config.xml I inputed my AWS Access Key, My Secret Key, and for authenticate I inserted the path to one of my aws key pair files that I initially created in the Amazon console. I think the issue is with the key pair I'm using and can't figure out why. Based on the error message below it seems like the application wants to import the key pair, but I'm using an already existing key pair.

Thanks in advance.

=====Error Message=====

File "/home/ubuntu/accords-platform/pyaccords/pygen/amazonEc2Act.py", line 28, in start

resCateg = amazonEc2_start(accesskey,secretkey,zone,keypair,categoryAtr)

File "/home/ubuntu/accords-platform/pyaccords/pygen/amazonEc2Action.py", line 93, in amazonEc2_start

key = conn.import_key_pair(mykeyname,mykey)

File "/usr/lib/python2.7/dist-packages/boto/ec2/connection.py", line 1821, in import_key_pair

return self.get_object('ImportKeyPair', params, KeyPair, verb='POST')

File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 916, in get_object

raise self.ResponseError(response.status, response.reason, body)

boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request

<?xml version="1.0" encoding="UTF-8"?> InvalidParameterValueValue (a ton of random letters and numbers) for parameter PublicKeyMaterial is invalid. Length exceeds maximum of 2048.cb3f6e55-fd38-444e-a1da-161268da9e27

— Reply to this email directly or view it on GitHub.

[twitherspoon]

That didn't work. How did you create your key? Did you use the link you sent me? Here's the error I'm not getting:

File "/home/ubuntu/accords-platform/pyaccords/pygen/amazonEc2Act.py", line 28, in start resCateg = amazonEc2_start(accesskey,secretkey,zone,keypair,categoryAtr) File "/home/ubuntu/accords-platform/pyaccords/pygen/amazonEc2Action.py", line 93, in amazonEc2_start key = conn.import_key_pair(mykeyname,mykey) File "/usr/lib/python2.7/dist-packages/boto/ec2/connection.py", line 1821, in import_key_pair return self.get_object('ImportKeyPair', params, KeyPair, verb='POST') File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 916, in get_object raise self.ResponseError(response.status, response.reason, body) boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request <?xml version="1.0" encoding="UTF-8"?>

InvalidKey.FormatKey is not in valid OpenSSH public key format38a5c2a0-ced3-465e-9276-2d91261fc074

ubuntu@ip-10-231-32-201:~/small-accords$ vi ec2config.xml ubuntu@ip-10-231-32-201:~/small-accords$ vi ../ec2manifest.xml

[twitherspoon]

Well Jamie, it's finally "working". I got pass the certificate issue. I pointed to the wrong file. I was pointing to the certificate file instead of the .pub file. So that got me past that issue.

The next issue I ran into is that CompatilbeOne is launching the wrong AMI. It only wants to launch ami-ad184ac4. And it's not using the AMI I specified in my manifest file. The region I specified is US-West-2 and since the AMI its trying to launch is in East it failed. I changed the region to US-East and it worked fine.

I did a grep for ami-ad184ac4 and found it in ec2client.pyc and amazonec2.xml, so I'm not sure which file CompatibleOne is pulling that ami from.

Lastly and this may be my issue with the cert I created is I couldn't log into the instance after it launched. It kept asking for a passphrase, but I don't recall linking one to the cert I created.

[ijm667]

Hello Tony, I am glad that you have finally managed to launch the instance.I have had a day off today after the hectic week at cloud expo london and all the preparation before.From my side I have not had any trouble logging into the launched machine but I am sure I didnt set a pass phrase, even a space would make a difference in this area. Maybe try and create a new key file and certificate and use that combination for the deployment of the machine. SincerelyJamie

Date: Fri, 28 Feb 2014 07:20:55 -0800 From: notifications@github.com To: accords-platform@noreply.github.com CC: ijm667@hotmail.com Subject: Re: [accords-platform] Error Using co-parser on ec2 manifest (#32)

Well Jamie, it's finally "working". I got pass the certificate issue. I pointed to the wrong file. I was pointing to the certificate file instead of the .pub file. So that got me past that issue.

The next issue I ran into is that CompatilbeOne is launching the wrong AMI. It only wants to launch ami-ad184ac4. And it's not using the AMI I specified in my manifest file. The region I specified is US-West-2 and since the AMI its trying to launch is in East it failed. I changed the region to US-East and it worked fine.

I did a grep for ami-ad184ac4 and found it in ec2client.pyc and amazonec2.xml, so I'm not sure which file CompatibleOne is pulling that ami from.

Lastly and this may be my issue with the cert I created is I couldn't log into the instance after it launched. It kept asking for a passphrase, but I don't recall linking one to the cert I created.

— Reply to this email directly or view it on GitHub.