search

GENI-NSF / geni-tools

Omni, stitcher, GCF sample aggregate manager, and other GENI tools.
Other
11 stars 15 forks source link

readyToLogin fails when omni falls over to use v2 with offered a v3 AM URL #643

Closed seledwards closed 9 years ago

seledwards commented 9 years ago

Calling readyToLogin with the v3 AM url for a GRAM/OpenGENI rack fails:

$ readyToLogin.py -r GEC20-Clemson demo -a https://clemson-clemson-control-1.clemson.edu:5001
17:59:45 WARNING : Requested API version 2, but AM https://clemson-clemson-control-1.clemson.edu:5001 uses version 3. Same aggregate talks API v2 at a different URL: https://clemson-clemson-control-1.clemson.edu:5002
ERROR: No manifest found from listresources at https://clemson-clemson-control-1.clemson.edu:5001; review the logs.
No login information found!!

Note that https://clemson-clemson-control-1.clemson.edu:5001 is the v3 url and the code is attempting to switch to the v2 url.

In particular, if I explicitly use the v2 url it works and gives the expected output:

 readyToLogin.py -r GEC20-Clemson demo -a https://clemson-clemson-control-1.clemson.edu:5002

Moreover if I call listresources a manifest is actually returned which implies that the error message is wrong:

omni.py listresources -r GEC20-Clemson demo -a https://clemson-clemson-control-1.clemson.edu:5001 -V2 -o

Note that readyToLogin is known to not work with v3.

Imported from trac ticket #643, created by sedwards on 06-12-2014 at 18:10, last modified: 06-13-2014 at 19:01 CCing: gpo-expt-support@...

seledwards commented 9 years ago 
$ readyToLogin.py -r GEC20-Clemson demo -a https://clemson-clemson-control-1.clemson.edu:5001 --debug
18:12:51 INFO    : Loading agg_nick_cache file '/Users/sedwards/.gcf/agg_nick_cache'
18:12:51 INFO    : Loading config file /Users/sedwards/.gcf/omni_config
18:12:51 INFO    : Using control framework portal
18:12:51 DEBUG   : Using framework type chapi
18:12:51 INFO    : Member Authority is https://ch.geni.net/MA (from config)
18:12:51 INFO    : Slice Authority is https://ch.geni.net/SA (from config)
18:12:51 DEBUG   : CH speaks CHAPI v1
18:12:51 DEBUG   : User Cert File: /Users/sedwards/.ssl/geni_cert_portal.pem
18:12:51 INFO    : Loading agg_nick_cache file '/Users/sedwards/.gcf/agg_nick_cache'
18:12:51 INFO    : Loading config file /Users/sedwards/.gcf/omni_config
18:12:51 INFO    : Using control framework portal
18:12:51 DEBUG   : Using framework type chapi
18:12:51 INFO    : Member Authority is https://ch.geni.net/MA (from config)
18:12:51 INFO    : Slice Authority is https://ch.geni.net/SA (from config)
18:12:51 DEBUG   : CH speaks CHAPI v1
18:12:51 DEBUG   : User Cert File: /Users/sedwards/.ssl/geni_cert_portal.pem
18:12:51 INFO    : Python: CPython 2.7.6
OS: Darwin-12.5.0-x86_64-i386-64bit
Omni: GENI Omni Command Line Aggregate Manager Tool Version 2.7
Copyright (c) 2014 Raytheon BBN Technologies
18:12:51 DEBUG   : Adding aggregate https://clemson-clemson-control-1.clemson.edu:5001 (unspecified_AM_URN) to query list
18:12:51 DEBUG   : Read GetVersionCache from /Users/sedwards/.gcf/get_version_cache.json
18:12:51 DEBUG   : Checking cache for https://clemson-clemson-control-1.clemson.edu:5001
18:12:51 DEBUG   : Pulling GetVersion from cache
18:12:51 DEBUG   : Config version spoken somewhere by all AMs
18:12:51 DEBUG   : Actually calling GetVersion
18:12:51 DEBUG   :  ... opts.noGetVersionCache set
18:12:51 DEBUG   : Added GetVersion success output to cache for https://clemson-clemson-control-1.clemson.edu:5001
18:12:51 DEBUG   : Wrote GetVersionCache to /Users/sedwards/.gcf/get_version_cache.json
18:12:51 INFO    : AM URN: unspecified_AM_URN (url: https://clemson-clemson-control-1.clemson.edu:5001) has version:
18:12:51 INFO    : {   'GRAM_version': '3.33',
    'geni_ad_rspec_versions': [   {   'extensions': [],
                                      'namespace': 'http://www.geni.net/resources/rspec/3',
                                      'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
                                      'type': 'GENI',
                                      'version': '3'}],
    'geni_am_type': 'GRAM',
    'geni_am_urn': 'urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am',
    'geni_api': 3,
    'geni_api_versions': {   '2': 'https://clemson-clemson-control-1.clemson.edu:5002',
                             '3': 'https://clemson-clemson-control-1.clemson.edu:5001'},
    'geni_credential_types': [{   'geni_type': 'geni_sfa',
                                     'geni_version': '3'}],
    'geni_request_rspec_versions': [   {   'extensions': [],
                                           'namespace': 'http://www.geni.net/resources/rspec/3',
                                           'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
                                           'type': 'GENI',
                                           'version': '3'}]}
18:12:51 INFO    : Loading agg_nick_cache file '/Users/sedwards/.gcf/agg_nick_cache'
18:12:51 INFO    : Loading config file /Users/sedwards/.gcf/omni_config
18:12:51 INFO    : Using control framework portal
18:12:51 DEBUG   : Using framework type chapi
18:12:51 INFO    : Member Authority is https://ch.geni.net/MA (from config)
18:12:51 INFO    : Slice Authority is https://ch.geni.net/SA (from config)
18:12:51 DEBUG   : CH speaks CHAPI v1
18:12:51 DEBUG   : User Cert File: /Users/sedwards/.ssl/geni_cert_portal.pem
18:12:51 INFO    : Python: CPython 2.7.6
OS: Darwin-12.5.0-x86_64-i386-64bit
Omni: GENI Omni Command Line Aggregate Manager Tool Version 2.7
Copyright (c) 2014 Raytheon BBN Technologies
18:12:51 DEBUG   : Found slice name demo
18:12:51 DEBUG   : Adding aggregate https://clemson-clemson-control-1.clemson.edu:5001 (unspecified_AM_URN) to query list
18:12:51 DEBUG   : Read GetVersionCache from /Users/sedwards/.gcf/get_version_cache.json
18:12:51 DEBUG   : Checking cache for https://clemson-clemson-control-1.clemson.edu:5001
18:12:51 DEBUG   : Pulling GetVersion from cache
18:12:51 DEBUG   : Config version spoken somewhere by all AMs
18:12:51 DEBUG   : Using project from commandline.
18:12:51 DEBUG   : No filename provided for credential
18:12:51 DEBUG   : add_c_n_spkfor start with self.opts.speaksfor = 'None'
18:12:51 DEBUG   : add_c_n_spkfor start with self.opts.cred = None
18:12:51 DEBUG   : add_c_n_spkfor new_creds = []; new_options = {'match': {'SLICE_EXPIRED': 'f', 'SLICE_URN': 'urn:publicid:IDN+ch.geni.net:GEC20-Clemson+slice+demo'}}
18:12:51 INFO    : Slice urn:publicid:IDN+ch.geni.net:GEC20-Clemson+slice+demo expires on 2014-06-19 12:01:08 UTC
18:12:51 INFO    : Gathering resources reserved for slice demo.
18:12:51 WARNING : Requested API version 2, but AM https://clemson-clemson-control-1.clemson.edu:5001 uses version 3. Same aggregate talks API v2 at a different URL: https://clemson-clemson-control-1.clemson.edu:5002
18:12:51 DEBUG   : Checking cache for https://clemson-clemson-control-1.clemson.edu:5002
18:12:51 DEBUG   : Pulling GetVersion from cache
18:12:51 INFO    : Switching AM URL to match requested version
18:12:51 DEBUG   : Using new AM url https://clemson-clemson-control-1.clemson.edu:5002 but same API version 2
18:12:51 DEBUG   : Connecting to AM: unspecified_AM_URN at https://clemson-clemson-control-1.clemson.edu:5002
18:12:51 DEBUG   : Will request RSpecs only of type GENI and version 3
18:12:51 DEBUG   : Got 1 supported RSpec versions
18:12:51 DEBUG   : Found a matching supported type/ver: geni/3
18:12:51 DEBUG   : Doing listresources with 1 creds, options {'geni_rspec_version': {'version': '3', 'type': 'geni'}, 'geni_slice_urn': 'urn:publicid:IDN+ch.geni.net:GEC20-Clemson+slice+demo', 'geni_compressed': True}
18:12:52 DEBUG   : AM returned uncompressed RSpec when compressed was requested
18:12:52 DEBUG   : RSpec begins: '<rspec type="manifest" xmlns="http://www'
18:12:52 INFO    : Listed reserved resources on 1 out of 1 possible aggregates.
18:12:52 DEBUG   : Getting RSpec items for AM urn unspecified_AM_URN (https://clemson-clemson-control-1.clemson.edu:5002)
18:12:52 INFO    : <!-- Reserved resources for:
    Slice: demo
    at AM:
    URN: unspecified_AM_URN
    URL: https://clemson-clemson-control-1.clemson.edu:5002
 -->
18:12:52 INFO    : <rspec type="manifest" xmlns="http://www.geni.net/resources/rspec/3" xmlns:flack="http://www.protogeni.net/resources/rspec/ext/flack/1" xmlns:planetlab="http://www.planet-lab.org/resources/sfa/ext/planetlab/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.geni.net/resources/rspec/3         http://www.geni.net/resources/rspec/3/manifest.xsd http://hpn.east.isi.edu/rspec/ext/stitch/0.1/ http://hpn.east.isi.edu/rspec/ext/stitch/0.1/stitch-schema.xsd">
    <node client_id="vm" component_id="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+node+clemson-clemson-compute-2" component_manager_id="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am" sliver_id="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+sliver+vmc16f9f7a-5c85-4890-8088-30cfa1f0216e">
        <sliver_type name="m1.xlarge">
            <disk_image name="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+imageubuntu-12.04" os="Linux" version="12"/>
        </sliver_type>
        <interface client_id="vm:if0" mac_address="fa:16:3e:89:33:95">
            <ip address="10.0.185.1" netmask="255.255.255.0" type="ip"/>
        </interface>
        <interface client_id="vm:if1" mac_address="fa:16:3e:78:60:97">
            <ip address="10.3.10.100" netmask="255.255.255.0" type="ip"/>
        </interface>
        <host name="vm"/>
        <services>
            <login authentication="ssh-keys" hostname="130.127.88.101" port="3000" username="nriga"/>
        </services>
    </node>
    <link client_id="link0" component_manager_id="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am" sliver_id="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+sliver+link23613d79-7346-4ace-b675-6845d9cdb266" vlantag="4087">
        <component_manager name="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am"/>
        <interface_ref client_id="vm:if0"/>
    </link>
    <link client_id="link1" component_manager_id="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am" sliver_id="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+sliver+link49dc8eae-af94-4542-a1b4-56b5dde2dfeb" vlantag="760">
        <component_manager name="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am"/>
        <interface_ref client_id="vm:if1"/>
    </link>
    <stitching lastUpdateTime="20140501:11:22:10" xmlns="http://hpn.east.isi.edu/rspec/ext/stitch/0.1/">
        <path id="link0">
            <hop id="1">
                <link id="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+interface+procurve2_4087:3">
                    <trafficEngineeringMetric>2147483647</trafficEngineeringMetric>
                    <capacity>100000</capacity>
                    <switchingCapabilityDescriptor>
                        <switchingcapType>l2sc</switchingcapType>
                        <encodingType>ethernet</encodingType>
                        <switchingCapabilitySpecificInfo>
                            <switchingCapabilitySpecificInfo_L2sc>
                                <interfaceMTU>1500</interfaceMTU>
                                <vlanRangeAvailability>4087</vlanRangeAvailability>
                                <suggestedVLANRange>4087</suggestedVLANRange>
                                <vlanTranslation>false</vlanTranslation>
                            </switchingCapabilitySpecificInfo_L2sc>
                        </switchingCapabilitySpecificInfo>
                    </switchingCapabilityDescriptor>
                </link>
                <nextHop>null</nextHop>
            </hop>
        </path>
        <path id="link1">
            <hop id="1">
                <link id="urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+interface+procurve2_760:12">
                    <trafficEngineeringMetric>2147483647</trafficEngineeringMetric>
                    <capacity>100000</capacity>
                    <switchingCapabilityDescriptor>
                        <switchingcapType>l2sc</switchingcapType>
                        <encodingType>ethernet</encodingType>
                        <switchingCapabilitySpecificInfo>
                            <switchingCapabilitySpecificInfo_L2sc>
                                <interfaceMTU>1500</interfaceMTU>
                                <vlanRangeAvailability>760</vlanRangeAvailability>
                                <suggestedVLANRange>760</suggestedVLANRange>
                                <vlanTranslation>false</vlanTranslation>
                            </switchingCapabilitySpecificInfo_L2sc>
                        </switchingCapabilitySpecificInfo>
                    </switchingCapabilityDescriptor>
                </link>
                <nextHop>null</nextHop>
            </hop>
        </path>
    </stitching>
</rspec>

18:12:52 DEBUG   : RSpec had no expires attribute
18:12:52 DEBUG   : RSpec had no EG geni_sliver_info with an expiration_time attribute
18:12:52 DEBUG   : Got None sliver expiration from manifest
18:12:52 DEBUG   : rspecCtr 1
ERROR: No manifest found from listresources at https://clemson-clemson-control-1.clemson.edu:5001; review the logs.
No login information found!!

Trac comment by sedwards (github user: seledwards) on 06-12-2014 at 18:12

seledwards commented 9 years ago

Ok. It looks like the DEBUG messages at the end are a red herring:

18:12:52 DEBUG   : RSpec had no expires attribute
18:12:52 DEBUG   : RSpec had no EG geni_sliver_info with an expiration_time attribute
18:12:52 DEBUG   : Got None sliver expiration from manifest

These two messages are printed by readyToLogin:

ERROR: No manifest found from listresources at https://clemson-clemson-control-1.clemson.edu:5001; review the logs.
No login information found!!

In readyToLogin, getInfoFromSliceManifest() looks for the manifest using the AM url as the key and it doesn't know that it's changed and so fails to find the manifest.

Trac comment by sedwards (github user: seledwards) on 06-13-2014 at 06:17

seledwards commented 9 years ago

Ok. I'm suspicious this is a broader problem that has existed for awhile and am updating the summary to reflect that.

Trac comment by sedwards (github user: seledwards) on 06-13-2014 at 06:20

MarshallBrinn commented 9 years ago

Just to be clear, does this appear to be a problem with GRAM or with readyToLogin or omni some combination? (That is, should GRAM be returning something different?)

Trac comment by mbrinn (github user: MarshallBrinn) on 06-13-2014 at 07:08

innnkkki commented 9 years ago

I don't think this is GRAM because omni calls (createsliver, sliverstatus, etc) successfully switches URLs, there is something specific to readyToLogin

Trac comment by nriga (github user: innnkkki) on 06-13-2014 at 09:06

innnkkki commented 9 years ago

As I suspected the problem is with readyToLogin in function getInfoFromSliceManifest

Because multiple AMs might be called, readyToLogin looks for the amUrl it passed to omni in the output result. However because omni switched URLs (to satisfy the AM API version) the AM URLs don't match.

I added some prints in this function to verify this and this is what i got:

AMURL:https://clemson-clemson-control-1.clemson.edu:5001
AMURL keys returned from omni:
['https://clemson-clemson-control-1.clemson.edu:5002']

Somehow we need to find a solution to this, not sure what the best strategy is. We can revisit this after GEC.

Trac comment by nriga (github user: innnkkki) on 06-13-2014 at 09:18

innnkkki commented 9 years ago

We are already dealing with this case on the sliverstatus call so I have fixed this by reusing similar code. I will work on Sarah on how to best commit these changes.

Trac comment by nriga (github user: innnkkki) on 06-13-2014 at 09:33

seledwards commented 9 years ago

We also noticed that the status notice didn't make a lot of sense for GRAM (it said Unknown which is a default value).

I changed it to that if you have the geni_status field, that always gets printed.

Only if there is no geni_status will it fall back to print the am_status.

Tested with InstaGENI, ExoGENI, and GRAM and they all make sense.

Trac comment by sedwards (github user: seledwards) on 06-13-2014 at 19:01