Support 5GTANGO catalogues as storage backend

mpeuster commented 6 years ago

[x] configure catalogue endpoint with ENV variables using the service's docker file (use localhost as fallback, e.g., local dev env)
[x] upload NAPD.yaml (Attention: ETS-only packages would not work with this backend!)
[x] upload VNFDs
[x] upload NSDs
[x] upload TESTDs
[x] upload original package file
[x] upload other arbitrary files to catalogue (e.g. test binaries)
[x] upload additional catalogue support mapping (endpoint in catalog?, implement in _build_catalog_metadata)
- [x] fix integration bug (400) File with {name => upb_logo.png, uuid => 2dd10810-d608-45cb-8712-4491df20fbca} not found in the Catalogue
[x] return package UUID and send it in callback
[x] unified upload error handling
[x] unit tests for storage.py (currently skip_store=True in test_unit_rest.py, re-enable!)
[x] expose skip_store as CLI and ENV config
[x] test with all three example packages from tng-schema and misc/

Skip all other files in the first prototype.

ENV variable naming:

 CATALOGUE_URL: http://tng-cat:4011/catalogues/api/v2

mpeuster commented 6 years ago

Upload order:

napd
vnfds (if exists)
nsds (if exists)
testds (if exists)
(any other file)
package file (*.tgo) = the ZIP
additional file mapping to help catalogue

Format of 6.:

"pd" : {
        "vendor" : "eu.sonata-nfv.package",
        "version" : "0.3.1",
        "name" : "sonata-demo"
    },
    "nsds" : [ 
        {
            "vendor" : "eu.sonata-nfv.service-descriptor",
            "version" : "0.2.1",
            "name" : "sonata-demo"
        }
    ],
    "vnfds" : [ 
        {
            "vendor" : "eu.sonata-nfv",
            "version" : "0.3",
            "name" : "firewall-vnf"
        }, 
        {
            "vendor" : "eu.sonata-nfv",
            "version" : "0.2",
            "name" : "iperf-vnf"
        }, 
        {
            "vendor" : "eu.sonata-nfv",
            "version" : "0.2",
            "name" : "tcpdump-vnf"
        }
    ],
    "deps" : [],
    "son_package_uuid" : "78cdd0e2-e3d4-4789-afc7-56269c202b76"

mpeuster commented 6 years ago

Agreement in call on 2018-04-26:

v4.0: As described in the comment above. Mostly like SONATA.
v4.X: Have more generic file storage possibilities.

mpeuster commented 6 years ago

Example for the content of the callback which is send from the packager to the GK to inform it that the package contents are finally stored in the catalog.

{  
    "event_name":"onPackageChangeEvent",
    "package_id":"24c616cf-fe01-4c08-ae44-45d43ae67576",
    "package_location":"http://tng-cat:4011/catalogues/api/v2/tgo-packages/24c616cf-fe01-4c08-ae44-45d43ae67576",
    "package_metadata":<all_other_stuff_tng-sdk-package_knows_about_the_package_ignore_it>,
    "package_process_uuid":"d5cea225-033f-4fc6-816f-4a642461086a",
    "package_process_status": "success"
}

Important: package_id is the UUID returned by the catalog when uploading the *.tng file. NOT the UUID of the package descriptor.

Is this the planned behavior, or should I send the UUID of the package descriptor? @panstav1 and @jbonnet ??

jbonnet commented 6 years ago

Well, @mpeuster, I was counting with the UUID of the package descriptor (which would contain an extra field in its metadata, indicating the package file UUID). Otherwise... how can the descriptor UUID be found?

mpeuster commented 6 years ago

Thats the point and also my view. (see the mail that I am about to send out).

mpeuster commented 6 years ago

Issue is resolved. See mail.

panstav1 commented 6 years ago

After the WP3 discussion, below is the updated format of the additional file mapping to help catalogue. The additions are the list of testds trios + a field "files" which includes the file_id, returned from the Catalogue when the individual file is uploaded, and the correspondig filename. @mpeuster and @jbonnet what's your view on this?

{
    "pd" : {
            "vendor" : "eu.sonata-nfv.package",
            "version" : "0.3.1",
            "name" : "sonata-demo"
        },
    "nsds" : [ 
            {
                "vendor" : "eu.sonata-nfv.service-descriptor",
                "version" : "0.2.1",
                "name" : "sonata-demo"
            }
        ],
    "vnfds" : [ 
            {
                "vendor" : "eu.sonata-nfv",
                "version" : "0.3",
                "name" : "firewall-vnf"
            }, 
            {
                "vendor" : "eu.sonata-nfv",
                "version" : "0.2",
                "name" : "iperf-vnf"
            }, 
            {
                "vendor" : "eu.sonata-nfv",
                "version" : "0.2",
                "name" : "tcpdump-vnf"
            }
        ],
    "deps" : [],
    "testds" : [
                {
                    "name" : "simple-ttcn3-test",
                    "vendor" : "eu.5gtango.huawei.simple-test-descriptor",
                    "version" : "0.1"
                }
                ],
    "files" : [ 
                    { 
                        "file_uuid" : "71cdd0e1-d7h9-6641-kgm8-7969c202b76"
                        "file_name" : "Example.ext"
                    }
                    ],
    "tgo_package_uuid" : "78cdd0e2-e3d4-4789-afc7-56269c202b76"
}

mpeuster commented 6 years ago

Looks ok for me. Just give me the endpoint to send this data, once it is implemented.

Same goes for the endpoint to upload arbitrary files (Example.ext). Once the catalogues support it, I add it to the packager.

jbonnet commented 6 years ago

@panstav1 Just a minor-minor thing: file_name instead of filename? Or fileuuid instead of file_uuid :)

panstav1 commented 6 years ago

Updated to file_name :)

mpeuster commented 6 years ago

Ok, almost there. This is an example of what I will send to the mappings endpoint if the WP3 test package is unpacked (just to let you see a typical request):

{  
    "pd":{  
        "vendor":"eu.5gtango",
        "name":"simple-ttcn3-test",
        "version":"0.1"
    },
    "nsds":[  

    ],
    "vnfds":[  

    ],
    "testds":[  
        {  
            "vendor":"eu.5gtango.huawei.simple-test-descriptor",
            "name":"simple-ttcn3-test",
            "version":"0.1"
        }
    ],
    "deps":[  

    ],
    "files":[  
        {  
            "file_uuid":"83322dc5-1af5-462b-bb95-2c7718e8bff7",
            "file_name":"upb_logo.png"
        },
        {  
            "file_uuid":"53372453-acc5-4327-9c32-efb1736f907c",
            "file_name":"LICENSE"
        },
        {  
            "file_uuid":"c380458a-9721-4ac3-a1f5-e80f8043ba56",
            "file_name":"MyExample"
        },
        {  
            "file_uuid":"783a0641-d0ac-4042-8f72-10d129a7574c",
            "file_name":"MyExample.cfg"
        },
        {  
            "file_uuid":"32c0a521-35cc-44f2-8beb-827c72151db1",
            "file_name":"runner.sh"
        }
    ],
    "tgo_package_uuid":"123dafea-1060-4781-b86d-3e84e29c8d4c"
}

jbonnet commented 6 years ago

It starts looking great! 👍 @panstav1 We'll need an example of the response....

mpeuster commented 6 years ago

Ok, run in an error when uploading this mapping, the tng-cat returns:

File with {name => upb_logo.png, uuid => 7ea6cde0-63cd-4054-a002-b64eb8e0ea91} not found in the Catalogue

ecen though a call to curl -H "Content-type:application/x-yaml" http://localhost:4011/api/catalogues/v2/files gives:

- created_at: '2018-05-04T11:41:27.502+00:00'
  grid_fs_id: 5aec46e791521800010000e4
  grid_fs_name: upb_logo.png
  md5: 6e02d6a13e57d4dc114da4a24961f2ae
  signature:
  updated_at: '2018-05-04T11:41:27.502+00:00'
  username:
  uuid: 7ea6cde0-63cd-4054-a002-b64eb8e0ea91

The used mapping is:

{"pd": {"vendor": "eu.5gtango", "name": "simple-ttcn3-test", "version": "0.1"}, "nsds": [], "vnfds": [], "testds": [{"vendor": "eu.5gtango.huawei.simple-test-descriptor", "name": "simple-ttcn3-test", "version": "0.1"}], "deps": [], "files": [{"file_uuid": "7ea6cde0-63cd-4054-a002-b64eb8e0ea91", "file_name": "upb_logo.png"}, {"file_uuid": "0de595d2-af6b-4376-ad07-bce53bb64e1a", "file_name": "LICENSE"}, {"file_uuid": "7122c971-dac4-4f8a-8a0c-ac1828a05c21", "file_name": "MyExample"}, {"file_uuid": "53ead82d-824f-4670-8e11-2c483a163980", "file_name": "MyExample.cfg"}, {"file_uuid": "5023475b-1747-4446-b835-5a025a59c2bb", "file_name": "runner.sh"}], "tgo_package_uuid": "2990f855-18c0-4dc2-8455-f055274abf76"}

Any ideas?

mpeuster commented 6 years ago

Fixed in tng-cat now. This means tng-cat integration is almost done. Needs to be tested a bit more next week.

panstav1 commented 6 years ago

Firstly, the tng-cat checks the existence of the tgo_package which is mentioned in the mapping file. Then, The mechanism checks for the existence of all files. Now it is fixed. In case of every file being present in the Catalogue, the response is a 200 code with the message "Updated mappings of " In case of a file being absent, the tng-cat return a 400 code with a message implying the {name, vendor, version} trio (for descriptors) or {file_name,file_uuid} (for files). So, suppose a VNF descriptor is missing and being referenced in a mapping file, the return would be a 400 code with a message "VNF Descriptor with {name => , vendor => , version => } not found in the Catalogue". Also, for files, a 400 code with "File with {name => , uuid => } not found in the Catalogue"

sonata-nfv / tng-sdk-package

Support 5GTANGO catalogues as storage backend #14