Enhancement request: Support for "well-known URIs"

[jpc4242]

Hi, Tal:

TOSCA 1.3 section 3.6.8.2.4 states:

URI: If the is a known namespace URI (identifier), such as a well-known URI defined by a TOSCA specification, then it SHOULD cause the corresponding Type definitions to be imported.

But it says such as, never says that the only possible case is if the URI is defined in a TOSCA specification.

Now let me put you in context. In some of our customers we need to parse ETSI NFV SOL001 descriptors (the SOL001 standard is based on TOSCA). All this descriptors import the ETSI-defined definitions, so for example:

tosca_definitions_version: tosca_simple_yaml_1_3
description: blah blah blah
imports:
  - https://forge.etsi.org/rep/nfv/SOL001/raw/v3.3.1/etsi_nfv_sol001_vnfd_types.yaml
 # and then the rest of the descriptor

(and, btw, this imported ETSI file imports itself in turn form another ETSI url)

The problems with that are:

• If customer systems are "air-gapped", with no access to internet for security reasons , then fetching the URL is impossible.
• MORE IMPORTANTLY: Some versions of the ETSI specifications have "small typos" that make them unusable. It is possible to make ETSI correct them (I've participated in some cases of that) but, specially for old versions, it is complicated (red tape)

Option 1 to overcome this is to ask all package vendors to fetch the ETSI specifications, fix them if needed, include the fixed files in the CSAR package, and replace the "https" imports with "file" imports

Option 2 is what we have implemented in our solution, (ab)using the TOSCA statement mentioned at the beginning of this message:

• Have preloaded on the system a set of files corresponding to the ETSI specifications, with typos fixed where needed
• Have a "mapping table" from well-known URLs to local filenames
• Whenever the solutions finds a URL in an import statement, lookup the URL up in the mapping table:
  • if found, use instead the file pointed to by the mapping table
  • if not, then proceed as usual (and get an error if you are on a air-gapped system)

Do you think it would be very difficult to implement a similar feature in puccini? The "mapping table" could be a JSON/yaml file whose name is passed as an argument. With my limited golang knowledge I found that phase1-read.go might be the right place for this. Inside goReadImports and between adding implicit imports and the loop where importspecs are handled, a new loop could replace on the fly every URL found in a mapping table with the absolute path name pointed to by the mapping table. I did some tests and confirmed that puccini already accepts absolute path names so this would not be a problem.

As an additional benefit, in a use case where a system has to process many packages and all of those packages import the same things from internet over an over, a user/admin could download the files once and add them to the mapping table. So even if you are not "air-gapped" this new feature would allow to save download bandwidth and speed processing.

BTW: our solution is not (yet) using puccini but we are seriously thinking about making the switch. The only blocking aspect we've found so far is what I am describing in this issue.

[tliron]

Thanks for the detailed description of the challenge. Actually, this is a problem that I have been aware of for a while.

I agree that the best solution is a mapping table. There might be an even better place for the code: the URL library in Kutil. This is the lowest level URL use for Puccini, so adding a mapping table there would actually affect all aspects of Puccini, not just imports: e.g. artifacts, metadata JavaScript imports, etc.

There has been a backlog of Puccini improvements/fixes that I'm working on now. Specifically a big branch that changes how data types are handled in order to support a somewhat obscure but important feature in nested map validations. Once that is done, I will work on this and a few other issues that have been waiting. I don't think it will be hard at all.

[tliron]

You can now map URLs like so:

puccini-tosca compile service.yaml -u fromurl=tourl

I would appreciate if you could build locally and see if it addresses your concern.

[jpc4242]

Hi Tal:

My team tested this and it is working. Thank you very much.

Closing the issue (and sorry about the delay)

[tliron]

You're welcome and thank you for the feature request!