Add support for validation and deployment of tar compressed CSARs

[anzoman]

Description

The opera package CLI command can produce zip or tar compressed TOSCA CSAR as an output. Similarly, opera unpackage can unpack the zip or tar compressed CSAR to a specified location. However, right now opera allows validating and deploying only those CSARs that are in zip format (the same applies for deprecated opera init command). So, I believe that we can improve opera CLI by adding the support for passing tar compressed CSAR files to opera validate and opera deploy CLI commands.

Steps

Refactor validate.py, deploy.py (and init.py) from src/opera/commands so that they can receive tar compressed CSARs. The format of the CSAR can be detected automatically (use determine_archive_format helper from src/opera/utils.py) like it's done in opera package and opera unpackage.

Current behaviour

Opera cannot validate, initialize or deploy tar compressed TOSCA CSARs. Right now only zipped CSARs are accepted.

Expected results

To be able to pass TOSCA CSAR tarballs to opera validate, opera init and opera deploy.

[anzoman]

@miwurster do you happen to know if a TOSCA CSAR can be any type of archive or only a zip file? I remember that TOSCA v1.3 doesn't explicitly state that CSARs can't be in other compressed file formats but on the other hand I believe that only zip files are valid for TOSCA CSARs.

[miwurster]

Hey. In the following link they only talk about a csar zip file: https://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.3/os/TOSCA-Simple-Profile-YAML-v1.3-os.html#_Toc26969474

Afaik in version 1.0 only zip was mentioned. And due to the fact that they don't mention any difference I would at least assume that an 1.3 compliant orchestrator supports zip files. However, further compression archives could be supported too, but that's nice to have imho.

[anzoman]

This feature has been added in version 0.6.7.

[Furqan-Iqbal]

Hi, I want to know how I will build CSAR files in xOpera CLI. Secondly if i already have CSAR file let suppose name is test, so what will be the command to deploy it.

[cankarm]

Hi @Furqan-Iqbal ,

Interesting question - see in xOpera CLI Documentation, you can find a table with all supported commands:

I invite you to invest some time and play with them - probably you would like focus on:

• package
• unpackage
• deploy
• undeploy

Hopefully, you will find all required information there. For deploying test CSAR, the command would be:

opera deploy test

[cankarm]

I forgot one crucial part.

In the documentation you have also tabs for Description and Example in case that you have problems with understanding the Usage part. The tabs are right next to the Usage tab, like shown in the screenshot below:

[abitrolly]

However, further compression archives could be supported too, but that's nice to have imho.

More formats increases attack surface and makes pipelines more fragile. As if zip bombs were not enough, tar.gz compressed files (and tar.bz2, tar.xz, etc.) do not allow random read access, so you need to decompress the tar to just inspect the tar, which will make deployment both memory and time ineffective.

[anzoman]

@abitrolly many thanks for your opinion, I understand your concerns about security. By supporting tar.gz compressed files we just wanted to be more flexible for our users since TOSCA does not explicitly say how should the CSAR be compressed. With xOpera we also allow having directory-based CSARs, which are probably better from security point of view, but on the other hand I believe that it's not the orchestrator's task to check the vulnerabilities in CSARs. The users should be careful when preparing, downloading and using TOSCA CSARs and should validate their deployment files across numerous SAST and DAST tools to ensure security. We're currently building a tool to do exactly this and it's called IaC Scanner. This component together with the xOpera orchestrator and the Template Library TOSCA storage portal could be an important step towards providing a secure orchestration environment.

[abitrolly]

@anzoman I would rather bring this uncertainty about supported file formats to the folks developing the standard. What is the story behind being flexible wit tar? It is because tar and gzip are available in Linux by default and zip/unzip need to be specifically installed?

Security is a broad topic. Do you think CSAR files are trusted source and it should not be checked for .zip bombs? Can this IaC scanner already detect them? Is there a rationale for developing it with a list of alternatives considered. I thought I've seen such multi-linter runners before. For me more interesting is a tool to is security scan deployed infrastructure, rather than sources. Right now the only mechanism to detect such changes is Terraform state, but it only includes things that were explicitly setup.

[anzoman]

@abitrolly yes, I agree that the people behind the TOSCA standard should better define its structure and security provisioning. Supporting zip and tar.gz archives in xOpera is flexible not only because of the Linux you mentioned - it's just that some users use zip or gzip to compress their files and we could also support other formats as well, but we chose these two because they are the top two and also because other TOSCA tools use these two.

And no, I don't think of a TOSCA CSAR as being a trusted source - rather the opposite. These archives can contain a lot of security vulnerabilities and that's why I prefer different types of IaC scanning. So, you mentioned zip bombs and AFAIK xOpera currently does not provide any real protection against them, so we could add a check for compressed CSAR size and number of files to prevent these cases. Our IaC Scan Runner on the other hand already has a size limit. So, apart from source scanning I also see a good point in scanning deployed infrastructure (like with Terraform state you mentioned) - we thought about this some time ago when we wanted for xOpera to detect the state of the infrastructure and determine what is happening with the deployed services. We concluded that this is a big task to implement because there are so many infrastructures that services run on and they also change quite often, so the implementation of this would definitely take a lot of time and resources. On the other hand we focused more on a simpler solution by being able to deploy and configure monitoring services (such as Prometheus) with TOSCA and those could tell what is going on with the deployed infrastructure and report the changes back to the orchestrator, which would than do some necessary actions (e.g., redeploy stopped services). Moreover, we designed a new opera notify command for this and implemented a support for TOSCA policy triggers withing our xOpera SaaS orchestrator. I kindly invite you to take a look and express your thoughts on that.

[abitrolly]

With xOpera we also allow having directory-based CSARs

Supplying directory with unpacked CSAR doesn't work.

$ opera deploy SugarCRM-Interop-20130803-patched/
error: argument template: can't open 'SugarCRM-Interop-20130803-patched/': [Errno 21] Is a directory: 'SugarCRM-Interop-20130803-patched/'
usage: opera deploy [-h] [--instance-path INSTANCE_PATH] [--inputs INPUTS]
                    [--workers WORKERS] [--resume | --clean-state] [--force]
                    [--verbose]
                    [template]
...

xOpera to detect the state of the infrastructure and determine what is happening with the deployed services. We concluded that this is a big task to implement because there are so many infrastructures that services run on and they also change quite often, so the implementation of this would definitely take a lot of time and resources.

Yes. That's a job for a hardcore Computer Science department. Perhaps even running some public Software Carpentry competition with prizes to draft new solutions and properly define the problem space. Catching system diffs is what forensics is doing, but I haven't seen this integrated as a DevOps tools.

could tell what is going on with the deployed infrastructure and report the changes back to the orchestrator,

It would be interesting to see the changes myself first. :D

[cankarm]

Thank you for expressing the interest in this topic, @abitrolly.

However, I have issues to understand what exactly we are solving here. Can you please be more specific and maybe open an issue with a clear topic and more detailed information?

BTW: CSAR - Cloud Service Archive - means it should be compressed. Therefore the directory-based means that there are files and folders in the compressed package.

I also understand that you would like to know what is happening with the application after the deployment, but here is a bunch of monitoring tools, which, if defined properly, can resolve much of the issues that you arise. xOpera has no focus on this area and it is not here to compete with such tools.

Now I would invite us to stick with the appropriate topic.

[abitrolly]

Nevermind. I am just exploring the CSAR concept and the reasons behind them. Its specification is hard to read because of description made relative to other obsolete specifications. Instead of being just one single source of truth with history and changes. It is also not on the GitHub, so it is not clear where to raise these concerns.

For development it is more convenient to operate on uncompressed dir with a watcher enabled. But I am not a TOSCA developer. Just helping @Furqan-Iqbal to get on with it.