Add a way to convert traditional roles to collections

alikins commented 5 years ago

Feature Request

To ease transitional from traditional style roles to the newer collection format, it would be useful to be able to convert a traditional style role into a collection that contains that role.

Use Case

What problem does this feature solve? Please describe. A clear and concise description of what the problem is. Ex. I have an issue when [...]

At some point, mazer will stop installing traditional style roles directly and will only support collections. To make it easier for users to use mazer and ansible 2.8 with collections, and to continue using traditional style releases with older ansible releases, this feature would provide way to convert an on disk traditional role repository, to a collection style repository.

Proposed Solution

Describe the solution you'd like A clear and concise description of what you want to happen. Add any considered drawbacks.

mazer 'convert-role'[1], when pointed at a directory with a role in it:

check if a collection name and/or namespace is provided via the cli
Will load an existing meta/main.yml
- get 'version' info from it
- get dependency info from it
if needed, transform any fields to collection style fields (ie, 'tags', etc)
verify and validate that the synthesized collection metadata
? create a new output directory to avoid transforming in place
create a galaxy.yml in top level with that info
create a roles/ subdir
create a roles/${namespace}.${name} dir based on collection metadata
move any other top level dirs under roles/${namespace}.${name}/

For ex, transform a role like:

ntp
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── LICENSE
├── meta
│   └── main.yml
├── molecule
│   └── default
│       ├── molecule.yml
│       ├── playbook.yml
│       ├── tests
│       │   └── test_default.py
│       └── yaml-lint.yml
├── README.md
├── tasks
│   ├── clock-rhel-6.yml
│   └── main.yml
├── templates
│   ├── clock.j2
│   └── ntp.conf.j2
└── vars
    ├── Archlinux.yml
    ├── Debian.yml
    ├── FreeBSD.yml
    ├── RedHat.yml
    └── Suse.yml

to:


ntp
├── galaxy.yml
├── README.md
└── roles
    └── ntp
        ├── defaults
        │   └── main.yml
        ├── handlers
        │   └── main.yml
        ├── LICENSE
        ├── meta
        │   └── main.yml
        ├── README.md
        ├── tasks
        │   ├── clock-rhel-6.yml
        │   └── main.yml
        ├── templates
        │   ├── clock.j2
        │   └── ntp.conf.j2
        ├── tests
        │   ├── README.md
        │   └── test.yml
        └── vars
            ├── Archlinux.yml
            ├── Debian.yml
            ├── FreeBSD.yml
            ├── RedHat.yml
            └── Suse.yml

There is existing code to do most of that, as it is used as implementation details of how mazer install installs traditional style roles as collections currently (a59757e5cbab905f63ecc8a1d468b7963da918a5)

Main drawbacks/concerns

will need to handle invalid or unspecified collection data (transform it and warn, errors, etc)
For roles that exist in a git repo, it may be necessary to use 'git mv' etc instead of shell 'mv' or loading/dumping of files in new locations. 'git mv' etc may be required to insure the most git history
Transformations of data from yaml may need to be careful to not lose ordering or comments, something pyyaml is not very good at.
very likely many existing roles will not enough info available to create a valid galaxy.yml without the user providing more information
unclear yet what The Right Thing is to do for modules and plugins installed in a role. Can probably move them to collection correct locations if there are no name conflicts. But with the newest plugin loader branch, the plugin and module source code may need to be modified.

[1] not final name

Alternatives

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Instead of transform an on disk role directory to an on disk collection directory, 'mazer build' could support creating a collection artifact directly from a traditional role.

Implementation

Teachability, Documentation, Adoption, Migration Strategy

mazer migrate-role-to-collection --namespace some_namespace --name some_name some_role/ --output-dir ./some_collection/

alikins commented 5 years ago

ping @chouseknecht @thaumos

alikins commented 5 years ago

@chouseknecht @thaumos A couple of open questions about this feature:

The "correct" location for plugins and modules in a collection is different than traditional roles ie,

collection: mycollection/plugins/callback/mycallback.py trad role: myrole/callback_plugins/mycallback.py

Should migrate_role move these? For this particular case of creating a collection with one role, there aren't any naming conflicts to worry about, so moving the plugins is doable.

Downside is it means mazer has to continue to know the mapping between old style plugin dirs and the collection style (ie, 'filter_plugins' -> 'plugins/filter', 'library/' -> 'plugins/modules', etc)

Another downside is that some plugins may not work in a collection. I don't have a specific example of this case, but discussions in the past indicated that some plugins that can be included in roles currently will not work in collections.
Based on the new collection loader branch of ansible/ansible, it looks likely that existing plugins and modules will require source code changes to work via collections.

I am assuming that modifying the plugin/module source code is out of scope for this request. Safe assumption?
Would this be for migrating any traditional role or just galaxy style traditional roles?

For determining the version of the existing role, for galaxy style roles migrate_role would have to assume there is a git repo and try to extract a version from the git tags.

There is no version info for roles that are not in a dedicated git repo (ie, when a git repo has multiple roles in it) and has to be provided as a cli option for migrate_role.

For the galaxy style repo ('ansible-role-foo' with versions tagged) is it worthwhile to try to determine the version automatically? Or just require it to be specified.
Should migrate_role attempt to translate trad role license identifiers to collection spdx style ids?

ie, a traditional role may have it's meta/main.yml 'license' attribute be 'GPL', but that is not a valid license id for the stricter collections format (where 'GPL-3.0-or-later' or such is required).

thaumos commented 5 years ago

Responses to previous comment...

Migrating plugins within a role: If it can easily be done, then yes; let's do it! If it takes a lot of monkey work to patch that in then avoid it. Regarding the point about certain plugins not working, that is not the case anymore. Everything HAS to work.
Source code modification: We should not have any tooling do modification. I hope I understand you correctly.
Role type migration: I think Galaxy roles is for our MVP. If we can do non-Galaxy roles later, then cool. However, let's not block ourselves. For version, let's have them specify.
License determination: No, let's not try to interpret. Have them specify.

One point I want to make aside from this... Let's make sure we aren't marrying ourselves to mazer being the entry point to this. I want this ultimately to be a molecule thing later.

alikins commented 5 years ago

@thaumos "* Source code modification: We should not have any tooling do modification. I hope I understand you correctly."

To clarify, I do not want mazer migrate_role to modify a roles plugin or module source code. This branch does not modify role plugin or module source code. And we agree that mazer migraterole should not modify the role plugin or module source code, even if that means plugins and modules used via the new collection loader will not work without further modification_.

mazer migrate_role will not attempt to modify module or plugin source code

alikins commented 5 years ago

@thaumos

Role type migration: I think Galaxy roles is for our MVP. If we can do non-Galaxy roles later, then cool. However, let's not block ourselves. For version, let's have them specify.

Currently (d293ee1164fc550c76272359c7a5a3e1b0b53e9e) migrate_role does no "auto" detection of version, and the version of the new collection must be specified with '--version'. That includes galaxy style roles and any others.

Adding some git based "auto" detection of the role version based on git tags could be added later however.

alikins commented 5 years ago

@thaumos

License determination: No, let's not try to interpret. Have them specify.

Seems reasonable, I'll add a way to specify.

I'll start with behavior like:

if a license is specified via CLI, and that CLI provided license is valid spdx, use the CLI provided license for the new collection
if a license is specified via CLI, and that CLI provided license is not valid spdx, exit with error.

And assuming no difficulties, also:

if no license is specified via CLI, and the role's existing license is a valid spdx id, just use the roles license for the collection license.
if no license is specified via CLI, and the role's existing license is not valid spdx, then exit with an error pointing to --license

ansible / mazer