[ENH] - Add ability to comment environment.yml

[kcpevey]

Feature description

I like to add comments to my envionment.yml to help with organization. For example, I might want groups packages for tests, documentation, ui apps, and core dependencies. I also like to comment individual lines with some explanation such as why this package is required, what part of the code needs it, or why it is pinned.

This is an example of the style and example comments I find helpful to use:

However, when I put this into conda store and create a new env, conda-store wipes all of my comments away and this is what I'm left with:

I'd like the ability to preserve my comments.

Value and/or benefit

People could add some much needed comments to their environment files on conda-store.

Anything else?

No response

[trallard]

I am guessing this is related to https://github.com/conda-incubator/conda-store-ui/issues/274 - since the # is not in the allowlist there might be some validation/sanitization happening to find and remove

[kcpevey]

I dont think its related to #274. That issue errors on invalid characters in the ui description which has to do with conda-store-ui validation. This issue doesn't error, it just looses information (# comments and extra whitespace) that are placed in the env spec.

I think its just a matter of roundtripping a yaml to include comments can be tricky. I don't know the backend, but I'd guess its using pyyaml which simply ignores these things, but we could switch to ruamel and keep the comments.

[trallard]

Ah I completely forgot we were running on pyyaml - I always forget also the differences between the two, so it would be good to have a look at how replacing one for another might impact the project overall.

[nkaretnikov]

I'll be working on this. First sub-task here is to identify the impact of making this switch.

One thing that was mentioned about pyyaml during weekly sync is that it might have better YAML support compared to ruamel.

[costrouc]

This issue is difficult due to the way that the environment.yaml is being stored currently as a json datastructure on the database.

Currently environment specifications are being stored in the database as a json structure. Additionally the specification in the schema is being stored as a dict.

In order to properly store comments we need to be storing the text and not the data representation. I agree that long term this needs to be done but I do think that this touches many aspects of the code and would impact the api. Additionally this issue would ensure that the order that users enter in text would be preserved (currently it isn't entirely).

Regarding ruamel vs. pyyaml. The usage of pyyaml by me in the past was honestly not understanding the difference between the two parser well. There are a long list of reasons we should be using ruamel: yaml 1.2 support, concrete syntax parsing, ability to insert comments and data in specific parts of the yaml. I think it is a no brainer to create an PR to transition to ruamel for everything.

At a minimum this PR will require a database migration.

Questions:

• should the api return the json represetation of the environment.yaml or the raw text? (this will change the rest api)

I think this PR is two lines of work:

• move exclusively to ruamel.yaml (easy)
• change backend representation of the specification as unicode text (hard)
• modifications to the api necessary to empower conda-store-ui etc.

[nkaretnikov]

My 2c:

• I have an (unreviewed) branch that ports code to ruamel while also adding tests: https://github.com/nkaretnikov/conda-store/tree/yaml-rt-272. Can submit this as a PR if folks think this is needed (e.g., new YAML standard support).
• conda-store works by taking user-supplied env, parses YAML, validates that it's a conda env, then saves certain pre-defined fields in the DB. When a user wants to edit a saved env, the process is reversed.
• The requirement to have data in the DB means that the representation needs to be serializable. Specifically, we currently store the conda specification using the JSON ORM field type.
• We cannot take advantage of ruamel's roundtripping feature because there's no easy built-in way to serialize parsed YAML with comments as JSON. ruamel is designed around roundtripping, not comment manipulation (there are some routines for that, but they are difficult to use). There's no built-in way to apply comments to a pure Python dictionary and get a commented YAML, for instance.
• ruamel internals: when you have comments, ruamel returns CommentedMap with the .ca field storing the comments. The .ca attribute ("comment attribute", I assume) uses Comment and CommentToken types, which are not serializable to JSON as is, because they use slots. It is possible to write your own serialization logic, but converting back to ruamel types, so that we could output YAML in the end, sounds tricky and hard to maintain, the ruamel types can change, etc.
• Note: ruamel will serialize CommentedMap to JSON by default, but it does so in a way that strips the comments, because it doesn't serialize the .ca structure.
• Alternatively, we could save a user-provided environment as a string in the DB. This won't require updating the YAML library that we use at all, but would require modifying the DB schema (to add a new field). In some DB backends, string fields also have limits, so that's something we might run into in the future. It also wastes space because we store duplicate data. With this idea, all our validation logic stays in place, it's just a new DB field. A pro: this is very user-friendly because the user will get the same YAML that they submitted initially if they request an edit.

For those wanting to follow the code with regard to YAML -> DB interaction. Here's an example of how env editing is implemented:

@router_ui.get("/environment/{namespace}/{environment_name}/edit/")
async def ui_edit_environment(
...
    # Get data from the DB
    environment = api.get_environment(db, namespace=namespace, name=environment_name)
...
    context = {
...
        # Finds matching specification and converts it back to YAML
        "specification": yaml.dump(environment.current_build.specification.spec),
...

class Specification(Base):
    """The specifiction for a given conda environment"""
...
    spec = Column(JSON, nullable=False)
...

Action items: next week, I'm planning to submit the following PRs, so we could discuss the details:

• My port to ruamel linked above (after I review it)
• Implement the "environment as a string in the DB" idea as it seems easier than writing manual (de)serialization.

UPD (5 Sep 2023): Tania and I agreed to have a follow-up discussion, starting from 11 Sep 2023.

[trallard]

@nkaretnikov - Please hold off on further PRs until we assess impact vs. effort.

[trallard]

This was too much effort vs gain so will close as not planned