Additional meta informations in extension repositories

[MDevster]

As discussed in Contao Mumble on March 15, we first need to define how the metadata is stored in the repository.

Possible meta information

• Multilingualism (title, description, keywords)
• Images
• Links to the manual and further instructions
• explanatory videos (teaser, installation, setup)

The data should then be made available via the existing indexer for the Contao website and for the Contao manager. It was decided that the data should be in the repository.

First ideas from Call:

.contao/LANGUAGE/.. .meta-info/LANGUAGE/..

More about this topic at https://www.gondao.de/gedanken-zum-contao-app-store.html [german]

[MDevster]

Insert a json file for every language, minimum requirement is english.

Example file:

{
  "title": "example meta file for contao bundles",
  "bundle-name": "[vendor]/[package]",
  "title": "Example Titel",
  "description": "Example Description",
  "keywords": ["meta", "example", "keyword"],
  "images": [
    {
      "title": "Extension Screenshot (thumbnail)",
      "type": "string",
      "media": {
        "binaryEncoding": "base64",
        "type": "image/png",
        "url": "https://example.org/example-en.png"
      }
    },
    {
      "title": "Extension Screenshot (full)",
      "type": "string",
      "media": {
        "binaryEncoding": "base64",
        "type": "image/png",
        "url": "https://example.org/example-en.png"
      }
    }
  ],
  "links": [
    {
      "rel": "self",
      "href": "https://example.org/de/article"
    },
    {
      "rel": "author",
      "href": "https://example.org/de/user"
    },
    {
      "rel": "docs",
      "href": "https://example.org/de/docs"
    },
    {
      "rel": "issues",
      "href": "https://example.org/de/issues"
    },
    {
      "rel": "source",
      "href": "https://github.de/[vendor]/[package]"
    }
  ],
  "videos": [
    {
      "title": "Installation",
      "url": "https://example.org/de/video1"
    },
    {
      "title": "Einstellungen",
      "url": "https://example.org/de/video2"
    }
  ]
}

External media like images and vidoes must be cached by the service. Or may only be on trusted servers like Github, Gitlab and Co.

[leofeyer]

We should also link the available translations:

{
  "translations": [
    {
      "de": ".contao/messages.de.xlf",
      "en": ".contao/messages.en.xlf"
    }
  ]
}

Actually, I think we should store everything in the .contao folder and link to each file in the contao.json file. This way we only have to download the contao.json file instead of the whole .contao folder (which might not be possible at all without downloading the whole repo).

@aschempp Does this make sense?

[m-vo]

I'm not sure if meta data should overlap with the entries inside the composer.json, maybe this could be avoided by design. If the entries in composer.json need to be translated, maybe the translation source should point to them instead of duplicating them.

(Btw. @leofeyer I like the idea of having one file referencing all the parts and this is no vote against it :grinning: as the composer.json has to be processed anyway.)

[aschempp]

I think we should follow the existing composer specification as closely as possible. That means for example that the links section is called suppport and not links.

I also think we should define a standard set of features. If I compare this to the Apple Store out of my mind, it is possible to set one video and up to 5 pics. Not unlimited of everything.

Please also have a look at the standards in Transifex and other tool how they support translation of JSON files. We might need to follow their standards if we want to support them.

Next thought is that it would be easiest to just link to the individual language files in the root composer.json. Example:

{
    "extra": {
        "contao-manager-plugin": "\\Vendor\\Bundle\\ContaoManager\\Plugin",
        "contao-manager-localization": {
            "de": "https://raw.github.com/foo/bar/de.json"
        }
    }
}

This would allow us to get absolute URLs from the composer.json file without needing to download anything. And if the files are stored in a folder called .contao or foobar or in Git or somewhere else does not bother us, as long as we can download and index them.

[MDevster]

@aschempp

I also think we should define a standard set of features. If I compare this to the Apple Store out of my mind, it is possible to set one video and up to 5 pics. Not unlimited of everything.

I already have this task on 16.03. passed to the graphic artist. He create us a small document as a guide for the developers. Here we could then define maximum values.

Much like https://developer.apple.com/app-store/product-page/ or https://support.google.com/googleplay/android-developer/answer/1078870?hl=de

[MDevster]

First version for voting :)

see https://github.com/pdir/contao-example-bundle

assumption

• English is minimum requirement, therefor not extra linked in composer.json; If file does not exist, simple representation of the extension as before in manager with existing data from composer.json
• Individual localizations do not have to contain all the information, fallback to English
• Title, description, icon and advertising graphics are required fields to display localisation

[m-vo]

English is minimum requirement, therefor not extra linked in composer.json;

But then using the dir /.contao would be hardcoded magic + you'd have to check for the english language file every time even if it does not exist.

[MDevster]

@m-vo I assume that it always exists, therefore minimum requirement. https://raw.githubusercontent.com/pdir/contao-example-bundle/master/.contao/en.json So there is no magic ;)

[frontendschlampe]

It should not be called contao, but should be a generic, independent name like .metadata or similar, because maybe someone wants to adapt it. ;-)

English is minimum requirement, therefor not extra linked in composer.json;

I also think, that it's no problem to link it too. It's a better overview. (short look in composer.json > contao-manager-localization > oh, only "de" and "it" - no "en" ... )

[leofeyer]

I'm 👍 for the first version. And I agree that the folder probably shouldn't be called .contao.

However, I doubt that Transifex can handle the .contao/en.json file as it only supports JSON key-value. Here's an example of how OpenStreetMap uses JSON files to localize their application:

https://github.com/openstreetmap/iD/blob/master/dist/locales/en.json

[Toflar]

I think we should not put the English metadata in the composer.json directly. Once we put description text in there it just bloats the composer.json causing Composer to use even more RAM (yes, little but still, I want to get this right) because json_decode() loads everything into memory and Composer does not (yet) use a JSON streaming parser to read only what it needs. So I agree, English should be a must but it should also go into that extra folder, not directly into the composer.json file.

[aschempp]

@Toflar I don't think it was meant to put the english in there directly, but use the standard Composer values like description and tags if none are given in the translation files.

And I agree that the folder probably shouldn't be called .contao.

To me, it does not matter how the folder is named. The links are in the root json file, so they can point anywhere. I'm fine with any name, the documentation should state that it's irrelevant.

Regarding images and videos; I do not only think there should be an imposed limit, but I actually meant to name them. Instead of an array of images (with potential types) why don't we just use a key like image1, image2 etc? The image structure in your example file is also too complicated imho, why would one need to specify a type or a binary encoding?

[Toflar]

I would like to present another solution we have talked about amongst the Contao core developer team that has several advantages over the ideas that have been discussed here:

We plan to introduce a new repository named contao/package-metadata that contains all meta data to the packages. This would have a lot of advantages:

• As it is centralized, we have full control over it and can ensure quality.
• It's very easy to index because you don't need to fetch all the data from different domains. It is in one central repository and can be updated on a regular basis.
• We can add metadata to packages that are not Contao packages such as other PHP libraries like phpoffice/phpspreadsheet.
• We can easily migrate the data. If we start by e.g. defining that there must be a logo.svg and then in 5 months we think it should be named superlogo.svg or we want to allow more than one logo, we can migrate the whole repository easily. This means we don't have to think about BC or ensure it works for all the packages.
• We could develop bots that take over tedious work for that repository.
• We could automatically sync package data with e.g. Transifex for translations.
• Pull requests across packages are easily possible. Anyone can use the edit function of GitHub, it doesn't require any git knowledge.
• We can automatically mention responsible people by using something similar to the deprecated Facebook's mention bot so if someone is creating a PR that changes something in e.g. the terminal42 directory, it could automatically notify me.
• If one day we're able to have a real store, we can easily export and import that data.
• etc.

Here's how the structure could look like to start with:

• terminal42
  • notification_center
    • meta_de.yaml
    • meta_en.yaml
    • logo.svg
• madeyourday
  • contao-rocksolid-slider
    • meta_de.yaml
    • meta_en.yaml
    • logo.svg
• etc.

We think that merging on this repository could be delegated to a group of people that can handle git and ensure quality. Pull requests don't have to be merged immediately, we think it doesn't harm if a package is not translated within 2 or 3 days, as it will still be listed in the index as it is right now. At least, this would be something to start with and as outlined, can always get extended 😃

Please, leave your thumbs ups/downs and comments so we can decide and work on that.

[leofeyer]

I like the idea a lot. Especially the following is a huge advantage:

We can easily migrate the data. If we start by e.g. defining that there must be a logo.svg and then in 5 months we think it should be named superlogo.svg or we want to allow more than one logo, we can migrate the whole repository easily. This means we don't have to think about BC or ensure it works for all the packages.

A central repo similar to symfony/recipes will ensure that the existing data is updated every time it needs to be. And all data will be updated at the same time, unlike now that each developer decides for their own when to update (if at all).

[frontendschlampe]

I think the ides is from a technical point of view very useful, but there're some problems:

1. One reason that we switch from old ER2 to Composer was, that the developer don't have to edit information on more then one place. With the solution he have to do it again. All data of an extension should be on one place - it's easier for the developer and it's in his own responsibility to add data.

2. If a developer don't serve his extension on Github, because he use another platform to serve his extension, he have to change his workflow and must supply information in the Github repo.

3. If we have a central place for meta information we will be responsible for them. Not only for correctness of spelling but also for the content and also the legal and political correctness. So if there's any problem (e.g. copyright of images and text) we will be the first point of contact, because the developer just makes a PR, but we will accept the PR so we will be responsible for them.

4. If I have understood correctly, you want to make a repository, where the developer have to make pull request. So there should be someone who will check and merge the pr. I don't think, that we have the manpower to do it just in time. No developer or manager of the extension wants to wait hours or days until his information will be online. And it's not only merging the pr ... the responsible person must also take care of the content and the things which will be commited. There will be a lot of teeny PR, too - just to fix a typo ...

5. The idea will may not become a standard for all other OS-projects based on Composer because of the voluminously work, you have to do for merging and checking information.

As it is centralized, we have full control over it and can ensure quality.

Right ... we have full control over it, but do we want to have full control?

It's very easy to index because you don't need to fetch all the data from different domains. It is in one central repository and can be updated on a regular basis.

I'm not a developer, but if the mentioned structure is the same in the repository of the extension there's only the difference, that we have to load it it from the repo of the extension/developer. Isn't it?

We can easily migrate the data. If we start by e.g. defining that there must be a logo.svg and then in 5 months we think it should be named superlogo.svg or we want to allow more than one logo, we can migrate the whole repository easily. This means we don't have to think about BC or ensure it works for all the packages.

It should be written in meta_de.yaml what's the name of the logo, the images or other things. so it's not our problem how the logo is named.

We could develop bots that take over tedious work for that repository.

I bot can't check the correctness of the data (only the correctness of the structure) - see no. 3

Pull requests across packages are easily possible. Anyone can use the edit function of GitHub, it doesn't require any git knowledge.

Do a developer want, that somebody can change his information which he served to his extension without checking again? I think not. What is, if the information will be wrong, which somebody add to an extension? Just look of some Contao issues for changing wording or translation - you need some comments to win over leo to change it (just an example, no complaint, because it's the right way to keep the quality) - at the extensions anybody can change without any explanation.

We can automatically mention responsible people by using something similar to the deprecated Facebook's mention bot so if someone is creating a PR that changes something in e.g. the terminal42 directory, it could automatically notify me.

Yeah ... some more work for the developer ...

We think that merging on this repository could be delegated to a group of people that can handle git and ensure quality.

Suppose I will work in this group and e.g. I merge a picture which has copyright problems - how is my legal protection? First of all, I will have the problems ...

Pull requests don't have to be merged immediately, we think it doesn't harm if a package is not translated within 2 or 3 days, as it will still be listed in the index as it is right now.

You can't know, if it harm or not because you can't decide about it. It's the decision of the developer. And maybe it will be frustrating the developer if the served information come some days later as the extension. And when the information will be finally online he find a silly typo and he have to wait some days again? I think that's not the way ...

At least, this would be something to start with and as outlined, can always get extended

It's not the problem of the idea ... there was an idea 4 weeks ago which also could be something to start with and can always get extended. ;-)

Please don't get me wrong, but the above mentioned things are my reflections about the idea and I think, it's the wrong way. We shouldn't make us more work as we already have. In my opinion the information of an extension don't have to be split to various places - there's already an extension repo on Github, Bibucket, Gitlab or any other platform - we just have to use it. ;-)

[leofeyer]

One reason that we switch from old ER2 to Composer was, that the developer don't have to edit information on more then one place. With the solution he have to do it again. All data of an extension should be on one place - it's easier for the developer and it's in his own responsibility to add data.

You cannot compare this to the ER2, where you have to manually add every single release.

You will only have to register your package once at the metadata repo – just like you have to register your package once on packagist.org. And you will not even have to take care of localization, because we can register the central repo at transifex.com and handle it for everyone.

but do we want to have full control?

Yes, we do. See me comment above. 🙂

[aschempp]

One reason that we switch from old ER2 to Composer was, that the developer don't have to edit information on more then one place. With the solution he have to do it again. All data of an extension should be on one place - it's easier for the developer and it's in his own responsibility to add data.

That is somewhat correct, but still not really true. You don't want to maintain the Russian translation of your package in your repository. You don't want to release a new version just because the description was updated.

If a developer don't serve his extension on Github, because he use another platform to serve his extension, he have to change his workflow and must supply information in the Github repo.

Totally correct. But you also should use GitHub for Contao, and you also need to know Packagist, you need to lots of other services.

If we have a central place for meta information we will be responsible for them. Not only for correctness of spelling but also for the content and also the legal and political correctness. So if there's any problem (e.g. copyright of images and text) we will be the first point of contact, because the developer just makes a PR, but we will accept the PR so we will be responsible for them.

No we're not. Packagist is not responsible for the registered packages. Contao is not responsible for the translations that are updated on Transifex. That's just how Open Source works. The crows knowledge applies, if someone pushes an incorrect information, someone else can fix it.

If I have understood correctly, you want to make a repository, where the developer have to make pull request. So there should be someone who will check and merge the pr. I don't think, that we have the manpower to do it just in time. No developer or manager of the extension wants to wait hours or days until his information will be online. And it's not only merging the pr ... the responsible person must also take care of the content and the things which will be commited. There will be a lot of teeny PR, too - just to fix a typo ...

This is a problem we can solve with technology, if it really applies. Be aware that most packages will have one pull request and not change for a long period of time. How often did you update the description of your extensions in the extension repository?

The idea will may not become a standard for all other OS-projects based on Composer because of the voluminously work, you have to do for merging and checking information.

I think it's exactly the opposite. Regardless of the fact that Packagist has no plans to be localized, the directory could be used for every and all extension, regardless of if the author wants to participate. Be aware that extension authors do not want to maintain translations.

[aschempp]

I think there are to very important facts to consider:

1. Only a central repository will be able to provide metadata for non-Contao packages. This will be needed once we implement suggestions (e.g. terminal42/contao-leads suggests phpexcel/phpexcel).

2. We can always extend the feature to allow an individual developer to have the files in a non-central place. The central repository is just the first draft, and it will be 10 times easier to implement than a non-centralized version. Which is exactly important because of the manpower issues.

[Toflar]

This is now live in version 1.0.0 🎉 See https://github.com/contao/package-metadata.