sailfishos-chum / main

Documentation and issue tracker for the SailfishOS:Chum community repository
https://build.merproject.org/project/show/sailfishos:chum
MIT License
26 stars 4 forks source link

Setting up a "store" #29

Closed rinigus closed 9 months ago

rinigus commented 2 years ago

As integration with OpenRepos does not seem to be happening at any significant speed, we should consider backing up Chum repos with some kind of experience similar to app stores. This issue is made to discuss it and agree on steps forward.

Ideally, we need an app to interact with Chum on device and some kind of web presence. I'll try to write up what would such experience could be and later some implementation aspects. In principle, we should aim for something similar to Storeman and OpenRepos.

App:

Web:

Ideally same as for app, in part that makes sense:

@mentaljam can surely add to that list or comment on it.

If we start from the app and can provide all data for that, maybe it will be easier to roll out web interface later. In the current state, sailfishos-chum-gui already shows that it is possible to get list, some info on app, install and remove it.

Apps are missing icons, issues, and so on. In principle, we can add such data and provide to ChumGui via some files online. For example, each app can add at OBS some special file meta.yaml that would list app's repo, link to issues, links to discussion forums (multiple if needed), link to releases, link to icon. Unless we can specify in SPEC and make it available to ChumGui from PackageKit. Those package-specific meta.yaml datasets can be collected by a crawler and published at some github repo or in some S3 type bucket. This would allow ChumGui to refresh it as soon as needed from a single source. The crawler can just update it (if needed) once an hour, for example.

For GitHub and GitLab repos, we can also show number of stars (and forks), if some simple API is available. That would indicate the status of the app similar to currently used stars in OpenRepos. Advantage of guiding users to online issue trackers and discussion forums is a great experience for developers. In addition, it allows to merge users from different platforms for the apps that are multi-platform.

Thoughts, ideas?

cc: @mentaljam @piggz

rinigus commented 2 years ago

Some examples of what we can do using anonymous access to GitHub API:

I hope similar API is there for GitLab

mentaljam commented 2 years ago

As far as I understand, the difficulty is to host (pay for hosting) all the stuff. So, as for me, a github repo or some kind of another "free" place for storing data is a good idea. I don't know about OBS hooks but I can imaging an AWS lambda that gets called when the chum repo is updated. Such a lambda should fetch the repo metadata, parse it, generate some data for the "store" and push it into a special github repo.

As for the app features:

list available software in categories/subcategories search available software install software remove software

can be done with pk/libzypp

see recent updates

partially can be accomplished with pk/libzypp, but could be better implemented with the lambda/github approach

each software package could have icon, screenshots, issues and discussion forums, some stats.

can be implemented with the lambda/github approach

rinigus commented 2 years ago

As the involved data amounts (storage and transfer) is probably small, maybe https://www.scaleway.com/en/object-storage/ would do. We can either use S3 API directly or they have ability to setup website from S3 bucket.

It is also possible that we could get some kind of lambda support from Jolla/OBS directly. We should just ask @lbt about options.

rinigus commented 2 years ago

@mentaljam,I will look meanwhile on whether there are APIs for GitLab as well that we can use. I am really looking forward for a "Store" that will have native integration with GitLab/GitHub for issues, some app-specific discussions link. This would allow us to pull users into a central location and not scatter feedback all over the place.

At some point, I will ask you how I could help with your GUI. Hopefully, that background work on GitLab would be helpful as well.

mentaljam commented 2 years ago

At some point, I will ask you how I could help with your GUI

I think we need an exact plan of the backend architecture that will then drive the GUI development.

poetaster commented 2 years ago

I also think asking @lbt if obs can be used directly would be ideal. The object storage is there in any case. Since updating a chum repo pulls from a repository, it would be the single place to store and be one hook less. The webhook aproach on release as it works in obs via github/gitlab seems ideal to me. But that only gets you to obs, not chum per se.

rinigus commented 2 years ago

GitLab has its own API and looks we can get useful info regarding project via REST https://docs.gitlab.com/ee/api/api_resources.html. API call syntax is https://gitlab.com/api/v4/ . Project ID is something that can be found on the project page. As GitLab can be hosted elsewhere, URL should be possible to set for each project.

Examples using Whisperfish - https://gitlab.com/whisperfish/whisperfish with project ID 13453292

So, in this sense, interaction with users supplied issues can be reflected via Github and Gitlab directly. As a result, we don't really need to setup any website for that part.

mentaljam commented 2 years ago

It only confuses me a little that we force users to create accounts at GitHub and potentially some instances of GitLab to post issues. If we use some kind of API to create issues directly from the app then it could be annoying to devs if users start posting simple "comments" as issues. I mean if a user posts something like "Great app, thanks" not realizing that they creates an issue at GitHub 🙃

rinigus commented 2 years ago

To avoid confusion, I suggest:

While we can view issues/forks/releases in anonymous mode, posting issues would require login at GitHub/GitLab or its instances. Which may annoy some users, but then will help developers way less to have it centrally for their app. In principle, this should increase a chance of the bug being fixed as it is not forgotten complaint in Jolla Store. Having also bit of a threshold could help with the quality of the filed issues as well.

I don't think we need to implement internally filing the issues or new posts in forums, at least not from the start. This can be offloaded to the browser together with handling logins and so on. Unless it is dead-easy to plug it into the app.

rinigus commented 2 years ago

I would like resurrect this topic and proceed with it. As we have no feedback on what's possible from OBS side, let's implement metadata separately and rely on PackageKit or libzypp on processing RPM data.

At Chum, we have a mix of libraries and applications. For the purpose of the store, I propose that each package that would like to be exposed in Chum GUI, would have to define metadata file with the relevant information. The ones missing this info, will be not shown in GUI. For example, there is no point in exposing support libs to GUI as users don't really need to install it separately. Unless they develop something and then can do it via command line.

As this metadata file has to be written by a developer, I propose to use YAML for it. For Pure Maps, it would be

name: Pure Maps
package: harbour-pure-maps
category: Applications/Maps
repotype: github
repo: https://github.com/rinigus/pure-maps
icon: https://raw.githubusercontent.com/rinigus/pure-maps/master/data/pure-maps.svg
screenshots:
  - https://raw.githubusercontent.com/rinigus/pure-maps/master/screenshots/main.png
  - https://raw.githubusercontent.com/rinigus/pure-maps/master/screenshots/menu.png
  - https://raw.githubusercontent.com/rinigus/pure-maps/master/screenshots/traffic.png
forums:
  - https://github.com/rinigus/pure-maps/discussions
issues: https://github.com/rinigus/pure-maps/issues

This file defines icon, screenshots and relevant links (forum together with issues). Forum is kept in plural to allow support for FSO, TMO and other locations. Issues are usually in central location, hence kept as a single item. It is expected, that metadata file would define package key for which it is applicable. Category has to be defined from the given list and separate from SPEC, as in SPEC, at least according to Fedora, it is deprecated. I have also added repo together with repotype which would allow us to load releases, issues, stars by using GitHub or GitLab API.

In principle, one package (SPEC) can define multiple RPMs and we may wish to have support for it. In YAML, we can separate and have multiple documents using --- as a separating line.

After processing all packages metadata, we would end up with JSON which is easy to load on Chum GUI side. For example, as in

[
    {
        "name": "Pure Maps",
        "package": "harbour-pure-maps",
        "category": "Applications/Maps",
        "repotype": "github",
        "repo": "https://github.com/rinigus/pure-maps",
        "icon": "https://raw.githubusercontent.com/rinigus/pure-maps/master/data/pure-maps.svg",
        "screenshots": [
            "https://raw.githubusercontent.com/rinigus/pure-maps/master/screenshots/main.png",
            "https://raw.githubusercontent.com/rinigus/pure-maps/master/screenshots/menu.png",
            "https://raw.githubusercontent.com/rinigus/pure-maps/master/screenshots/traffic.png"
        ],
        "forums": [
            "https://github.com/rinigus/pure-maps/discussions"
        ],
        "issues": "https://github.com/rinigus/pure-maps/issues"
    }
]

There are several ways we can now generate and distribute these files. One option is to have them at OBS as one of the package sources (named chum.yaml or similar). Alternative, would be to make dedicated GitHub project and add those via PRs. Latter would allow to categorize and drop package keyword as well through use of subdirectories and corresponding metadata filenames. For example, GitHub project would have a file

packages/applications/maps/harbour-pure-maps.yaml

All packages would be combined into a single JSON and put into that repository allowing Chum GUI to pull the file from GitHub.

I would prefer to have metadata for each package and overall generated JSON in the same GitHub repo and avoid OBS+Github combination. It would allow us to just regenerate such store definition file on each import and not to hunt down changes in OBS. Changes in OBS can be tracked automatically, if needed, but it is not very simple and may require something checking it periodically.

Thoughts / comments?

poetaster commented 2 years ago

I like this approach. The only thing I'm not too keen about is aggregating in a central repository. I suspect keeping things in sync might get hairy. I was thinking a lazy man's solution would simply use the .spec's Description field. So, if that field contains, for instance subset:

Description    : 
    View maps, find places and routes, navigate with turn-by-turn instructions,
    search for nearby places by type and share your location.

    repo: https://github.com/rinigus/pure-maps
    icon: https://raw.githubusercontent.com/rinigus/pure-maps/master/data/pure-maps.svg
    screenshots: 
      - https://raw.githubusercontent.com/rinigus/pure-maps/master/screenshots/main.png
      - https://raw.githubusercontent.com/rinigus/pure-maps/master/screenshots/menu.png
    forums:
      - https://github.com/rinigus/pure-maps/discussions
    issues: https://github.com/rinigus/pure-maps/issues

So, stick to the yaml idea but first just in the spec? I'm not sure this is even feasible.

Obviously, your idea of making json available has the advantage of being simple to ingest and process in QML.

rinigus commented 2 years ago

If we add YAML into description field, there are few disadvantages:

Notice that this YAML doesn't have a release info. Which, in practice, means that we have to make aggregation once in a while.

Advantage of having it in a SPEC would be easier control by developers in a longer run. For example, change links, add screenshots and such. It would also not need any central aggregation as it seems to be easy to generate package model from just parsing description for additional info.

So, to summarize, maybe adding it to SPEC would be better in longer term.

poetaster commented 2 years ago

Ah, I hadn't thought of the first point. But I have a bunch of bug releases ;) Fair point.

Yeah, it did think it's a bit much if you also package for other distros. On the other hand, spec files permit conditionals. It does feel a bit spammy. On the other hand, it's quite descriptive :)

My thinking was, make a yaml stanza under description to start experimenting with what it might look like. Logic like sorting on Category and QML for displays can use the same field mappings even if later we see a need to move the data.

It does feel like a hack. I have to admit to liking hacks more than is healthy.

rinigus commented 2 years ago

From discussion at IRC:

So, I suggest to have then %description in the format


%description
This is my awesome app

This is continuation of regular app description

name: Awesome App
category: Applications/Misc
...

So, the last paragraph of description would be parsed for YAML. Probably the other paragraphs should be processed for word alignment as well, using regular approach.

I will look into writing %description parser, hopefully tonight

piggz commented 2 years ago

Here is some progress on this. JwSNShUbIfIiHcNDIQqpXAXY Current code is https://github.com/piggz/sailfishos-chum-gui/tree/metadata And an example .spec is https://github.com/piggz/harbour-advanced-camera/blob/metadata/rpm/harbour-advanced-camera.spec

piggz commented 2 years ago

I borrowed a few components from storeman, hope thats ok @mentaljam

mentaljam commented 2 years ago

Hi!

Sorry if missed, what's the benefit of using the description field of .spec? That a developer doesn't need to push some additional files to the Chum GitHub repo?

mentaljam commented 2 years ago

I borrowed a few components from storeman, hope thats ok @mentaljam

Which I've borrowed from the official store app😉

rinigus commented 2 years ago

Looks nice!

Re keeping it in SPEC: it will simplify updates of the fields for developers. Other benefits are:

@piggz: I really love the fact that we have pulley menu working again :) .

Note that for Github and Gitlab we can detect many fields automatically as they have corresponding API. See my comments from earlier discussion over here regarding it. So, for those, we can even skip adding issues link in SPEC. But maybe we should start filing separate issues under chum-gui regarding design specifics.

As for BEGINCHUMMETADATA, let's avoid it. Instead, let's parse the last paragraph for Chum data in description. Note that you can avoid package field - we don't need to match package to metadata anymore. For GitHub, repotype can be dropped as that we can deduce automatically.

What about the list of packages? Have you had a chance to work on those?

Maybe you could push the changes into a new branch of chum-gui?

I'll be able to join you tonight, I hope. So, let me know where I should start. May I take Github API parsing, for example?

rinigus commented 2 years ago

@piggz: Just saw that you convert YAML->JSON string and parse it in QML. As I'd like to add more integration with Github/Gitlab (such as showing current issues from there, release info for changelogs), it would make sense to parse it already in C++ and export data as properties. For example, "name" would be used to show package in the list and not the name of RPM. Similar exposure is expected for icon, number of stars in Github/Gitlab.

Thaodan commented 2 years ago

Would it make sense to use appstream as backend for metadata? Its already in a similar manager in gnome and kde. https://www.freedesktop.org/wiki/Distributions/AppStream/

rinigus commented 2 years ago

After some discussion, we ended by preferring adding metadata to SPEC description. See above for summary and pros/cons of that approach. In SPEC, we can, in principle have a link to appstream XML and load from there. However, I just remember how painful it was to fill it. See example of Pure Maps: https://github.com/rinigus/pure-maps/blob/master/packaging/pure-maps.appdata.xml . When you compare it with the examples above and the one used by @piggz, you could clearly see the difference. As we ask developers to fill it in, I would expect many would be happier to do it with SPEC extra than with appstream. Which would hopefully increase % of packages having such metadata.

Thaodan commented 2 years ago

I understand. Just asking did you try to generate the appstream metadata from rpm? See here: https://github.com/hughsie/appstream-glib#appstream-builder

Thaodan commented 2 years ago

Also appstream can use a yaml backend instead of xml.

Thaodan commented 2 years ago

https://docs.fedoraproject.org/en-US/packaging-guidelines/AppData/#_metainfo_xml_file_creation

When you compare it with the examples above and the one used by @piggz, you could clearly see the difference. As we ask developers to fill it in, I would expect many would be happier to do it with SPEC extra than with appstream.

If I compare the example here to one the Fedora the difference is mainly XML vs YAML format.

mentaljam commented 2 years ago

@Thaodan, does appstream-builder support additional tags in RPM .spec? I don't know much about AppStream but fast googling makes me think that developers have to write additional metadata files in any case. For example, I didn't find a way how appstream-builder can get screenshots from a .spec file (searched here). On the other hand, if YAML is supported, it seems to me that it doesn't matter where you put the additional metadata: in the .spec or in the appstream file.

But the main disadvantage is that we return to the model with some kind of crawler for building the appstream metadata for OBS repositories.

rinigus commented 2 years ago

I understand. Just asking did you try to generate the appstream metadata from rpm?

I don't remember doing it. Just note that we need data which is in addition to the one provided by RPM SPEC. So, while we can generate base XML (assuming that it works), it will have to be edited by developers further.

Also appstream can use a yaml backend instead of xml

I wasn't aware of it (https://www.freedesktop.org/software/appstream/docs/sect-AppStream-YAML.html#spec-dep11-example). In that example we have multiple apps, so only one section would have to be considered. I suspect that AppStream libs will fail to load if you don't provide some minimal info which, in part, maybe duplicating what is in SPEC already.

Now, we can support the both ways as well - instead of adding metadata in specified YAML spec, provide AppStream link. I just suspect, based on my experience of filling it, not many will be thrilled doing so. So, I'd prefer to have that little metadata section in SPEC and accept any PR coming our way that adds AppStream link support. :)

poetaster commented 2 years ago

I'd like to suggest that we simply consider using the YAML field naming conventions from appstream but stick to first using the spec files. I've been doing a bunch of communicating with other developers and think it would be great, for the start to keep the barriers low.

piggz commented 2 years ago

So, that would be some subset of https://www.freedesktop.org/software/appstream/docs/sect-AppStream-YAML.html#spec-dep11-fields ?

Thaodan commented 2 years ago

OBS also seem to support Appstream: https://en.opensuse.org/openSUSE:AppStore

rinigus commented 2 years ago

The devil is in the details. as @poetaster, I would keep the required info to minimal and as simple to enter as possible. Again, if OBS supports AppStream and there are developers wishing to use - sure, why not. Just someone would have to research the area and write the code. Use of AppStream parsers on partially filled data will most probably fail.

Note that when we have all data loaded from SPEC description field and all code working, it is possible to extend or convert to AppStream iff it will be found advantageous. At this stage, we have a bare-bones GUI that could really use some development. Taking into account the constraints, as discussed above, adding fields to SPEC description field looks to be the simplest way forward.

rinigus commented 2 years ago

@piggz: I guess we could use some subset. They probably made some research before formalizing it and we can rely on it.

piggz commented 2 years ago

I pushed an update which uses the last paragraph method.

I also went through the appstream yaml, and tweaked the format @rinigus proposed into this: name: Advanced Camera package: harbour-advanced-camera type: desktop-application developername: Adam Pigg categories:

Im still open to doing it either in the description, or, if OBS can generate the data easily, using that, but id like to see a demo of that. Perhaps phase 2?

poetaster commented 2 years ago

Ok. I was about to start implementing so it's good to know, uh, where. So, last stanza of Spec?

And a small nitpick: the field names in the appstream spec. are Name case. Should I use Name case?

And one last question, repotype is not really about type, that would be darcs, svn, git. So, we require git. Presumably we'll support gitea/gogs ?

rinigus commented 2 years ago

@piggz, looks good! Thanks for pushing it into local branch.

I was looking into GitHub GraphQL API and will base Github support on that. It is rather handy and allows us to fetch exactly the data that we need and probably via single call per package page, without rather verbose extras as we get in REST API. The sticky point is that we need to provide a token for accessing that API, even in read-only. However, I was thinking of creating a dedicated GitHub user and create a token for that. Similar API is available for GitLab. When I get to GitLab, we will see if we can support self-hosted instances - would depend on token requirements.

Looking at changed format, it seems to be fine. We don't need package field at all, as it is associated via SPEC already. We don't need repotype: github as that can be deduced from URL of repo. In case of GitHub (and probably GitLab), your example could be reduced to

name: Advanced Camera
type: desktop-application # not sure we need that
categories:
- Media
- Video
custom:
- repo: https://github.com/piggz/harbour-advanced-camera
icon: https://raw.githubusercontent.com/piggz/harbour-advanced-camera/master/harbour-advanced-camera.svg
screenshots:
- https://github.com/piggz/harbour-advanced-camera/raw/metadata/screenshots/screenshot1.png
- https://github.com/piggz/harbour-advanced-camera/raw/metadata/screenshots/screenshot2.png
- https://github.com/piggz/harbour-advanced-camera/raw/metadata/screenshots/screenshot3.png
url:
- donation: https://www.paypal.me/piggz

Developer Name (GitHub handle and probably full name) can be obtained from GitHub, homepage can be set in GitHub and could be either GitHub repo or any arbitrary page (see Pure Maps); issues have standard location; help can be set to discussions automatically, donation maybe as well, but I am not sure.

I would propose to load all data automatically from GitHub/GitLab and then merge it with provided YAML description taking into account that YAML-provided data overwrites automatic one. I am currently considering having an abstract package class with sub-classes for GitHub, GitLab, plain YAML implementations. That will allow to expand it later to AppStream as well. Will hopefully start on coding tonight...

poetaster commented 2 years ago

Which brings me back to my last question which I would now switch to a recommendation. Let's not use apis we don't NEED to use. I've probably mised something, but what is it we need to fetch from the repos if we already have all the meta data in the spec? In part I say this since I'm considering moving all my code to self-hosted :)

rinigus commented 2 years ago

And one last question, repotype is not really about type, that would be darcs, svn, git. So, we require git. Presumably we'll support gitea/gogs ?

it is more about API that allows to fetch info regarding a project.

As for need: it would allow to show Releases info and current issues. We can also show GitHub/GitLab forks, stars, and such. Whether it is an overkill, I don't know.

As for self-hosted: do you plan GitLab like? if so, this should work as long as you give us a token (not sure if it needed, will have to experiment first).

mentaljam commented 2 years ago

We don't need repotype: github as that can be deduced from URL of repo.

What about self-hosted GitLab instances?

rinigus commented 2 years ago

@mentaljam : self-hosted would need it, indeed. Assuming we can get queries working on those - may depend on token availability. If token is required, we would have to add that together with URL base to support each self-hosted GitLab separately.

Right now I don't know how many apps we have in self-hosted GitLab instances. For Chum, I suspect it is zero or small number - at least nothing jumps into my mind.

piggz commented 2 years ago

@rinigus fine to have a minimal yaml, but id maybe keep some optional fields (was going to add that later) so that eg, if help is provided on a sfos forum post, that can be linked instead of detecting eg github discussions url.

also agree with @poetaster to use CamelCase

rinigus commented 2 years ago

Ok. I was about to start implementing so it's good to know, uh, where. So, last stanza of Spec?

@poetaster, see https://github.com/piggz/harbour-advanced-camera/blob/metadata/rpm/harbour-advanced-camera.spec#L37

mentaljam commented 2 years ago

Then we may think about rather strict requirements for repositories: only git at GitHub or central GitLab. If someone prefer to use another VCS or another web-service for it they could use a separate GitHub/GitLab repo for storing packaging rules and metadata for Chum. In any case I don't think there would be much users who don't use GitHub/GitLab.

poetaster commented 2 years ago

And one last question, repotype is not really about type, that would be darcs, svn, git. So, we require git. Presumably we'll support gitea/gogs ?

it is more about API that allows to fetch info regarding a project.

As for need: it would allow to show Releases info and current issues. We can also show GitHub/GitLab forks, stars, and such. Whether it is an overkill, I don't know.

As for self-hosted: do you plan GitLab like? if so, this should work as long as you give us a token (not sure if it needed, will have to experiment first).

I run gogs and gitea instances. I still don't understand what data we need. Nothing against GraphQL ;)

poetaster commented 2 years ago

Ok. I was about to start implementing so it's good to know, uh, where. So, last stanza of Spec?

@poetaster, see https://github.com/piggz/harbour-advanced-camera/blob/metadata/rpm/harbour-advanced-camera.spec#L37

Ok, so still under Description with conditional with newline between. Thanks!

rinigus commented 2 years ago

@mentaljam : git / svn and so on don't really matter. Plan is to use API that GitHub and GitLab provide to enrich user experience. For packages/apps that cannot do that, we can always use metadata provided via SPEC %description (and/or AppStream). As such, not strict requirement - we can always fallback to data we have.

poetaster commented 2 years ago

@rinigus Ok. The need is not a need. But then, alternate app stores is kind of one of humanities big problems :).

poetaster commented 2 years ago

Ok. I've updated 2 thus far. Im going to push the releases on chum:testing . These: https://github.com/poetaster/harbour-beamdimension/blob/main/rpm/harbour-beamdimension.spec https://github.com/poetaster/harbour-concretemixer/blob/main/rpm/harbour-concretemixer.spec work?

rinigus commented 2 years ago

Please drop "Package" field, it is not needed at all. If we keep it in examples, it will later will be adopted by all... :(

poetaster commented 2 years ago

Yo. Sadly, these don't build. The specs are updated without the Package field.

rinigus commented 2 years ago

As discussed at IRC. tar_git misreads SPEC and picks up 'Name' in Chum description section. Something a proper RPM SPEC parsing does not (as rpmspec, for example). To avoid clashing with tar_git, we should use some other keyword instead of name. mal suggested PackageName