Improve packages

[essen]

Some notes and ideas to improve packages:

• decide on how to name conflicting packages (dupe$(AUTHOR)$(PROJECT)? $(PROJECT)from$(AUTHOR)?)
• tags: https://wiki.debian.org/Debtags/TaggingGuidelines https://anonscm.debian.org/viewvc/debtags/vocabulary/trunk/debian-packages?view=markup
• have a maintainer and/or curator field listing who's responsible when your stuff breaks
• potentially useful fields: author, bug-reports link, license, mirrors
• mark broken packages as broken (possibly per platform)

[crownedgrouse]

hi Loïc , I gave you the URL for Debian tags. I'm very interrested in adding tags for better search of packages in index. I could maybe handle this. And if you are ok, I need your point of view on how you see this.

[essen]

The implementation is one thing, the logistics are another. I would like the logistics to be solved before any implementation is done. In other words: who is going to maintain these?

[crownedgrouse]

Logistics depends on implementation probably. It was the goal of my question. Should this be maintained by owner or by erlang.mk 'champions' ? I think some tags can be automatically set by content analyze , also. Make search is nice but almost useless with more than 400 untagged packages,  with bad description for many.

[crownedgrouse]

are you ok with some automatic tagging ? like source language(s) (for instance both erlang and c ) , I can probably list what tags can be set automatically. Moreover I'm working on a new feature on geas in order to use it on source files in alternative of beam files. Could be more usefull for the package status page, for example. It also will help to know what erlang versions can be used to compile the project.

[essen]

I'm OK with basically anything automated, in theory.

[crownedgrouse]

ok, will do some introspection, analyse and come back to you ASAP with a proposal.

[crownedgrouse]

note as far Debian moved to git, here is new reference URL : http://anonscm.debian.org/cgit/debtags/vocabulary.git/tree/debian-packages

[crownedgrouse]

my first draft : https://github.com/crownedgrouse/steam thanks to look at this when a little time ...

[essen]

What about a project that use Gun? It's still using an HTTP client, which isn't httpc. I don't think adding all clients and libs like you did for emysql and epgsql is a good idea. On the other hand looking at deps to figure out what they are and then setting tags appropriately should work.

[crownedgrouse]

It is only a draft. Needs more work obviously like adding gun and so on. If you are OK with the concept I can go on

[essen]

I am criticizing the concept here, not saying Gun is missing.

The problem with the current solution is that you need to add every library in order to detect things. Not very different from manually maintaining tags.

It would be great if you could instead look at dependencies, figure out what they are, and then use that information for the tags.

[crownedgrouse]

Yep, got it. I will think on this.

[crownedgrouse]

From mainly ranch's inheritance, I get those tags for gun

1> steam:tags("/home/eric/git/gun").  
{ok,['implemented-in::erlang','network::client',
     'network::server','protocol::ip','protocol::ssl',
     'protocol::tcp']}

No automatic clue that ranch does http from functions calls, neither gun. gun is only a client if I well understood, but ranch can be used both client and server. So inheritance give wrong assumptions.

So far, I'm a bit stuck. This would need to create a tag list on app/module per exported function and bring only those called by parent. Much more complicated. Any smarter idea ?

[essen]

Removing "network::server" would involve a much deeper analysis of the source code (client uses connect, server uses listen through start_listener). No better idea so far.

The problem with having everything in the source is that it is not much different than filling tags manually.

[crownedgrouse]

And in gun, module is a variable case Transport:connect(Host, Port, TransportOpts) of depending if it is ssl/tcp . I need to see what abstract code this creates. For now if both client/server is found, I will check if parent call a function connect or listen and choose one of both. If none it is better to remove both tags. If false positive, it is always possible to add some module/app in the code. I know, it is not a smart solution for you, neither me...

[crownedgrouse]

Two other thoughts.

• I will add possibility to project owner to add or overwrite tags from attribute in their code. This will then need no maintenance from me or you.
• Adding a module or application in steam code fix all projects depending on it. Not exactly the same thing than tagging manually all those projects...

[crownedgrouse]

my last result ... Ok , isn't it ?

1> steam:tags("/home/eric/git/cowboy").
{ok,['interface::daemon','network::client',
     'network::server','protocol::http','protocol::ip',
     'protocol::ssl','protocol::tcp','scope::suite',
     'web::server']}
2> steam:tags("/home/eric/git/ranch"). 
{ok,['network::client','network::server','protocol::ip',
     'protocol::ssl','protocol::tcp','scope::suite']}
3> steam:tags("/home/eric/git/gun"). 
{ok,['interface::daemon','network::client','protocol::http',
     'protocol::ip','protocol::ssl','protocol::tcp',
     'scope::suite']}

[essen]

Not sure what interface::daemon means but otherwise sure.

[crownedgrouse]

Daemon means it is an application running background on contrary to command line program. If it Otp application I set this tag, if it is escript I set command line. But wrong assumptions can exists, it is still experimental.

What next ? How do you see things ?

[essen]

Maybe there it would make more sense to deviate from Debian and instead point out when something is an OTP application, OTP library application, or escript, or...

[crownedgrouse]

I want to stay on same semantic. I do not want to invent other tags not actually existing but this semantic can be translated to other with a simple fun(). It is possible to add/replace otp::application when interface::daemon found. I need steam for a more global project related to debian, (well all my projects are related to debian). So erlang.mk can use its own semantic.

[essen]

Yeah but then you are missing interesting things like detecting erlang.mk plugins, the presence of C code and so on.

[crownedgrouse]

No language detection exists already with implemented-in facet, I added it after my post related to ninenines apps . I need however to detect plugin. Not done for now. To be complete I will write you apart with the proposal on debian/erlang.mk/otp tag transcoding. You will have a better sight then.

[crownedgrouse]

google drive invitation sent.

[essen]

Yep thanks, I will look a bit later this week.

[crownedgrouse]

To evaluate erlang-mk semantic proposal : steam:tags("/path/to/application/root/directory/",'erlang-mk') . . See you soon.

[crownedgrouse]

Just to say that I finished all erlang.mk semantic transcoding. Easy to change semantic if needed after your evaluation.

[crownedgrouse]

gentle reminder. Did you have a little time for this subject ?

[essen]

Working my way through PRs right now, but about to go cycling. Worst case I'll be traveling for 5 days which tend to make me work on things more.

[crownedgrouse]

No problem. I asked because having a little time this We to work on this if necessary. There is many PR and issues to prioritize. Waiting for news though.

[essen]

If you can send me a PDF of the notes you have by email I can take a look at it in the train tomorrow.

[crownedgrouse]

was offline this afternoon. just saw your post. Will try to merge some ideas.

[crownedgrouse]

done. Have a good trip.

[crownedgrouse]

nothing new on "Improving packages" ? Oops. "Improving dependencies index" ;>) ?

[essen]

Oh my bad, I was so busy working on the book and for the client that I didn't take a look and forgot. Keeping the tab open to process it tomorrow.

[essen]

Hm it wants me to have a Google account. Can you send me a PDF or something?

[crownedgrouse]

I sent you a PDF in a mail before you leave for your trip, see "done. Have a good trip." post upper. Will send you again.

[essen]

Thanks. I'm probably blind.

[essen]

Looks good to me. Just wondering why the single : instead of two :: like at the beginning. I'm not sure if anything is missing, I suppose that can be improved over time.

I suppose a next step would be to figure out how to automate their addition.

[crownedgrouse]

:: separate facet from tag. Single : separate tag from subtag. Example devel::lang:c.

[crownedgrouse]

Before automation, I wanted to know how to store those infos. I was thinking to a reverse index. A file with lines. Each line composed of a tag and all projects name linked to this tag. This way, easy with a grep and a cut and some sort -u to list projects.

[crownedgrouse]

As far I know you are compiling all projects as testing of Erlang.mk. Is it possible to use this to get tags of projects at same time ?

[essen]

I also want to have some automated fetching of tags from github and store them for comparison to know when projects update and semi automate increments.

[crownedgrouse]

Could you write me apart your specs and ideas in order I can propose something ?

[crownedgrouse]

Propose PR #549

[crownedgrouse]

You are probably busy, but can you check if you got my mails with a tags.index attached ?

[essen]

Three of them. Sorry about that, I took on too much work.

[crownedgrouse]

no worries , take your time. Just wanted to be sure you did not miss my mails again :>)

[crownedgrouse]

giving up tweeter does not give you more time ;>) ???

[essen]

It's a process, I've made progress but I'm not fully there yet. At least I relax better on evenings/week-ends now.