pcp-webjs not being shipped via bintray for ubuntu trusty (Bugzilla #1143)

[pcpemail]

On 2016-04-21 13:50:51 -0500, Zack Cerza wrote:

lsb_release -a

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.4 LTS Release: 14.04 Codename: trusty

apt-cache policy pcp

pcp: Installed: 3.11.1 Candidate: 3.11.1 Version table: *\ 3.11.1 0 500 https://dl.bintray.com/pcp/trusty/ trusty/main amd64 Packages 100 /var/lib/dpkg/status 3.11.0 0 500 https://dl.bintray.com/pcp/trusty/ trusty/main amd64 Packages 3.10.9 0 500 https://dl.bintray.com/pcp/trusty/ trusty/main amd64 Packages 3.10.8 0 500 https://dl.bintray.com/pcp/trusty/ trusty/main amd64 Packages 3.10.7 0 500 https://dl.bintray.com/pcp/trusty/ trusty/main amd64 Packages 3.10.6 0 500 https://dl.bintray.com/pcp/trusty/ trusty/main amd64 Packages 3.10.5 0 500 https://dl.bintray.com/pcp/trusty/ trusty/main amd64 Packages 3.8.12ubuntu1 0 500 http://archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

apt-cache search pcp-web

pcp-webapi - Performance Co-Pilot (PCP) web API service

On 2016-04-21 18:14:22 -0500, Nathan Scott wrote:

Hi Zack,

The pcp-webjs code is not part of PCP. Hence its not really appropriate to deb package it as part of the core PCP build (although there are options like we do for rpm, see below). You could open a github issue (that seems to be the only upstream bug tracker for pcp-webjs, AFAICT?) and request debs there ...

https://github.com/performancecopilot/pcp-webjs/issues

... but upstream projects rarely do packaging, so you might find no luck there either. Someone would have to sign up to do the deb packaging work (the person who maintains the PCP deb packages has expressed concerns about packaging pcp-webjs in its current state).

There seems to be no actual release model for pcp-webjs, no documentation, and it still contains a redundant, dated copy of Vector for some strange reason, in spite of numerous requests for its maintainer (fche) to stop duplicating that.

If those issues were resolved, we could consider producing debs via the mechanism Raphael describes here (this is like the rpm model we use today): https://raphaelhertzog.com/2010/09/07/how-to-use-multiple-upstream-tarballs-in-debian-source-packages/

Alternatively, Vector and webjs could just be deb packaged "properly", as the standalone projects they are, with their own bintray repos. Vector already has one setup, pcp-webjs could certainly have one too (that might also help to define a release process for pcp-webjs).

Ultimately though, either of these approaches is going to require someone to volunteer time & effort to make it happen - I'm swamped, such that even the deb packaging of pcp is falling behind (lacks systemd support, pmda sub-packages & proper dependencies, automatic pmda ./Remove on package removal, etc, etc... it is alot of work as-is - so my vote is for option # 2 at this stage).

On 2016-04-22 07:12:57 -0500, Frank Ch. Eigler wrote:

(the person who maintains the PCP deb packages has expressed concerns about packaging pcp-webjs in its current state).

Who is that person and what are the concerns?

There seems to be no actual release model for pcp-webjs, no documentation, and it still contains a redundant, dated copy of Vector for some strange reason,

The "strange reason" is that it is a tested release of vector instead of questionably tested random git snapshots that your changes to the pcp.spec et al. now bundle.

On 2016-04-22 13:36:02 -0500, Zack Cerza wrote:

Thanks for the response. I'm a little confused about this:

(In reply to comment # 1)

The pcp-webjs code is not part of PCP. Hence its not really appropriate to deb package it as part of the core PCP build (although there are options like we do for rpm, see below).

If it's "not appropriate" for pcp-webjs debs to be shipped with the rest of the pcp packages, why is that done on the rpm side?

There seems to be no actual release model for pcp-webjs, no documentation, and it still contains a redundant, dated copy of Vector for some strange reason, in spite of numerous requests for its maintainer (fche) to stop duplicating that.

Well, also, the Grafana version seems to be quite out-of-date as well.

On 2016-04-22 14:14:43 -0500, Frank Ch. Eigler wrote:

Well, also, the Grafana version seems to be quite out-of-date as well.

Unfortunately, grafana versions beyond this one no longer function as pure browser-side webapps, and instead have a server-side daemon running. This is in addition to the data source that pmwebd impersonates. If one can segment newer grafana into a pure webapp, I'd be glad to pull it into the webjs collection; it was not an easy job last time I tried it.

See also https://web.elastic.org/~fche/blog3/archive/2015/09/17/pcp-and-grafana-v2

On 2016-04-23 02:24:10 -0500, Nathan Scott wrote:

Hi Zack,

(In reply to comment # 3)

Thanks for the response. I'm a little confused about this: [...]

Yep, its very confusing. We use the RPM %source keyword to inject external Vector and pcp-webjs source tarballs into the PCP RPM builds (in the PCP git tree, see build/rpm/fedora.spec first ~10 lines).

The Debian packaging method we're using is not the "quilt" method described in that earlier URL (https://raphaelhertzog.com/2010/09/07/how-to-use-multiple-upstream-tarballs-in-debian-source-packages/) ... as a result, its not so simple to just inject external sources there as we do for RPM currently.

We also have a team of people paid to maintain the RPMs, whereas the Debian packages are maintained out of love, solo. As a result far less time ends up being invested there - unless more volunteers were to appear (anyone? Bueller?) which would be awesome & result in higher quality pcp debs.

However, and confusing things even further (!), it's not clear that doing this deb packaging refactoring (to use quilt format with external sources) is really the best option ... it's far simpler to package each source tarball separately, like nature intended. And this can be done now, independently. shrug - so, its complicated. Hopefully this clears some of the confusion, anyway.

(In reply to comment # 2)

[...] what are the concerns?

(there were 3 listed in earlier comments.)

The "strange reason" is that it is a tested release of vector instead of questionably tested random git snapshots that your changes to the pcp.spec et al. now bundle.

Hmm, more confusion here. The bintray packages are all ./Makepkgs based, which does not use Lukas' Fedora-rawhide-git-snapshotting process you're referring to.

We always use the latest upstream stable release of Vector for the bintray packages. So the "strange reason" is invalid - please remove the duplicate copy of Vector in pcp-webjs, and focus on helping to test Vector upstream before each release, such that everyone benefits.

I understand Martin is gearing up for a new Vector release within a week or so - I'm sure testing assistance would be appreciated right now, rather than afterwards this time.

And the PCP maintainers are also gearing up for a PCP release right now, so we'd love to see help on the QA front this week too. It'd be great to see that persistent pmmgr test failure Kens reported on for the last several releases tackled, for example - and any other QA efforts would be much appreciated too.

On 2016-04-29 16:55:10 -0500, Frank Ch. Eigler wrote:

[...] what are ["the person's"] concerns? (there were 3 listed in earlier comments.)

You were speaking of yourself in the third person. That is confusing.

There seems to be no actual release model for pcp-webjs,

A formal model had not been necessary, because there was so little development. But relief is in sight as we build more wrapper code around grafana, etc. Henceforth, the "release model" consists of a tag on the tree, wherefrom anyone can generate a tarball:

git archive --output=pcp-webjs-3.11.2.tar --prefix=/pcp-webjs/ --remote=git://sourceware.org/git/pcpfans.git pcp-webjs-3.11.2

no documentation,

It has exactly the same amount of documentation in the release tarballs as vector does: zero. It has considerable (at least as much as vector) documentation online, at the graphite/grafana upstream projects' web sites, where the webjs index.html points.

and it still contains a redundant, dated copy of Vector

That "dated" copy of vector is exactly the same version you just bundled in the debian pcp 3.11.2 bintray builds. Mere redundancy is harmless and your prior experience in removing it transfers directly.

So, is that all stands in the way of debian webjs subpackages a webjs analogy of your code in commit # 19ee10f609?

On 2016-05-03 02:54:05 -0500, Nathan Scott wrote:

(In reply to comment # 6)

Henceforth, the "release model" consists of a tag on the tree, wherefrom anyone can generate a tarball:

OK, that's a promising development.

The tag name chosen is a bit unfortunate - almost had downloadable source tarballs "for free" from github ...

https://github.com/performancecopilot/pcp-webjs/releases

... but because the tag name doesn't follow convention, this falls in a bit of a heap. The tarball is now "pcp-webjs-pcp-webjs-x.y.z.tar.gz" and inside it has the same incorrect path prefix.

I'm punting that it didn't follow convention because you wanted to duplicate an existing PCP release number instead of starting one for pcp-webjs (seems odd to me, but whatever floats your boat) - and the problem then became your way of forcing all this code to live in the pcpfans.git tree alongside pcp branches?

That tree already has a "3.11.2" tag from upstream pcp, so you had to go with "pcp-webjs-3.11.2" - is that right? Looks like you might be trying to shoe-horn too much into pcpfans.git, if that's indeed the case, and perhaps a pcp-webjs.git would simplify things on sourceware.

It has exactly the same amount of documentation in the release tarballs as vector does: zero.

There's several different documentation needs here. One specific kind of missing doc (that I've asked for in the past) is a description of rebuilding the components from their upstream sources.

It would also be good to have a top-level README that describes why the project exists, its goals, and so on too - that would explain to the casual observer how to setup and use these tools, and also give a developer a head start if they wanted to hack on it (esp. since alot of it is "compiled" javascript code that has git-commit'd AIUI).

Expect most people to find the project on github, so its worth doing things right there: https://github.com/performancecopilot/pcp-webjs ... suggests "Help people interested in this repository understand your project by adding a README.". (if you don't want a github repo anymore, given the tag-induced problems above, please just nuke it)

The docs should point to where the graphite and grafana code was forked from: https://github.com/grafana/grafana/releases ? https://github.com/graphite-project/graphite-web ?

So - which versions of the above, why those particular versions, what versions of Internet Explorer are known to not work, etc ... maybe a quick start guide to get up and running quickly like Vector and PCP have.

In the case of grafana, it'd be worth discussing why it requires that older upstream release / branch & why it doesn't work with current versions, what that patch is all about, and so on. (for potential developers)

and it still contains a redundant, dated copy of Vector

[...]. Mere redundancy is harmless

There are good reasons not to create duplicate copies of large amounts of code. It would be preferable to remove the extra shell snippets that we have had to add to deal with this (in the spec & makefiles), rather than propagate this further and into the deb build too.

cheers.

[natoscott]

pcp-webjs no longer exists - we now provide a modern Grafana web interface for PCP via grafana-pcp.