Debian 5.9.1 / 5.9.2 package build candidate

[zebity]

Hi Debian Packagers,

I am submitting this pull to help get ball moving on 5.9.1 debian / ubuntu build. I have tested this on Ubuntu 19.04 got to point of build full set of packages (core, utils, dev, dbg, source, libcollectdclient1, libcollectdclient-dev).

I have also installed core into machine and verified that runs ok.

Removed patches for liblvm2app deprecation & i2c dectectiom as these build via upstream.

I had issues with testing of configure result which then force build failure (debian/rules) so update to let build continue.

changeling is just brief summary. There are been quite a few upstream updates into 5.9.1.

Package build fails is you use tar ball. To build package successfully I had to:

1. clone collectd / collectd
2. git checkout (branch) collectd-5.9
3. build.sh , configure, make dist
4. copy generated compressed tar
5. git clone pkg-debian/collectd (my master with tag 5.9.1-1 added)
6. extract tar into pkg-debian: bzcat collectd-5.9.1.6.g1f778e9+.tar.bz2 | tar xvf - -C pkg-debian --strip-components=1
7. Run build: dpkg-buildpackage -us -uc

@mfournier & @tokkee , your review and feedback would be appreciated so we can make some progress on next collectd release.

Thanks

Zebity

[zebity]

Responding here to @mrunge recommendation to move to 5.9.2 (see https://github.com/collectd/pkg-debian/issues/17) due to defects in 5.9.1.

I have updated changelog to reflect 5.9.2 (and title of this pull request) build and added / push new 5.9.2-1 tag in my master and done git checkout to candidate collectd-5.9 branch.

All build oks (based on collectd-5.9 branch) if I do build when I have code in pkg-debian directory.

Fails when I try to build from ../collectd_5.9.2.orig.tar.gz generated from my collectd master (with updated version-gen.sh to reflect 5.9.2 target in collectd-5.9 branch), but this pkg-debian issue not a collectd one.

@bzed - lets see how we go towards getting new 5.9.2 package out. Any chance you could help with working out why package build fails when using ../collectd_5.9.2.orig.tar.gz based package build ? I looked at various forked pkg-debian trees, but they are all well behind master so not much help with detective work...

[zebity]

Hi Packagers,

I have done further testing and can confirm that collectd-5.9 branch is working on Ubuntu 19.04 I did debian pkg build from collect-5.9 branch using candidate 5.9.1/5.9.2 package build tree.

I have tested: cpu, df, disk, entropy, interface, irq, memory, process, retool, swap, virt (+ ExtraStats) and write_log.

[2019-09-24 23:17:11] plugin_load: plugin "cpu" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "df" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "disk" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "entropy" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "interface" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "irq" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "load" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "memory" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "processes" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "rrdtool" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "swap" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "users" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "write_log" successfully loaded. [2019-09-24 23:17:11] plugin_load: plugin "virt" successfully loaded. [2019-09-24 23:17:11] Systemd detected, trying to signal readiness. [2019-09-24 23:17:11] virt plugin: reader virt-0 initialized [2019-09-24 23:17:11] Initialization complete, entering read-loop. [2019-09-24 23:17:11] write_log values: blue.entropy.entropy 3780 1569331031^M

[2019-09-24 23:17:11] write_log values: blue.interface-eno4.if_dropped.rx 0 1569331031^M blue.interface-eno4.if_dropped.tx 0 1569331031^M

[2019-09-24 23:17:11] write_log values: blue.interface-eno4.if_packets.rx 2303995 1569331031^M blue.interface-eno4.if_packets.tx 765889 1569331031^M

[2019-09-24 23:17:11] write_log values: blue.interface-vnet26.if_errors.rx 0 1569331031^M ... ... blue.entropy.entropy 3780 1569331031^M

[2019-09-24 23:17:11] write_log values: blue.interface-eno4.if_dropped.rx 0 1569331031^M blue.interface-eno4.if_dropped.tx 0 1569331031^M

[2019-09-24 23:17:11] write_log values: blue.interface-eno4.if_packets.rx 2303995 1569331031^M blue.interface-eno4.if_packets.tx 765889 1569331031^M

[2019-09-24 23:17:11] write_log values: blue.interface-vnet26.if_errors.rx 0 1569331031^M

[2019-09-24 23:17:11] write_log values: blue_work04-kube-ubuntu-18_04_d947f4b7-fc40-4a88-badd-f6deb25f11ff.virt.domain_state.state 5 1569331031^M blue_work04-kube-ubuntu-18_04_d947f4b7-fc40-4a88-badd-f6deb25f11ff.virt.domain_state.reason 0 1569331031^M

[2019-09-24 23:17:11] write_log values: blue_master-od-macos-10_13_a42f4b94-bd9a-451a-a1a8-96d6aea1097a.virt.ps_cputime.user 6948230000000 1569331031^M blue_master-od-macos-10_13_a42f4b94-bd9a-451a-a1a8-96d6aea1097a.virt.ps_cputime.syst 4887520000000 1569331031^M

[2019-09-24 23:17:11] write_log values: blue_master-od-macos-10_13_a42f4b94-bd9a-451a-a1a8-96d6aea1097a.virt.virt_cpu_total 45467950000000 1569331031^M

[2019-09-24 23:17:11] write_log values: blue_master-od-macos-10_13_a42f4b94-bd9a-451a-a1a8-96d6aea1097a.virt.memory-total 17179869184 1569331031^M

[2019-09-24 23:17:11] write_log values: blue.processes.ps_state-blocked 1 1569331031^M

[2019-09-24 23:17:11] write_log values: blue.processes.fork_rate 358357 1569331031^M

[2019-09-24 23:17:11] write_log values: blue_master-od-macos-10_13_a42f4b94-bd9a-451a-a1a8-96d6aea1097a.virt.cpu_affinity-vcpu_0-cpu_0 1 1569331031^M

So clearly virt plug in work mostly by @rpv-tomsk has been pulled in ok and wrapper work has got ride of need to turn off W-all flag and rrdtool fix has been merged.

Not getting extra virt metadata, but need to look into my config for that I believe (problem is not with packaging).

@mrunge , now I know how to do do the packaging, just let me know when you would like to test 5.10 branch ;-)

Final comment: I am able to build with or without libstatgrab . I am sure I read somewhere that libstatgrab was only valuable for freebsd build, but cannot find where I read this...

[zebity]

@bzed @mrunge @rpv-tomsk

I have spent quite a bit of morning re-testing and checking debian build. In particular if problem were due to my failure to understand how to get package built or just the nature of the process.

After a bit more testing and reading I have concluded that the rather clunky process is just how it is at the moment unless we put some additional automation in place.

I have together summary on my blog page:

Here are my build steps:

mkdir build-collectd cd build-collectd/ git clone https://github.com/collectd/collectd.git cd collectd/ git checkout collectd-5.9.2 git tag -a collectd-5.9.2-1 -m "Add 5.9.2 tag" ./build.sh ./configure make dist cd .. ln -s collectd/collectd-5.9.2.1.tar.bz2 collectd_5.9.2.orig.tar.bz2 git clone https://github.com/zebity/pkg-debian.git bzcat collectd_5.9.2.orig.tar.bz2 | tar xvf - -C pkg-debian --strip-components=1 cd pkg-debian/ dpkg-buildpackage -us -uc

The fiddlely bit is the git describe (tag annotations) file and version naming approach, which can and likely will diverge if updates get out of sync as the collectd and pkg_debian are on different branches.

As part of collectd release process it would be helpful if some CI automation could be triggered to also do a debian build to we can see if it breaks... and hopefully automatically flowing into debian release through with main collectd release.

@mfournier & @tokkee , another gentle call to ask if you could please help with getting 5.9.2 flowing into debian / ubuntu stream as defects have been sorted and likely 5.10 will be following in the not to far future, so we need to get some process going to help with currency.

[zebity]

@mfournier @tokkee package rules update to add additional flags (CFLAGS += -Wno-error=address-of-packed-member -Wno-stringop-truncation) for GCC V9 (now part used in Ubuntu 19.10 distribution).

This addresses: https://github.com/collectd/collectd/issues/3341 for Ubuntu / Debian and ensure packaging is ready for next GCC version.

@octo @mrunge @sunkuranganath if any of you have personal contact with @tokkee (as per https://github.com/collectd/collectd/issues/3223 release thread discussion) could you see how we can get updates done to get Debian/Ubuntu updated beyond current ancient and now not working 5.8 package that keeps sitting unchanged while all around it is evolving. Any such efforts would be appreciated.

[bzed]

@zebity I'm preparing an upload at the moment. But I've taken all old versions and created a proper new repository here: https://salsa.debian.org/debian/pkg-collectd - the common place where packages in debian are being maintained.

I've merged your changes, added some fixes. The CI is building right now, as soon as it passes, I'll upload it.

[zebity]

@bzed that is great to hear 👂, if we manage to get this to flow through I will be breaking out the Champagne !

Thank you very much. BTW I am happy to help with this go avoid having all the work going to single set of hands.

If your build is based on 5.9.2 and you are including 19.10 Ubuntu target, then we will have to get pull requests (https://github.com/collectd/collectd/pull/3342) I put in for for sysctl related fix.

Likely do: 5.9.2 up to Ubuntu 19.04 & 5.10.0 for 19.10

Again thanks for persisting.

[bzed]

@zebity I don't care much about Ubuntu, but @cpaelzer might be interested :)

[bzed]

@zebity and regarding help - that would be appreciated. You could get a salsa.debian.org guest account and send pull requests there, I'm happy to merge.

[zebity]

@bzed thanks for pointer (and registration underway) ;-)

@cpaelzer fyi, I have been beating my head against wall trying to get updated Ubuntu package for collectd since I raised this defect: https://github.com/collectd/collectd/issues/3143 (in Apr 19) and then came across this: https://github.com/collectd/collectd/pull/3153

Now that it seems there is some headway on getting new debian package going could we please get this flowing into Ubuntu as collectd package is well and truely out of date with current Ubuntu release cycle.

Cheers,

Zebity

[zebity]

@bzed @cpaelzer not sure why but work on debian sides appears to have kicked off flow into Ubuntu stream: https://code.launchpad.net/ubuntu/+source/collectd So even if you don't care @bzed , good things appear to be happening ;-) @mrunge thanks for approval. If we can get 5.9.2 , then I expect 5.10.X with further fixes should be able to get going shortly.

Thanks again to all for help.

[cpaelzer]

Thanks @bzed / @mrunge for the upload of 5.9.2!

@zebity all code flows into the link you found it essentially imports all Ubuntu/Debian uploads into git. But it doesn't mean it will appear in Focal.

The Delta Ubuntu currently carries is all about FTBFS with newer toolchains.

See: https://git.launchpad.net/ubuntu/+source/collectd/diff/?id=007a913e936752e88691a1aa84c2c1a0a96be6c7&id2=6e490489627771808d18b0d2a6c5f5dea4c18fed And all FTBFS fixes in: https://git.launchpad.net/ubuntu/+source/collectd/tree/debian/patches?h=ubuntu/focal-devel

I'm giving this 5.9.2 a build try in Focal right now. If it builds fine we can make it a sync again asap. If it doesn't then I expect Debian to break as well sooner or later and we'd need either the fixes I mentioned or directly 5.10 in Debian to fix it for good.

[cpaelzer]

Build failed in focal, from the errors it seems at least https://git.launchpad.net/ubuntu/+source/collectd/diff/?id=007a913e936752e88691a1aa84c2c1a0a96be6c7&id2=6e490489627771808d18b0d2a6c5f5dea4c18fed will be needed.

That would be glibc 2.30 which Debian still is on 2.29 so you are ok for now.

[cpaelzer]

Heads up, going forward in glibc and compiler versions this is what you'll need (that works In Ubuntu/focal right now on top of this new upload here):

• CFLAGS += -Wno-error=address-of-packed-member -Wno-stringop-truncation
• CFLAGS += -Wno-error=address-of-packed-member -Wno-stringop-truncation -Wno-cpp -Wno-error=format-truncation

[cpaelzer]

Final FYI, then I'll stay silent :-) I started a merge of this into Ubuntu here

[zebity]

@bzed @cpaelzer @mrunge

I have done updates to align this pull request with work done by @bzed over on debian repository: https://salsa.debian.org/debian/pkg-collectd This includes: adding mqtt (mosquitto), moving to python3, removing my configure hack (need to disable gpu plugin to avoid configure failure), NEWS rather than NEWS.debian.

I have built on Ubuntu 19.10 with GCC 9.2.1. using collectd 5.9.2 with additional sysctl/sysctlbyname update: https://github.com/collectd/collectd/pull/3342

At this point I do not see any further value in working on debian/ubuntu packaging from here and will do any updates via debian tree.

Many thanks to @bzed & @cpaelzer for getting updates flowing through to debian and ubuntu trees.