Binary packaging for distros

obfuscurity commented 10 years ago

The project has been pretty slack in the past about making sure the binary packages work properly and are supported across the most popular versions. I don't think we need to go crazy, but it would be nice to set some goals, achieve them in master, and then backport the necessary bits for 0.9.x.

Here's a shopping list of the distro versions I'd like to see us support. If I'm crazy or there are dependency issues (i.e. Python or Django), let's talk about that here.

Fedora 20
CentOS 5
CentOS 6
CentOS 7
Ubuntu 14.04 LTS
Ubuntu 12.04 LTS
Debian 6
Debian 7

/cc @thorrsson @bmhatfield @robbkidd @mzupan @mckern @rwoolley, who have all volunteered to help with packaging or have worked on Graphite packaging in the past.

/cc @brutasse @SEJeff @bitprophet

P.S. This issue is currently only open in graphite-web, but obviously we'll need to consider carbon, whisper and ceres as well.

mckern commented 10 years ago

First read through, those don't look crazy. That list is really only supporting two platforms, with minor variations among the flavors of those platforms (init scripts, dependency package names, versions on older releases like EL5 and Ubuntu 12.04 LTS). We'd need to work out the dependencies, what's provided by the vendor already, what's provided but too old to use, and then how to resolve that. I haven't looked at any existing binary packages in about 12-18 months but I can do that this weekend. Any documentation about how they're built right now?

obfuscurity commented 10 years ago

I think the only bits (at least in graphite-web) currently involved in packaging are setup.py, setup.cfg, and then a postinstall script at distro/redhat/misc/postinstall.

obfuscurity commented 10 years ago

P.S. I think that @thorrsson has started to dig into the rpm stuff already, so please be sure to sync up before duplicating efforts.

deniszh commented 10 years ago

I can say that support 0.9.13 in Ubuntu 12.04 and Debian 6 will be quite hard. AFAIK Ubuntu 12.04 has only Django 1.3 - http://packages.ubuntu.com/precise/python-django and Debian 6 has Django 1.2 - https://packages.debian.org/squeeze/python-django and 0.9.13 requires Django 1.4 And AFAIR I faced with many other issues with python packages even on 0.9.10.

mzupan commented 10 years ago

Do we want to have the files live in the same directories across platform or stick to debian/ubuntu standards vs redhat standards. Sometimes they don't differ much but mainly configs they differ.

robbkidd commented 10 years ago

I'm using 0.9.12 on Ubuntu 12.04 by judicious application of hw-cookbooks/graphite. Would it be crazy to go down the path of an omnibus package install of graphite and all its dependencies? I think most of what the Heavy Water graphite cookbook installs by way of python packages could be done in an embedded python in an omnibus install.

deniszh commented 10 years ago

IMHO https://github.com/hw-cookbooks/graphite is good candidate for filling 'Install by Chef' docs section. Binary package implies installation of some amount deb or rpm packages and that's all. You can transfer these packages to own repo and install it even without Internet access - and it must work after that.

mckern commented 10 years ago

:+1: to not building omnibus packages if it can be avoided. It'll inflate packages considerably (where do you dry the line in the stack?) and it'll increase complexity. @deniszh Have you already started digging into deb packages?

josephholsten commented 10 years ago

I'd prefer to provide the debs via a launchpad ppa. 90% of the challenge there is just following debian packaging guidelines. I'm already acquainted with debhelper and dh-golang, and I think just grokking dh_python(1) should be enough to get us where we want to be.

For the debian stack, I recommend we focus on Ubuntu 14.04 first and try to get into Utopic (14.10). Then we can figure out a backporting strategy for Ubuntu 12.04, Debian 7 & Debian 6.

Do we want to support both apache and nginx webservers? mod_python? uwsgi? gunicorn? I've personally liked boring apache+mod_python, but I understand that's not the coolest tech.

For carbon, we'll probably want to support sysv-init, upstart & systemd. It'll almost certainly need tweaked ulimits . Otherwise they should be relatively boring.

obfuscurity commented 10 years ago

I would also prefer to avoid omnibus packages, as well as using external cookbooks as our build specification. Anything we release should be managed in-tree imho.

mckern commented 10 years ago

I think we should start w/ HTTPD + mod_python if it's the simplest supportable configuration; it doesn't need to be cool. It's easier to pick one target and increment than to try to shotgun them all. It'll also make reusing logic a little easier, so that even though we can't reuse post/pre scripts we can make sure that the packages do the same things on different platforms.

mckern commented 10 years ago

Oh, and (unfortunately) :+1: to supporting sysv-init, upstart, and systemd. It's a pain, but it's an upfront pain.

Quick status check for the names that have chimed in, who's working on what specifically already? Would like to avoid duplicating effort or competing implementations if we can suss out who will tackle what.

josephholsten commented 10 years ago

Oh, I'm working on debhelperizing for ubuntu 14.04. I'll be using apache httpd & mod_python.

obfuscurity commented 10 years ago

It sounds like @thorrsson and @mckern should sync up on rpm work and @josephholsten is taking lead on debs. Anyone else that wants to pitch in, please do, but touch base with those gentlemen first.

Thanks everyone! :sparkling_heart:

P.S. I would prefer that we target apache + mod_wsgi over mod_python unless someone has a good reason not to. Our own example vhost config uses mod_wsgi and our documentation covers mod_wsgi for virtualenv installs. We currently have no documentation (other than a passing mention) for mod_python; I have no real arguments against standardizing our packages on it, but we should also add an example config and documentation for mod_python if we wish to go down that route.

robbkidd commented 10 years ago

/me discards omnibus idea /me starts reading up on debhelper and dh-python

(Update: dh_python deprecated in 12.04+, apparently. Directs to using dh_python2)

mckern commented 10 years ago

HTTPD + mod_wsgi sounds like a plan. @thorrsson, how's things on your end?

josephholsten commented 10 years ago

got it, mod_wsgi it is.

josephholsten commented 10 years ago

I'm working on the debian packaging in #865. My debian-specific notes will be going there.

thorrsson commented 10 years ago

Hey folks, my $0.02 for the comments from today:

I have boxes initialized for testing on OEL6.5, Centos (6.5, 7), Fedora 20 and RHEL6.5 (I can probably initialize a EL 5 box fairly quickly too)
I don't think that we need to get into the omnibus game, the existing cookbooks are solid and fairly easy to wrap (as are the puppet modules)
mod_python should not be an option, there was a noticeable performance difference compared to mod_wsgi, I would stick to wsgi
I wouldn't mind including optional nginx/uwsgi configs as bonus points, though I am not sure if we should simply express a dependency on the webserver and provide example configs or actually manage the site configs and the webserver. Whenever I deployed graphite at $previous_jobs, The webserver was always managed by my Configuration Management As well, If we handle the webserver and the config we may need to account for shared server situations where folks don't run graphite exclusively on the box. I have also seen a ton of different existing apache configs (sites-* vs /conf vs hacking the httpd.conf etc) that we may not want to try and work around

Hi @mckern! This is where I am at as of last night: I am building on Centos 6.5 for the time being, though spent a good part of yesterday evening getting an environment that works (thanks to just replacing my laptop :) ) I have started with Carbon, based off of master as well as ensuring I can port back to 0.9.x I put in a PR last night that Jason merged to fix the install-lib location that was broken (it dumped carbon libs into ./webapp), and we will need to make a decision on https://github.com/graphite-project/carbon/issues/148 regarding managing the installation of the init scripts or not (I hacked this into my local fork in order to build on Centos, and determines if we need to worry about init script installation vs just creating the scripts)

I can produce an RPM that is targeted at python 2.7 and manages Carbon and the init scripts using bdist_rpm at this point. I also generated the .spec while I was at it and found that we needed to add the long_description (which was also merged today). I also tried out building with FPM, which had some quirks that I didn't bother working around. The artifact installs cleanly, with a dependency on whisper (which I built out really quickly just to get carbon installed) The services all start as expected as well.

On Sat, Aug 23, 2014 at 5:24 PM, Ryan McKern notifications@github.com wrote:

HTTPD + mod_wsgi wounds like a plan. @thorrsson https://github.com/thorrsson, how's things on your end?

— Reply to this email directly or view it on GitHub https://github.com/graphite-project/graphite-web/issues/864#issuecomment-53171866 .

deniszh commented 10 years ago

Currently I'm installing Graphite on my prod envs using deb packages built by FPM. Is it good idea or it's better to stick with standard Debian build system? +1 for Apache + mod_wsgi Also maybe we could use supervisord for starting daemons? It's pythonish and cross-platform. 24 авг. 2014 г. 1:14 пользователь "Ryan McKern" notifications@github.com написал:

[image: :+1:] to not building omnibus packages if it can be avoided. It'll inflate packages considerably (where do you dry the line in the stack?) and it'll increase complexity. @deniszh https://github.com/deniszh Have you already started digging into deb packages?

— Reply to this email directly or view it on GitHub https://github.com/graphite-project/graphite-web/issues/864#issuecomment-53169896 .

obfuscurity commented 10 years ago

It seems to me that having a real build spec for debs would be a good thing. But to be honest, I haven't followed the development of fpm closely to know how well a job it does at porting one format to another.

bmhatfield commented 10 years ago

As someone who has previously deployed Graphite via packages (rpms, then debs, now straight python packages installed by pip), I think this thread is headed in the right direction.

Some thoughts:

I have previously had success using "python to OS package" plugins to setup.py, bdist_rpm is built in, and bdist_deb is a short install away: https://pypi.python.org/pypi/stdeb
I think it's probably wise to try to "stick to the distro" in terms of init support. Don't switch or add supervisors on people (eg, no runit, no supervisord, bluepill, etc). This means likely writing a few init scripts. Word to the wise: carbon-cache and carbon-relay are not supported by upstart without 'hacks' such as forcing them to run in the foreground, which affects where logs are written. It might be worth doing some work on their daemonization approach in this regard.
I think it's also probably wise to "stick to the distro" in terms of Python support. I think this will be the most controversial, as I found that people sometimes want to virtualenv graphite, etc. This also means that we'd have to be very clear about which distros are supported (as @obfuscurity already laid out!). It would also indirectly prescribe which Python versions must be supported, which may or may not be something we want.
Carbon was the most difficult to package in my opinion, due to making sure the config files are sane. Currently Graphite generally assumes deployment into /opt/graphite - will we change that up to be more "FHS"-like? (You can make an argument that /opt is fine, but I think ya'll see where I'm going with that).
Graphite web is the second hardest. I see some discussion above about choosing to deploy under a webserver (eg, apache, nginx, whatever) or specifying a dependency. My current thinking is specify a chosen webserver as an optional dependency (eg, apache), and let the admin choose. Though this is quite different than where we are at today, it's probably wise to provide a python-native django app server if they don't choose anything, detectable by postinstall.
Speaking of Django and other 'probably not available' python libraries: If we're going to host this via ppa (or s3 http apt repo, which is also quite doable), I'd recommend (though not excitedly) either cloning or packaging each dependency of Graphite into it's own OS package hosted in the PPA as well.
Because the above is a ton of work: Is it worth building "dependencies included" packages of Graphite instead of separating out each Python library dependency? In the Debian world, that's a no-no, but like the way some people vendor Bottle when building a Bottle app, it might make sense to vendor all of the dependencies of graphite-web into the graphite-web package, rather than depending on them to be installed separately. It'll make the package larger (though not omnibus large), but it will dramatically simplify deployment and upgrades in the future. My gut instinct is that this is what people "probably want" anyways - especially if said dependencies only appear to Graphite-web/whatever.
Whisper is quite easy to package.
What will the support be for Ceres and such?

gwaldo commented 10 years ago

@deniszh The hw-cookbooks/graphite cookbook is what I intend on using as the recommendation for the 'install-via-chef' section of the docs, though this is heavily caveated.

First, is that last I checked a week or two ago, they are looking to cut a new release that is much more library/provider oriented than the current version.

Second, is that if we get the packaging more sane, that could simplify the cookbooks needed overall.

bmhatfield commented 10 years ago

So, now that I've dropped a bunch of random thoughts, let me approach this a different way.

Pre-caveat: I believe strongly in OS packages wherever applicable. I don't love the proliferation of every language gets it's own package manager.

So here goes: What problem are we trying to solve by packaging Graphite into OS packages? As is probably obvious from my above comment, I used to invest a lot of effort into packaging Graphite. But I also invested an equal amount of effort into making sure my configuration manager understood what the package was doing so it could "correct" it for my deployment (eg, move /opt/graphite/storage/ to a larger partition). Each thing the package did in a roughly 'standards' way, I had to go back and tweak to make things work better for my particular deployment. Every time Graphite needed to be upgraded, I was amazed at the amount of changes that went into both packaging and configuring it vs. the previous version.

Ultimately, I ended up in a place where I just mirror the pypi tarballs and install them using pip, then perform my configuration tasks as needed. Somehow, it ends up feeling simpler, though I agree it's a lot of work repeated community wide.

Anyways, I think it's worth being clear what goals we are trying to achieve by creating 'the way' of installing Graphite; to make sure that we actually achieve them. Packaging won't in and of itself make deploying Graphite meaningfully easier, given that much of the challenge of installing it comes after the bits are actually installed onto the machine.

EDIT: PS, to make sure this is super clear: I am pro-binary packages for Graphite. But it'd be a good idea to align on what problems we can solve, vs which we cannot. We can likely solve "make getting the bits on the machine" more like what Admins are used to w/r/t to 'sudo aptitude install graphite-web' or something, but probably not make it particularly easier to install Graphite.

obfuscurity commented 10 years ago

On Sun, Aug 24, 2014 at 09:23:17AM -0700, Brian Hatfield wrote:

So, now that I've dropped a bunch of random thoughts, let me approach this a different way.

Pre-caveat: I believe strongly in OS packages wherever applicable. I don't love the proliferation of every language gets it's own package manager.

So here goes: What problem are we trying to solve by packaging Graphite into OS packages? As is probably obvious from my above comment, I used to invest a lot of effort into packaging Graphite. But I also invested an equal amount of effort into making sure my configuration manager understood what the package was doing so it could "correct" it for my deployment (eg, move /opt/graphite/storage/ to a larger partition). Each thing the package did in a roughly 'standards' way, I had to go back and tweak to make things work better for my particular deployment. Every time Graphite needed to be upgraded, I was amazed at the amount of changes that went into both packaging and configuring it vs. the previous version.

IMHO we're trying to solve the problem of installing Graphite for users who are inexperienced with Django and/or Python package/environment tooling (pip, virtualenv, etc).

Ultimately, I ended up in a place where I just mirror the pypi tarballs and install them using pip, then perform my configuration tasks as needed. Somehow, it ends up feeling simpler, though I agree it's a lot of work repeated community wide.

Anyways, I think it's worth being clear what goals we are trying to achieve by creating 'the way' of installing Graphite; to make sure that we actually achieve them. Packaging won't in and of itself make deploying Graphite meaningfully easier, given that much of the challenge of installing it comes after the bits are actually installed onto the machine.

I don't expect anything beyond this, i.e. we should not lean on our packaging to handle complex configurations (either vertical or horizontal scaling).

Jason Dixon http://obfuscurity.com/ https://twitter.com/obfuscurity

brutasse commented 10 years ago

Graphite is popular enough that it's already provided in official distro packages (at least for debian and ubuntu). IMO, if we want to provide distro package files we have to work with debian / ubuntu / centos people to write packaging scripts they're happy with directly in our repos. This way distro packages and user-built packages will look the same and it'll be easer for people to switch from an official package to a custom one. So, can we get in touch whith Debian / Ubuntu / Centos developers to have them bless the results of what we're doing? Or look at what they've done in official repos and start from there? We need to get to a point where distro developers don't have to maintain packaging patches. Also let's hear what they say about supporting older OS versions than current stable/LTS… I'm not sure 12.04 / debian 6 is doable, might have to check.

(and if they're all in favor of dropping the /opt/graphite prefix, let's do it asap)

What I did for graphite-api is an omnibus package (fpm) that provides all dependencies, completely isolated from system packages and with an init script that starts a service with gunicorn. I left apache/nginx configuration instructions in the docs. Debian people seem to be interested in having it packaged in sid+backports, in which case I'll merge whatever pull request they make to get a debian/ directory they're happy with. (and they probably won't go the omnibus way :)

deniszh commented 10 years ago

@brutasse, only 14.04 and 14.10 ubuntu has current graphite packages. RHEL / CentOS / Fedora / Debian 6 / Debian 7 / Ubuntu 12.04 are not. And yep, in Ubuntu there's no /opt/graphite prefix, all packages are installing to appropriate (by distro) places. /etc/graphite/local_settings.py looks odd to me, but good from distro perspective.

mckern commented 10 years ago

@thorrsson Have you got this work in a branch somewhere? And have you got the .spec files you've created available anywhere?

thorrsson commented 10 years ago

I'll try to clean up what I have and push it up to my fork today

On Sunday, August 24, 2014, Ryan McKern notifications@github.com wrote:

@thorrsson https://github.com/thorrsson Have you got this work in a branch somewhere? And have you got the .spec files you've created available anywhere?

— Reply to this email directly or view it on GitHub https://github.com/graphite-project/graphite-web/issues/864#issuecomment-53211622 .

thorrsson commented 10 years ago

I pushed the .spec for carbon to: https://github.com/thorrsson/carbon/tree/rpm_spec It is working at this point based on master of my fork (which is right now the same as master of the upstream)

this spec was generated with : python2.7 setup.py bdist_rpm --packager "Tim Hunter tim@thunter.ca" --provides graphite --obsoletes 0.9.11 --spec-only

and the RPM can be built from the same command without the --spec-only :) This was an effort to make sure it could build 'as-is' so that I knew the starting point was sane

On Sun, Aug 24, 2014 at 3:56 PM, Tim Hunter tim@thunter.ca wrote:

I'll try to clean up what I have and push it up to my fork today

On Sunday, August 24, 2014, Ryan McKern notifications@github.com wrote:

@thorrsson https://github.com/thorrsson Have you got this work in a branch somewhere? And have you got the .spec files you've created available anywhere?

— Reply to this email directly or view it on GitHub https://github.com/graphite-project/graphite-web/issues/864#issuecomment-53211622 .

mckern commented 10 years ago

@thorrsson I've forked carbon to my namespace, and checked out your rpm_spec branch. I've got an updated carbon.spec that runs through mock on EL7. Here's a list of the files that this packages:

[root@redrobot build]# rpm -qp --filesbypkg ./RPMS/carbon-0.9.12-1.noarch.rpm 
carbon                    /usr/bin/carbon-aggregator.py
carbon                    /usr/bin/carbon-cache.py
carbon                    /usr/bin/carbon-client.py
carbon                    /usr/bin/carbon-relay.py
carbon                    /usr/bin/validate-storage-schemas.py
carbon                    /usr/lib64/carbon-0.9.12-py2.7.egg-info
carbon                    /usr/lib64/carbon/__init__.py
carbon                    /usr/lib64/carbon/__init__.pyc
carbon                    /usr/lib64/carbon/__init__.pyo
carbon                    /usr/lib64/carbon/aggregator/__init__.py
carbon                    /usr/lib64/carbon/aggregator/__init__.pyc
carbon                    /usr/lib64/carbon/aggregator/__init__.pyo
carbon                    /usr/lib64/carbon/aggregator/buffers.py
carbon                    /usr/lib64/carbon/aggregator/buffers.pyc
carbon                    /usr/lib64/carbon/aggregator/buffers.pyo
carbon                    /usr/lib64/carbon/aggregator/receiver.py
carbon                    /usr/lib64/carbon/aggregator/receiver.pyc
carbon                    /usr/lib64/carbon/aggregator/receiver.pyo
carbon                    /usr/lib64/carbon/aggregator/rules.py
carbon                    /usr/lib64/carbon/aggregator/rules.pyc
carbon                    /usr/lib64/carbon/aggregator/rules.pyo
carbon                    /usr/lib64/carbon/amqp0-8.xml
carbon                    /usr/lib64/carbon/amqp_listener.py
carbon                    /usr/lib64/carbon/amqp_listener.pyc
carbon                    /usr/lib64/carbon/amqp_listener.pyo
carbon                    /usr/lib64/carbon/amqp_publisher.py
carbon                    /usr/lib64/carbon/amqp_publisher.pyc
carbon                    /usr/lib64/carbon/amqp_publisher.pyo
carbon                    /usr/lib64/carbon/cache.py
carbon                    /usr/lib64/carbon/cache.pyc
carbon                    /usr/lib64/carbon/cache.pyo
carbon                    /usr/lib64/carbon/client.py
carbon                    /usr/lib64/carbon/client.pyc
carbon                    /usr/lib64/carbon/client.pyo
carbon                    /usr/lib64/carbon/conf.py
carbon                    /usr/lib64/carbon/conf.pyc
carbon                    /usr/lib64/carbon/conf.pyo
carbon                    /usr/lib64/carbon/events.py
carbon                    /usr/lib64/carbon/events.pyc
carbon                    /usr/lib64/carbon/events.pyo
carbon                    /usr/lib64/carbon/exceptions.py
carbon                    /usr/lib64/carbon/exceptions.pyc
carbon                    /usr/lib64/carbon/exceptions.pyo
carbon                    /usr/lib64/carbon/hashing.py
carbon                    /usr/lib64/carbon/hashing.pyc
carbon                    /usr/lib64/carbon/hashing.pyo
carbon                    /usr/lib64/carbon/instrumentation.py
carbon                    /usr/lib64/carbon/instrumentation.pyc
carbon                    /usr/lib64/carbon/instrumentation.pyo
carbon                    /usr/lib64/carbon/log.py
carbon                    /usr/lib64/carbon/log.pyc
carbon                    /usr/lib64/carbon/log.pyo
carbon                    /usr/lib64/carbon/management.py
carbon                    /usr/lib64/carbon/management.pyc
carbon                    /usr/lib64/carbon/management.pyo
carbon                    /usr/lib64/carbon/manhole.py
carbon                    /usr/lib64/carbon/manhole.pyc
carbon                    /usr/lib64/carbon/manhole.pyo
carbon                    /usr/lib64/carbon/protocols.py
carbon                    /usr/lib64/carbon/protocols.pyc
carbon                    /usr/lib64/carbon/protocols.pyo
carbon                    /usr/lib64/carbon/regexlist.py
carbon                    /usr/lib64/carbon/regexlist.pyc
carbon                    /usr/lib64/carbon/regexlist.pyo
carbon                    /usr/lib64/carbon/relayrules.py
carbon                    /usr/lib64/carbon/relayrules.pyc
carbon                    /usr/lib64/carbon/relayrules.pyo
carbon                    /usr/lib64/carbon/rewrite.py
carbon                    /usr/lib64/carbon/rewrite.pyc
carbon                    /usr/lib64/carbon/rewrite.pyo
carbon                    /usr/lib64/carbon/routers.py
carbon                    /usr/lib64/carbon/routers.pyc
carbon                    /usr/lib64/carbon/routers.pyo
carbon                    /usr/lib64/carbon/service.py
carbon                    /usr/lib64/carbon/service.pyc
carbon                    /usr/lib64/carbon/service.pyo
carbon                    /usr/lib64/carbon/state.py
carbon                    /usr/lib64/carbon/state.pyc
carbon                    /usr/lib64/carbon/state.pyo
carbon                    /usr/lib64/carbon/storage.py
carbon                    /usr/lib64/carbon/storage.pyc
carbon                    /usr/lib64/carbon/storage.pyo
carbon                    /usr/lib64/carbon/util.py
carbon                    /usr/lib64/carbon/util.pyc
carbon                    /usr/lib64/carbon/util.pyo
carbon                    /usr/lib64/carbon/writer.py
carbon                    /usr/lib64/carbon/writer.pyc
carbon                    /usr/lib64/carbon/writer.pyo
carbon                    /usr/lib64/twisted/plugins/carbon_aggregator_plugin.py
carbon                    /usr/lib64/twisted/plugins/carbon_aggregator_plugin.pyc
carbon                    /usr/lib64/twisted/plugins/carbon_aggregator_plugin.pyo
carbon                    /usr/lib64/twisted/plugins/carbon_cache_plugin.py
carbon                    /usr/lib64/twisted/plugins/carbon_cache_plugin.pyc
carbon                    /usr/lib64/twisted/plugins/carbon_cache_plugin.pyo
carbon                    /usr/lib64/twisted/plugins/carbon_relay_plugin.py
carbon                    /usr/lib64/twisted/plugins/carbon_relay_plugin.pyc
carbon                    /usr/lib64/twisted/plugins/carbon_relay_plugin.pyo
carbon                    /var/carbon/conf/aggregation-rules.conf.example
carbon                    /var/carbon/conf/blacklist.conf.example
carbon                    /var/carbon/conf/carbon.amqp.conf.example
carbon                    /var/carbon/conf/carbon.conf.example
carbon                    /var/carbon/conf/relay-rules.conf.example
carbon                    /var/carbon/conf/rewrite-rules.conf.example
carbon                    /var/carbon/conf/storage-aggregation.conf.example
carbon                    /var/carbon/conf/storage-schemas.conf.example
carbon                    /var/carbon/conf/whitelist.conf.example
carbon                    /var/carbon/storage/lists
carbon                    /var/carbon/storage/log
carbon                    /var/carbon/storage/rrd
carbon                    /var/carbon/storage/whisper

Check out the diff between the autogenerated spec and the one I've tweaked, and let's figure out:

How to ensure that all the scripts work correctly when prefix is set to /usr (spoiler alert: sed all up ins)
How to safely move configuration out of /var/carbon/conf and into /etc/carbon
How to get it working if we install this package

hggh commented 10 years ago

There already debian packages available: https://packages.debian.org/search?keywords=graphite-carbon https://packages.debian.org/search?keywords=graphite-web

They will be shipped with Jessie - if there is demand, I can create backports and upload it to backports.org

vincentbernat commented 10 years ago

Current distro packaging for Debian/Ubuntu can be found here: http://sources.debian.net/src/graphite-web/latest/debian/ http://sources.debian.net/src/graphite-carbon/latest/debian/

mckern commented 10 years ago

Here's something that Fedora is cooking up: http://pkgs.fedoraproject.org/cgit/graphite-web.git/tree/

thorrsson commented 10 years ago

Hey folks, I have had some personal life happen and am probably not going to have any spare cycles to work on this. Sorry to bail out, feel free to keep me on the thread and I can provide input and possibly review if time permits. Cheers Tim

On Tue, Aug 26, 2014 at 10:06 AM, Ryan McKern notifications@github.com wrote:

Here's something that Fedora is cooking up: http://pkgs.fedoraproject.org/cgit/graphite-web.git/tree/

— Reply to this email directly or view it on GitHub https://github.com/graphite-project/graphite-web/issues/864#issuecomment-53454293 .

mckern commented 10 years ago

Taking a look at the Fedora RPM spec files today, and experimenting to see if they're of use/benefit here. Will try building across all the requested RPM platforms, and will report back.

obfuscurity commented 10 years ago

:fire:

mckern commented 10 years ago

@obfuscurity thoughts on the use in Fedora of the python- prefix for carbon & whisper?

obfuscurity commented 10 years ago

@mckern I've never been fond of the python- prefixing. Is there a reason for it?

mckern commented 10 years ago

Usually denotes that something is specifically going to drop libraries into your python libpath 9r requires python to be of any use (much like the ruby- and rubygem- prefixes). Here's the relevant details from Fedora.

obfuscurity commented 10 years ago

Can we reconcile between Fedora's prefix requirements and a clean upgrade path for existing RPM users?

mckern commented 10 years ago

Where can I find some existing non-Fedora RPMs or spec files? Using metadata like Requires or Obsoletes, we could craft packages that simply replace existing packages as part of a known upgrade but I'd need to know what sort of metadata the existing packages provide.

Or am I misunderstanding, and you're proposing that we make it possible to upgrade from Fedora/EPEL graphite/carbon/whisper packages to ones provided first-hand by the Graphite project?

mckern commented 10 years ago

@obfuscurity Any followup to the outstanding questions ^^ ?

obfuscurity commented 10 years ago

@mckern Hmm, I don't know about the specs. I would have assumed that someone on this thread might know where to find them. @bmhatfield perhaps?

I would love to bring in the official distro maintainers and get their $0.02 on this effort. I worry now that this is seen as duplication of their work, but I'd still rather have an officially maintained package hooked into our build (and tests /cc @jssjr) than force users to wait on the upstream maintenance cycles.

mckern commented 10 years ago

I may have misunderstood; I was working from the assumption that there were already pre-existing but less-than-immaculate packages.

obfuscurity commented 9 years ago

Can anyone take a look at https://github.com/graphite-project/carbon/issues/355 with me?

damm commented 9 years ago

Any reason you can't Omnibus Graphite? there's only really a few targets and trying to address many versions will only complicate life.

SEJeff commented 9 years ago

Not a fan of omnibus, I'd rather be a good upstream and work with distributions to get our software included in their repos, and make it easy for our users to install on the various distros with somewhat reasonable native support. Omnibus encourages curl ...| sh, which makes me a very sad Keanu.

sebito91 commented 9 years ago

I'm actually a member of the fedora-admin community and have been touting Graphite and collectd for Fedora 22. We are currently working to add graphite to the existing metrics collection backend (currently using RRDTool) so I'm happy to help with that arm.

On Monday, January 12, 2015, Jeff Schroeder notifications@github.com wrote:

Not a fan of omnibus, I'd rather be a good upstream and work with distributions to get our software included in their repos, and make it easy for our users to install on the various distros with somewhat reasonable native support. Omnibus encourages curl ...| sh, which makes me a very sad Keanu.

— Reply to this email directly or view it on GitHub https://github.com/graphite-project/graphite-web/issues/864#issuecomment-69693204 .

obfuscurity commented 9 years ago

@sebito91 That would be great, and now's the time to pitch in. Per https://github.com/graphite-project/carbon/issues/355, it seems that our build tarball is missing some distro stuff. Admittedly, I built it on a Mac, but I'm surprised that would matter. We can't very well make multiple tarballs available, one per distro/platform, for a single release (for PyPI).

graphite-project / graphite-web

Binary packaging for distros #864