search

matomo-org / matomo-package

Matomo release script (official package), and debian/ubuntu package (allows sysadmins to deploy Matomo within seconds using "apt-get install piwik -V")
28 stars 27 forks source link

Is the Matomo Debian package dead? #131

Open nomandera opened 3 years ago

nomandera commented 3 years ago

I absolutely hate posts like this because FOSS should not be about timescales and deliverables BUT this is also the official apt repository for a commercial product listed to this day as a supported method of installation.

I dont want to labour the point but some details are listed here

To summarise:

I apologise for making this post but we are closing in on a year of missed updates and I am now at a cross roads. The effort to port from apt to manual install will be huge and I both dont want to do it or tell people how much its going to take in man hours.

Please can we have a roadmap.

Findus23 commented 3 years ago

I just updated the title to make it clear this is about the debian packages as this repository also contains build-package.sh which is the main way all Matomo releases are created and is updated for this all the time.

nomandera commented 3 years ago

Yes thank you that is a worthy and important clarification to make.

mattab commented 3 years ago

Thanks @nomandera for your interest. I've talked to the previous maintainer and I can confirm that he is not going to maintain the Debian package anymore.

So at this point we have two options:

Is there maybe anyone reading this who would be interested to take over the debian package for Matomo?

we do have quite solid docs and scripts that @aureq created, but it does need some work and knowledge of Debian.

Dreamsorcerer commented 3 years ago

Not sure it makes sense to close all the other issues. If there is a new maintainer, then many of those issues probably need to be reopened.

mattab commented 3 years ago

@Dreamsorcerer Most issues seem really low impact, and hadn't been commented/touched for years. Trying to be pragmatic and have a clean slate for whoever comes next (if we're lucky to find a new maintainer...)

Dreamsorcerer commented 3 years ago

No problem, I guess just a "reopen if this still affects you" then. I do see that some of them appear to have been fixed, although probably need some documentation.

nomandera commented 3 years ago

Is there any option for Team Matomo to step in and take over even if it is a temporary measure?

During my initial research of install options the official announcement and apt repository left no doubt that the project was backed by Team Matomo.

image

Sorry if this seems negative, just feeling a bit left out in the rain here given that the very first time we need the Matomo team to maintain the Debian Matomo repository the response it to twilight the project and call for the community to step in or it will be closed.

mattab commented 3 years ago

@nomandera thanks for the note! the blog post now redirects to the official non-debian installation guide, to remove any possibly confusion.

we've also redirected https://debian.matomo.org/ for now, to make sure nobody else could get false expectations.

FYI Looking at some basic server logs stats, we can see that Since Jan 1st 2020 there were only 244 downloads of the release matomo_3.10.0-1_all.deb. So it looks to me like Debian package is not a very popular installation method.

Big thanks and kuddos to @aureq for his work all these years maintaining the package! :clap:

BenSturmfels commented 3 years ago

@nomandera thanks for the note! the blog post now redirects to the official non-debian installation guide, to remove any possibly confusion.

@mattab just wanted to note that I found the redirection from the Debian repo to the installation guide quite confusing! Would it be possible to redirect to a paragraph explaining that the Debian repository is no longer supported? I don't think the redirection alone communicates this effectively - I very nearly opened an issue about the redirection heh.

Thanks for all your work on Matomo - I've been a happy user for many years! Thanks @aureq too!

BenSturmfels commented 3 years ago

PS. This is still live in the docs:

How do I install Matomo on Debian GNU/Linux servers? https://matomo.org/faq/how-to-install/#faq_17844

Perhaps this is a good place to explain that the Debian package is no longer supported?

nomandera commented 3 years ago

Since it is looking likely that the repository will be gone now forever could instructions on how to migrate to a supported install be provided as a sign of good faith from Matomo?

Findus23 commented 3 years ago

@nomandera If I can find some time in the next days, I'll write something. As I never used the package and can't reliably test it, this would be more of a community wiki for everyone to contribute to.

nomandera commented 3 years ago

I very much appreciate the efforts @Findus23 as currently we are all feeling pretty annoyed at Matomo over here.

We followed the official install guide, purchased and renewed several addons during the subsequent years, contributed time and effort in this project and for our troubles we are now left with a a load of completely unmaintainable installs in the Douglas Adams style of "So long and thanks for all the fish".

This is what happens if a project just kills a repo:

Reading package lists... Done  
E: Failed to fetch http://debian.matomo.org/dists/piwik/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: The repository 'http://debian.matomo.org piwik InRelease' is no longer signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
mattab commented 3 years ago

@BenSturmfels Thanks for the note the FAQ has been fixed

Thanks for the hint @nomandera - we restored the website at https://debian.matomo.org/ so it still works for now. (i hadn't realised setting a redirect would have broken the system...)

To install Matomo on Debian:

  1. your system can meet the requirements
  2. then proceed to install Matomo by following the instructions in our Matomo Installation Guide.

And when you install Matomo using these steps, you can enter the DB credentials of your existing database (the one that was managed by Debian before, you will find the DB credentials in the config/config.ini.php file), so your database will be reused and no data be lost.

Maybe @Findus23 will provide more details, or you can write here any questions or issues, we will help you migrate

BenSturmfels commented 3 years ago

Thanks @mattab! I still feel it would have been helpful to me to have a more direct statement in that FAQ answer such as "The previous Debian package repository is no longer actively maintained. We recommend migrating to the our standard installation approach." Thanks again.

mattab commented 3 years ago

@BenSturmfels we've just updated the page at https://debian.matomo.org/ - hope it's more clear. i'll leave the FAQ as is I think as people would easily find https://debian.matomo.org/ if they search :+1:

BenSturmfels commented 3 years ago

Thanks @mattab

nomandera commented 3 years ago

OK we are definitely making progress but we are not there yet.

The instructions 

To install Matomo on Debian:

    your system can meet the requirements
    then proceed to install Matomo by following the instructions in our Matomo Installation Guide.

is fine for a new install but if you were to follow this as-is for an exiting apt install you would end up with a Frankensteins monster of two installs.

I would suggest we should be looking at an initial backup phase (of what I cant say) and then an apt remove or even and apt purge of apache2 matomo.

WARNING: untested please anyone following this thread dont just try this without being able to recover.

So what actually

Findus23 commented 3 years ago

Create a backup of everything (Matomo files and Database) is always a good thing.

In theory nothing needs to be backed up (even config/config.ini.php can be regenerated by going through the installer again). But there are a few things that will be lost:

You can download every release of Matomo from https://builds.matomo.org/ which would allow you to migrate to the identical version and afterwards update. In theory you can do both things at once, but I think doing them separately is easier.

Things you also need to set up, that the debian package included:

Also I noticed that an apt remove deletes the Matomo directory so maybe copy it somewhere else first.

nomandera commented 3 years ago

Just to confirm.

Is Matomo going to provide a set of upgrade instructions to port apt existing users to a mechanism that is still supported or is this on the community?

I ask because I have been tasked with pulling this off here and I dont want to forge ahead trying to work this out on the fly if an correct/official methodology is forthcoming.

Findus23 commented 3 years ago

@nomandera As noone asked here anymore, I assumed the steps explained above were enough.

But if you have any questions, I can answer them when they arise.

nomandera commented 3 years ago

Thanks for the follow up. I just assumed that since apt was an official install method and since it has now been officially removed that some guide to port would be forthcoming.

There is a big difference between a series of tips and a official supplied and tested migration document.

Please realise that those of us with this in commercial production are now in a situation of having to build, R&D and test this procedure. This is unplanned cost and not an easy sell.

It is what it is I suppose but Matomo did not shine here. Far from it.

e-gaulue commented 3 years ago

Hi all,

The wave is going to rise as admins will move from Buster to Bulleyes.

@Findus23 You can download every release of Matomo from https://plugins.matomo.org/ which would allow you to migrate to the identical version and afterwards update. In theory you can do both things at once, but I think doing them separately is easier.

Where? Following this link, it's far to be clear for me. Most of us will look for 3.14.1, the last that was provided.

Has the installer changed a lot between 3.14.1 and now? I mean file location and permission. If not, my proposal is to get this 3.14.1 installer package. Then install. This should result in having 2 matomo installation. Then, I'll try either to migrate old data in new install, or keep old package parts : logrotate, cron... Then ugrade

I can provide the shell command I used. I was on a classical Buster and move to a classical Bulleyes. But I need the 3.14.1 installer.

Regards,

Findus23 commented 3 years ago

@e-gaulue You are right, I mistyped the URL. I meant https://builds.matomo.org/

e-gaulue commented 3 years ago

Debian admins are used to have obsolete packages as long as they are stable and secured. Those interested in newer packages can move to solution (Matomo, phpmyadmin, wordpress, ...) team own Debian packages.

Could anyone bring me to matomo team recommandations regarding apache configuration. I googled and found lots of people saying: "here is the way I installed it", but I did not find the team recommandations. I know we are on the border and one could say: "that's not the matomo job but the apache admin one to secure it".

For instance, in https://github.com/matomo-org/matomo-package/blob/master/debian/conf/apache.conf, we have a kind of partial proposal which aim is to secure installation. Is it still good? Could we have a better one for >=4 version?

By the way, it would be a shame to throw this Debian packaging stuff, as it doesn't look so unusable. I'm really sorry I do not have any time to spend on it (and I'm not a Debian package expert), but looking at it, it would take really few time for one of them to refresh this code.

e-gaulue commented 3 years ago

Hi all,

I did manage to move from debian package to matomo classical install, but debian package was great: easy to install, easy to upgrade, easy to backup, easy to trust... So I looked at the files in the debian directory of this current git repository and I didn't find any reason why building 4.5.0 as a debian package would break. So I cloned the project. I made really few changes and got my matomo_4.5.0-3_all.deb package.

Of course, when you try to install it, it complains, but it's really easy to change and correct.

I'm in a situation I can say: "it works for me", but I need help if you want something considered stable.

I'm working with debian for 20 years now, but I don't consider myself as an expert. And regarding Matomo, I'm just a sys admin, not a user. I've got the experience of other web application (wordpress), so I think I understand most of its architecture.

If ever I can help, what is the real need? Get deb packages for which Matomo version and to which Debian destination? buster, bulleyes...

Regards,

Findus23 commented 3 years ago

Could anyone bring me to matomo team recommandations regarding apache configuration.

I don't know Apache well (I never used it), so take this with a grain of salt. But I think the only Apache setup things that are related to Matomo are handled by the .htaccess files that Matomo generates. So the only recommendation on how to set up your Apache instance that is exclusive to Matomo is: Make sure you have .htaccess files enabled in your config.

In nginx this is of course different and one can't generate config files. That's why I published https://github.com/matomo-org/matomo-nginx/ which should be enough to secure an nginx Matomo setup

e-gaulue commented 3 years ago

For those interesting in testing a beta package of matomo 4.5.0 (as of today), you can try the one in the current pull request waiting for validation.

Here is a cook book for Bullseye (maybe also good for Buster):

  1. Set up your gpg credential to build the package, if you don't already have any 
    echo 'DEBEMAIL="Your@email.org"
DEBFULLNAME="Your Name"
export DEBEMAIL DEBFULLNAME
' | tee -a ~/.bashrc
  2. Get the building dependancies 
    sudo apt install gnupg2 lintian debhelper devscripts build-essential fakeroot
  3. Get the sources and check for virus/trojan: this will show what I've changed from the official repo. 
    cd
git clone https://github.com/e-gaulue/matomo-package.git
cd matomo-package
git log
git show
  4. Import the matomo signature as the packager start with downloading the sources and checking them. 
    cd
wget http://builds.matomo.org/signature.asc
gpg --import signature.asc
  5. Compile: You can put whatever you want in the changelog file, as this will be YOUR package anyway. 
    cd matomo-package
make release
  6. Install 
    cd
sudo dpkg -i matomo_4.5.0-3_all.deb
  7. If the previous command complains about dependencies you can install them with 
    sudo apt install -f
nomandera commented 3 years ago

Superb work.

I am extremely interested in this but I have a worry. The APT install method has been officially dropped as a supported install type which leaves me at a critical decision point.

If I commit to testing this I will incur an associated internal cost and then when ready a subsequent deployment cost.

However if the package is not official, tested and released in a timely manner, potentially stalling again in the future due to it being a "one man show not backed by Matomo Inc" I will be back to where we started.

Not to take away from your excellent work but...

could we have an official position statement from Matomo if they plan to back this and commit to keeping it maintained should community efforts need bolstering.

At home/fun/FOSS none of this matters but this now I am managing a bunch of commercial production installs which are stuck.

sgiehl commented 3 years ago

The Matomo team is currently not able to provide an official support any longer as we don't have someone who is familiar enough with the topic and don't have enough capacity to handle that.

@e-gaulue would you commit to provide required updates/changes needed for future releases? If so we could maybe reconsider the decision to drop official support.

e-gaulue commented 3 years ago

Well, my sight is to consider matomo has an opensource project. As far as I understand, matomo can also be used as a paid service.

On the opensource side, there is a community, on the other side, there is a company. According to me, the interactions between them are important: both are needed. If you use the community version, you know you take your own risk: you won't be able to complain to anybody and won't get any money back in case of trouble (will you with the paid version?). If you use the community version, you must report and try to help, that's the way opensource projects live.

As far as I understand, the Debian package is offered by the community side.

I wouldn't say there was a great effort to provide a wonderful Debian package. Being a web application, this usually ends to shell scripts. Those I worked on were rather classical: I mean there was nothing magic that could lead anybody to use the debian package instead of the tar.gz one. But:

Regarding updates/changes for futur release, as far as matomo structure is not fully modified from one version to another, I think you just need one hour to provide a new deb package. It will take more time for a migration from one Debian version two another (I would say 2 days).

So, I can follow motomo version as long as it's not fully modified, with a few days delay. But, I can't say the same with a Debian version change. My delay would be 4-6 months (presently Bullseye is here since the end of august). After, if we are on the community side and people are rather helpful, we could even get it better.

In one word, the answer is Yes if you accept those delays (welcome to the Debian world).

Regards,

sgiehl commented 3 years ago

@e-gaulue Thanks for your feedback. As I'm not very familiar with building Debian packages or have any clue about the effort that would be required on our side to publish such packages, I'm not able to make any decisions here. maybe @mattab @tsteur or @Findus23 are able to help making an official decision on that.

Findus23 commented 3 years ago

@e-gaulue I agree that if there is someone familiar with how Debian packaging works, then keeping the package updated to a new Matomo version should not be a lot of work (as Matomo updates don't affect how the package looks like).

I can always help with the "getting Matomo to work on Debian" part of the issue as I'm a Debian user on all of my devices. But I am not familiar with the packaging part and therefore don't want to maintain it myself.

Apart from changing PHP versions even upgrades between major Debian versions shouldn't be that much work and Matomo should always support the PHP version of the current Debian stable, so as long as people don't want packages for old Debian versions (without newer PHP versions from deb.sury.org), things should be fine there too.

So if you are interested in helping here, I'm really not against creating a new Matomo debian package. But semi-related: I think then, we should move the debian related things out of this repository and only use it for build-package.sh as those two things are not really related (I think the debian package should use the built matomo.zip as a source).

e-gaulue commented 3 years ago

I totally agree.

I lost time trying to understand if there was relationship between build-package.sh and the Debian package Makefile. And at first sight, I did even not understand why the Debian package was dead if there are still commits.

But the Debian package totally depends on the result of build-package.sh, so some Lintian recommandations should directly be implemented in it. That's why I made two different pull requests. After, it's just a question of documentation for the developers. But, my last pull request assume the first one has been/will be validated.

sgiehl commented 3 years ago

We actually could also move the build-package.sh away from this repo. We are currently preparing an automated github action for building a release. So we could also move that script to the main repo within that change and use this repo for the debian package only

e-gaulue commented 2 years ago

I've worked again on the old debian scripts. I did just validate all the Makefile actions were executable. So it gives few new commits.

I've seen pull requests were still not merge. One of them focus on build-package.sh because we get lintian errors or warnings. Lintian is a 'lint' tool for checking debian packages. It has got plenty of rules regarding what looks strange in them. Reason why we have one of this pull request. While not corrected, I will still get those messages. It's not really a problem now. But if we want to move to some automations like: if no Litian error is found, directly push the package to servers, it will be a break.

For the second pull request, I can pull it again with last changes and a better commit message.

I will have a bit of time during this end of year. How could we work on the last stage of the package generation: its upload on the server? I'd like to provide 4.5.0, 4.6.0, 4.6.1 and 4.6.2.

Regards,

sgiehl commented 2 years ago

Hi @e-gaulue. Thanks for the update. Unfortunatelly I can't be of much help with all that stuff. Maybe @tsteur or @mattab can say more on that. But regarding the build-package.sh I'm able to say, that it most likely will get removed here, once https://github.com/matomo-org/matomo/pull/17594 is finished.

mattab commented 2 years ago

Hi @e-gaulue Thank you for your message and offer to contribute! That's great to hear. Would you mind sending us an email via https://matomo.org/contact/ and i'll get back to you in the next week or two? Wishing you all happy end of year!

e-gaulue commented 2 years ago

Hi @mattab, I did send an email, but I still have no answer, any news about this?

colans commented 2 years ago

@e-gaulue Until you're able to connect with the maintainers, would you be able to set up a PPA or something similar on your own, making whatever assumptions are necessary for now?

This would have several advantages:

  1. Those of us who are waiting could start testing it and provide feedback.
  2. We'd have something temporary to use in the interim.
  3. The maintainers would be able to review what you've set up and report any problems.

To state my bias, my Ansible role is the most popular one on Galaxy and it depends on the unmaintained Debian repository. I can't plan for what to do next until I see where this is going. (Do I rewrite the code to do installations the "official" way, or wait until there's a maintained Debian repo? I don't want to do a bunch of work on it if I can simply wait a bit.)

mattab commented 2 years ago

Hi @colans Thanks for your message. i have now been in contact with @e-gaulue and i invite you to wait a bit more time as we may have an update to post in the next few weeks: to be confirmed soon.

e-gaulue commented 2 years ago

Hi all,

It should be ready by the end of this week ! I manage the full stack now. Just have to provide main stable version since 4.0.0 one by one. It's on the way.

e-gaulue commented 2 years ago

Hi all,

For those interested in trying the "new" repository, you can add "next" to the classical https://debian.matomo.org/ URL.

Be careful, this documentation has been written as if it was already the production repository. So do not forget to add next to the URL in the source.list. And remember you'll have to remove it as soon as this site will be in production.

I don't see any reason why it would not work for you, but I believe there are hundreds of different configurations, so we never know. Feel free to discuss here any point that could be improved regarding this package availability and installation.

If most of you reports success, this "next" alias will be removed to become the production URL at the beginning of April.

Best regards,

colans commented 2 years ago

@e-gaulue So, instead of https://debian.matomo.org/, use https://debian.matomo.org/next before April 1st?

e-gaulue commented 2 years ago

@colans Yes, exactly, I just wanted to avoid this URL to be referenced by search engines. I added noindex,nofollow.

colans commented 2 years ago

Seems to work fine when upgrading from an old DB, even though I got this error (which didn't affect anything):

[...Lots of DB updates...]
Matomo has been successfully updated!
Segmentation fault (core dumped)
dpkg: error processing package matomo (--configure):
installed matomo package post-installation script subprocess returned error exit status 139

It killed my terraform apply, but then I re-ran it without problems.

So I'd say this is a success. Thank you! :rocket:

tombreit commented 2 years ago

Same here: Quoting @colans : Seems to work fine when upgrading from an old DB, even though I got this error (which didn't affect anything):

[...]
Matomo database will be upgraded from version 3.14.1 to the new version 4.7.1.
[...]
Matomo has been successfully updated!

Segmentation fault
dpkg: error processing package matomo (--configure):
 installed matomo package post-installation script subprocess returned error exit status 139
Errors were encountered while processing:
 matomo
E: Sub-process /usr/bin/dpkg returned an error code (1)

Thank you for bringing this package back to life.

e-gaulue commented 2 years ago

That's the second time this point is underline. I will look at it.

WhilelM commented 2 years ago

Segmentation fault here too, but during tables update.

Matomo database will be upgraded from version 3.14.1 to the new version 4.7.1.

[...] Executing UPDATE piwik_archive_numeric_2019_09 SET name = 'done' WHERE name = 'done.';... Done. [265 / 537] Executing UPDATE piwik_archive_numeric_2019_10 SET name = 'done' WHERE name = 'done.';... Done. [266 / 537] Executing UPDATE piwik_archive_numeric_2019_11 SET name = 'done' WHERE name = 'done.';... Done. [267 / 537] Executing UPDATE piwik_archive_numeric_2019_12 SET name = 'done' WHERE name = 'done.';... Done. [268 / 537]

Installation is interrupted. Lost message mentionning that's the table piwik_segment that was in update at the segfault.

In kern.log got this

Mar 28 12:22:23 server kernel: [12925988.725592] php[32306]: segfault at 17 ip 00007f8ce4e15dc7 sp 00007ffd11a95d50 error 4 in pdo.so[7f8ce4e0d000+c000] Mar 28 12:22:23 server kernel: [12925988.725647] Code: 9f 73 ff ff 48 89 ef e8 e7 79 ff ff e9 42 fe ff ff 66 90 4c 89 f8 48 8b 7c 24 08 4d 8b 06 48 c1 e0 05 48 03 43 18 48 8b 0c 38 <0f> b6 41 18 3c 39 0f 8e d9 07 00 00 48 89 ea 48 89 ce 4c 89 c7 e8

I restored a backup of database to give it another try, followed by apt install -f. New segfault, at the same point.

Executing UPDATE piwik_archive_numeric_2022_01 SET name = 'done' WHERE name = 'done.';... Done. [294 / 541] Executing UPDATE piwik_archive_numeric_2022_02 SET name = 'done' WHERE name = 'done.';... Done. [295 / 541] Executing UPDATE piwik_archive_numeric_2022_03 SET name = 'done' WHERE name = 'done.';... Done. [296 / 541] Executing ALTER TABLE piwik_segment ADD COLUMN hash CHAR(32) NULL AFTER definition;... Done. [297 / 541] Segmentation fault dpkg: erreur de traitement du paquet matomo (--configure) : installed matomo package post-installation script subprocess returned error exit status 139 Des erreurs ont été rencontrées pendant l'exécution : matomo E: Sub-process /usr/bin/dpkg returned an error code (1)

[...]

In kern.log got this

Mar 28 17:14:00 server kernel: [ 8222.154096] php[11785]: segfault at 17 ip 00007faa2a822dc7 sp 00007ffd6d160220 error 4 in pdo.so[7faa2a81a000+c000] Mar 28 17:14:00 server kernel: [ 8222.154149] Code: 9f 73 ff ff 48 89 ef e8 e7 79 ff ff e9 42 fe ff ff 66 90 4c 89 f8 48 8b 7c 24 08 4d 8b 06 48 c1 e0 05 48 03 43 18 48 8b 0c 38 <0f> b6 41 18 3c 39 0f 8e d9 07 00 00 48 89 ea 48 89 ce 4c 89 c7 e8

e-gaulue commented 2 years ago

@WhilelM

If the trouble is just in the database upgrade process, this means new matomo script files are already here. I would just reset the database to the backup and go to the back office admin update page or invoke /usr/bin/php /usr/share/matomo/console core:update. Maybe you will get more information as it will be interactive.

You can also stop package installation to automatically run core:update as non interactive by doing: sudo dpkg-reconfigure matomo.

Regards