Download and install plugins from (central) repository from within Piwik admin UI

[robocoder]

Proposal: The client is a Piwik plugin. The server, hosted at ActiveAnalytics.net during development, will be moved to plugins.piwik.org.

Offer delta releases which package only the changed files; the update script would remove any files deleted from the distro.

Related tickets:

• 607 - simpler way to install plugins

• 728 - manual update check

• 936 - periodic UpdateCheck for 3rd party-plugins

• 1043 - developer infrastructure

Out-of-scope for the first iteration of this plugin and ticket:

• rate the plugin; report problems; make donations
• price, checkout, and digital delivery for non-free plugins
• plugin submission policy
• use of continuous integration server and plugin unit tests to spot problems before a core Piwik update
• plugin licensing issues (see http://www.gnu.org/licenses/gpl-faq.html#GPLAndPlugins)

[robocoder]

Attachment: screenshot PowerUp.jpg

[mattab]

Anthon, I would like to see such an effort hosted by Piwik itself, for example on a subdomain at plugins.piwik.org ? I would gladly let you lead this project. I believe there is an enormous value in having such a feature in Piwik; this is what made Wordpress what it is now. This also very much in line with our objective to increase the number of third party plugins. Let me know your thoughts :)

[anonymous-matomo-user]

As someone new to piwik, it took me a while to find the plugins at trac. That's why I would like to have such a central plugin to find and install plugins.

BUT: I would prefere it to be hosted by piwik, not by a third party. A plugin like this has quite some power over my system, being a kind of gatekeeper. And although I have no reason to mistrust VIPsoft or ActiveAnalytics.net (never heard of them before, so no offense, guys), I would rather trust piwik.org.

[robocoder]

jimbo: you missed the part about "during development".

[anonymous-matomo-user]

I took a look on the powerup website, functionality looks nice, but there should be an restriction which allows only users with trac logins for piwik.org to upload plugins.

Then there should be an option to reserve Pluginnames, during the first phase of development - as there won't be and upload at first ... .

Additionally the user should be forced to upload the code to a piwik.org server, and not specify a download url. Reasons:

• External Servers may share infected downloads
• External Servers may be down
• External Download Sources can't be checked with unit tests
• With one Server and perhaps some mirrors the download experience is always the same, that is the goal of apple's appstore and the one of wordpress as well.
• Payment can be handled centrally, so that there is a rule, the commercial plugins need to spend a small amount to the piwik team to ensure further development
• ...

Great work so far - go on ;)

[anonymous-matomo-user]

oh and i forgott to say, that developers should sign a copyleft agreement to have the rights of the opensource plugins ...

Sth. like http://www.tine20.org/licenses/metaways_icla_1.1.pdf

Best regards Kay

[mattab]

Kay, all good points. We're discussing internally how the plugin repository should work and this is a work in progress :) but certainly a great idea!

[robocoder]

At the moment, I want to focus on getting the plugin and backend server working to my satisfaction.

Policies and hosting infrastructure are relevant discussion, but not in this ticket.

[anonymous-matomo-user]

@vipsoft: absolutly a great idea and that's why i spend the time to give feedback ;) of course software is work in progress, i just wannted to provide some feedback to make id eveb better ;)

Best regards Kay

[mattab]

I talked with Piwigo project lead, who built the system to host Piwigo's plugins. A few points:

• they are using the open source PEM project to handle plugins hosting;
  • adding plugins, version history, screenshots, description, download, etc.
  • REST API to fetch all the data (eg. from a piwik client plugin) like: list of versions, compatibility with piwik core versions, fetch download stats, etc.
  • Several ways to upload a plugin: submit the zip, submit zip via external URL, or submit zip by giving the SVN path
  • it also provides a UI used to list plugins on the project website.
  • Piwigo example: List of all plugins and an example plugin page
  • Another example: Listing of plugins
• they have seen a huge increase of activity on the project once they enabled this tool with ability for any plugin to be visible on all Piwigo installs (gives credibility to the plugin author)
• they don't moderate a priori the plugins, only a posteriori. I don't think though that we can afford to do this before 1.0 as our API is not stable yet.
• they host the plugins on the same repo as the main core repo (increases the general SVN activity, plus only one repo to maintain).
• currently there is a manual process where new plugins paths are created when the plugin author is interested to get a SVN. However, the system sends an email with the command line to execute to init the SVN so that it is nearly automated. Maintainer can then send the email with password and path to the plugin author.

PEM project is open source and available at: https://gna.org/projects/pem/ Piwigo project lead is also dev of this software and they are planning to improve the UI aspect of the "plugin listing page" to make it a bit easier to navigate. But other aspects are working and in production for a few years now.

I think this solution could help us a lot and save us of similar features. The project is active and in production, and I didn't find other similar projects while searching for it.

What do you guys think?

[robocoder]

The Typo3 extension manager is also a potential candidate:

• http://forge.typo3.org/projects/show/typo3v4-em

[robocoder]

GA Application Gallery: http://www.google.com/analytics/apps/

[robocoder]

Designate "Piwik Labs" for plugins in alpha/beta/preview state that users can test drive and provide feedback.

[mattab]

Maybe we can just flag Alpha/Beta plugins clearly as being non stable, the other state being 'stable' ? They could both be listed in the same UI, etc.

[robocoder]

Add ability to install third-party libraries (./libs), e.g., HTMLPurifier. This keeps the core distribution smaller, and allow for independent updates (i.e., not tied to Piwik's release schedule).

Whether or not we track the dependencies is debateable. For example, the HTMLPurifier stubs will use a mock object if the library isn't installed.

Another example would be Zend Framework. It has a large footprint, so we include only a subset of the components. If the Example plugins were moved to the repository, then Zend_Feed, Zend_Uri, Zend_Validate, ... could be separate.

[halfdan]

As an addition to Alpha/Beta/Stable we should add a "Reviewed" state for plugins that were reviewed by at least one Piwik developer. This would add additional credibility to some plugins. Could also be offered as a service by trusted Piwik consultants.

[mattab]

problem is that the 'reviewed' state should be reset at each plugin release (since we would have to re-review each release)

[anonymous-matomo-user]

in TYPO3 there reviewed extensions - but correct there is a review needed for every release.

Additionally there is a discussion about common / suggested plugins. - What about such a state? (Doesn't need to be reviewed every release, but regularly ... )

Regards Kay

[anonymous-matomo-user]

Hi,

I started a project along those suggestions.

http://plugin.suenkel.org

Of course the project is after about one month in a early stage, but it provides a lot of the specified functionality:

• central pluginstore
• 10 plugins from trac/github available
• fully integreated in Piwik admin settings
• Jenkins-CI: basic test of the plugins (Selenium2 login, phpunit, xhprof, integration)
• Remote update and install plugins (also as zip-filed upload)
• release switch (stable, beta, alpha, developer)
• remove/deinstall plugins (in a "as friendly as possible" way)
• PHPUnit/Selenium2 tests, xhprof/XHProfGUI, static code-metrics with Jenkins
• localisation: text snippets with oTranCe easy translation (ltr only)

Of course the wishlist is very long for such a project...

[mattab]

This is awesome, very promising and exciting. So many users would benefit from an open plugin repository for Piwik analytics!

However I have to be honest: there will be quite a lot of work to re-code or simplify the code a lot. I will give a code review and various thoughts/feedback. I look forward to the next iteration and will try to give feedback asap.

Code/UI review:

• in AutoPluginUpdateAdmin_Cache -> why copy pasting the code and saving in a file? the plugins status if stoed, can simply use Piwik_SetOption and Piwik_GetOption to store in the DB
• I would propose to rename the plugin name from AutoPluginUpdateAdmin to PluginsMarketplace ? or we can decide later a name.
• Too many SMELL in Downloader... please simply extract the directory $plugins into the plugins/ directory and never extra any other top level file from the ZIP ok ? this way code will be simple and safe. This downloader.php shoul be super small :)
• Installer.php: fetchPluginnameFromSource is the same as some code in PluginsManager - please always reuse the existing functions since a lot of code in there is duplicated which would cause more time to review and fix bugs later...
  • A shocking exmaple is: remove() in installerCore -> all this code is already there in pluginsManager (or require small refactor there) ie. it is important that you refactor the code code to make your plugin possible rather than put all the code into your plugin causing spaghetti code. So please, submit patch to core if you need any new helper or refactor, sounds good?
  • the file Installer and InstallerCore should be 1 file and as small as possible.
• I m not sure why AutoPluginUpdateAdmin_Manager also has a list of steps to process (3 classes seem to hold hte list of steps). This class is super complex and seems to reimplement some of PluginsManager ? also code should be more simple :)
• Code in http.php should reuse the http helper (maybe just add POST support #3325)
• ZIP feature, use this text:

Install a plugin in .zip format
If you have a plugin in a .zip format, you may install it by uploading it here.

• Also this work should be included in core, can you please submit the patch for the core modification of adding the ZIP upload? we can commit asap..
  • The plugin code is pretty complex, do you think you could simplify the data model and code as much as possible.. it will be easier to start with less :)
  • We don't need the "Expert mode" to manage the plugins. It will be best to keep as simple as possible and include all features in the default UI
  • When installing a plugin, the UI with refresh is nice, but it's better to do more simple, and simply display the status message above each other with the loading button. This way users can see the full workflow. Also please don't add new CSS or special styles as it's best reuse exisitng CSS!
  • In core we also need to add "Delete" button, displayed once a plugin has been deactivited. Delete will remove all files from the plugin folder.
  • Note: Plugins.ZIP files should not contain the plugins/ folder but directly contain the code to extract in piwik/plugins/
  • We don't need the release switch that complicated: 1) Stable and 2) Beta is more than enough for Piwik plugins!
• However I think the setting should be for each plugin. Because, I might want to be using a Beta of one plugin but the stable of another plugin. I am not sure how this should be done, but for sure the golden role is to KISS !

Plugins repository:

• What technology do you build plugins.piwik.org with? ZFramework or as a WP plugin ?
  • we will integrate with github, so the plugins code will come from github, and the README could be automatically generated from the README in the repo (similarly to wordpress)
  • Later it will be nice to add Comments Review + Star ratings, to help discoverability
• The browse plugin should show similarly to Wordpress plugins the following columns:
  • Name. Below the name add the link "Details" and "Install now"
  • Version
  • Star Rating
  • Description

Awesome that you integrated already 10 plugins!

The integration in the admin settings should be better I think and be fully integrated within the existing "Plugins" menu. There should be a submenu for "Listing Third Party plugins" and "Add new plugin" (submenu are coming in admin: #1552 )

Ok Review done, that's a lot of feedback,

Looking forward to seeing all the changes applied and reviewing next iteration. Let me know if I can help you in any way or if you have any question!

[anonymous-matomo-user]

hi matt,

thanks for the quick review. I have to think about each single point...

Correct me if my overall counclusion fails:

• most of the suggestions are seeking to backport/refactor the functionality to the core: I aggree with that, but I have to dig a little bit deeper in the core/unittest and the patch-process
• Kiss, combine the functionality to "one" PHP-class yes and no... divide and conquer
• reduce complexity of the Manager.php that could be difficult, cause the installation-process is very complex:
  • plugins need manual config (e.g. oxid) otherwise they will break piwik
  • plugins should be able to be updated in "active" and "deactivated" state
  • plugins may have a "majorupdate" or not
  • plugins, most often, do not implement the "uninstall"
  • plugins may need "updates/*.php" from previously installed versions
  • some plugins are zip-ed with NAME/, some with "plugins/NAME/"
  • ... I think this could only be solved by a clear definition of third-party plugins and extended Plugin::getInformation(), which is actually beyond my scope
• the store/marketplace itself is realized in Kata (a lightwheight MVC)

after christmas holidays we'll go further

[mattab]

most of the suggestions are seeking to backport/refactor the functionality to the core: I aggree with that, but I have to dig a little bit deeper in the core/unittest and the patch-process

Perfect... building such a important and "central" system for the Piwik architecture, will be definitely core code at least for all the code dealing with enable/disable etc. the Plugins.

The installation process is slightly complex, but all this complexity should be within a new class "PluginsManager_Updater" or similar, that will contain this logic. But the updates should be picked up by the CoreUpdater anyway so if they contain any DB update, it will be executed on next refresh ?

some plugins are zip-ed with NAME/, some with "plugins/NAME/" we should enforce "NAME/*" in the ZIP or fail otherwise.

I think this could only be solved by a clear definition of third-party plugins and extended Plugin::getInformation(), which is actually beyond my scope

Yes, just do the most simple way: the updates should always be checked by default for example (no need to set a special getInformation() flag). Let me know any other problem or missing getInformation() so we can add it if reqiured.

The general idea is to integrate all changes into the core file so that this plugin is simply a very basic UI to call the "plugins.piwik.org" API to get the JSON list of plugins. Does it sound good?

Enjoy the end of year! See you next year and looking forward to working together on this amazing project!

[anonymous-matomo-user]

Hi,

I used the time until piwik went to github:

• added voting/rating to the Marketplace
• added additional plugins (but some of them are outdated or incompatible...)
• build a Sandbox to test the plugins via Jenkins (automatically)
• add a deployment process (it takes about 10 min install a process to sync a plugin coming from github with the sandbox-tests)
• add developer registration and API with basic CRUD a plugin (http://plugin.suenkel.org/devapi/)
• add support for the new admin menu

Replying to matt:

[..]

Code/UI review:

• in AutoPluginUpdateAdmin_Cache -> why copy pasting the code and saving in a file? the plugins status if stoed, can simply use Piwik_SetOption and Piwik_GetOption to store in the DB

• the cache-file (while processing) is changed with high frequency

• on files "flock" could be used to serialize parallel processing

• the Piwik_CacheFile/Option does not provide a TTL

• var_dump vs serialize But Ill find a workaround.

• I would propose to rename the plugin name from AutoPluginUpdateAdmin to PluginsMarketplace ? or we can decide later a name. Done

• Too many SMELL in Downloader... please simply extract the directory $plugins into the plugins/ directory and never extra any other top level file from the ZIP ok ? this way code will be simple and safe. This downloader.php shoul be super small :) If you read the comment carefully , you will realize that most of them are warnings not to use the downloader.php as library. But Ill fix that.

• Installer.php: fetchPluginnameFromSource is the same as some code in PluginsManager - please always reuse the existing functions since a lot of code in there is duplicated which would cause more time to review and fix bugs later...

The PluginsManager is not able to explore plugins outside of plugins/*. Most of the PluginsManager methods are declared as private. Without refactor this code, is was not possible to derive the class, so I decided to go the proxy-way as far as possible.

• A shocking exmaple is: remove() in installerCore -> all this code is already there in pluginsManager (or require small refactor there) ie. it is important that you refactor the code code to make your plugin possible rather than put all the code into your plugin causing spaghetti code. So please, submit patch to core if you need any new helper or refactor, sounds good?

You call it shocking. I call it, as you might have read in the comments, proxy with extreme caution. Most of the steps using the Plugin/Core classes, as far as they were implemented there And there is not only a small refactor there needed (e.g. uninstall database changes)

• the file Installer and InstallerCore should be 1 file and as small as possible. As mentioned above. A proxy-like to the core until the functionality will exist there
  • I m not sure why AutoPluginUpdateAdmin_Manager also has a list of steps to process (3 classes seem to hold hte list of steps). This class is super complex and seems to reimplement some of PluginsManager ? also code should be more simple :)

simple? get marketplace info, download, unzip of course no side effects, no compromising the piwik installation quite easy, sure ;) the manager and processor were derived from another project, where they are working for a long time. So I reused most of them.

[..]

• Also this work should be included in core, can you please submit the patch for the core modification of adding the ZIP upload? we can commit asap..
  • The plugin code is pretty complex, do you think you could simplify the data model and code as much as possible.. it will be easier to start with less :)
  • We don't need the "Expert mode" to manage the plugins. It will be best to keep as simple as possible and include all features in the default UI
  • When installing a plugin, the UI with refresh is nice, but it's better to do more simple, and simply display the status message above each other with the loading button. This way users can see the full workflow.

Erm, almost all installers I know are running that progress bar UI. I cant see the benefit for the user, to change this well-known behavior.

[..]

• We don't need the release switch that complicated: 1) Stable and 2) Beta is more than enough for Piwik plugins!
  • However I think the setting should be for each plugin. Because, I might want to be using a Beta of one plugin but the stable of another plugin. I am not sure how this should be done, but for sure the golden role is to KISS ! I disagree with that. The piwik-installation is either in production-mode, so the user will not install any alpha/developer-releases, or its a test installation, so she/he will also try out unstable releases. (and this imho is KISS enough without paternalism) But I agree to reduce the the release-types

Plugins repository:

• What technology do you build plugins.piwik.org with? ZFramework or as a WP plugin ? Kata and Zend, wordpress
  • we will integrate with github, so the plugins code will come from github, and the README could be automatically generated from the README in the repo (similarly to wordpress) As mentioned above. If the github-plugins will have the quality, that could be done easily
  • Later it will be nice to add Comments Review + Star ratings, to help discoverability Stars/Rating done, review by a wordpress-page for each plugin (coming soon)
• The browse plugin should show similarly to Wordpress plugins the following columns:
  • Name. Below the name add the link "Details" and "Install now"
  • Version
  • Star Rating
  • Description

Awesome that you integrated already 10 plugins!

12, if they would not fail the Sandboxtest

The integration in the admin settings should be better I think and be fully integrated within the existing "Plugins" menu. There should be a submenu for "Listing Third Party plugins" and "Add new plugin" (submenu are coming in admin: #1552 ) Started, the plugin supports the submenu

The last thing, I want to implement is the CLI-mode. Then I could care about integration/merge the functionality to the core

[mattab]

Thanks everyone for the discussion and suggestions. Im closing this ticket because it's getting long and scope too broad.

@csuenkel2 sorry we wont use your code as is, as you know we would choose to improve core rather than write code around it. The plugin marketplace code will be minimalist / KISS.

Plugins repository V1 will be built for Piwik 2.0 only. Will create ticket in a few weeks once we flesh out details.