jordansissel commented 7 years ago

I generated a plugins status with rake generate_plugins_versions on the master branch.

There are many plugins which will not install at all in Logstash. Generated this list with the following:

% rake generate_plugins_versions
<wait a few minutes for this to finish>
% cat plugins_version_docs.json | jq '.failures  | to_entries[] | .key' | fex '"1' | sed -e 's@^.*$@* [&](https://github.com/logstash-plugins/&)@'

MASTER

Plugins which fail to install in master

5.6 failures

You can rerun rake generate_plugins_versions on any other branch( 5.6, etc) to determine which plugins are failing.

Rough proposal:

1) Fix any broken plugins we desire 2) Remove any still-broken plugins from the logstash docs for any affected branches.

jordansissel commented 7 years ago

MASTER

Reasons plugins have failed to install in master:

% cat plugins_version_docs.json | jq '.failures  | to_entries[] | "* \(.key) fails due to \(.value.klass): \(.value.message[47:90])"' | sed -e 's/^"//;s/"$//'

logstash-filter-punct fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-codec-compress_spooler fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-codec-oldlogstashjson fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-filter-cipher fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-filter-collate fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-input-gemfire fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-input-zenoss fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-input-zeromq fails due to Bundler::VersionConflict: gem \"logstash-core\":\n In snapshot (Gemfile
logstash-output-zeromq fails due to Bundler::VersionConflict: gem \"logstash-core\":\n In snapshot (Gemfile
logstash-output-gemfire fails due to Bundler::VersionConflict: gem \"jar-dependencies\":\n In snapshot (Gemf
logstash-filter-metaevent fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-filter-zeromq fails due to Bundler::VersionConflict: gem \"logstash-core\":\n In snapshot (Gemfile
logstash-codec-s3plain fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-filter-oui fails due to Bundler::VersionConflict: gem \"sequel\":\n In snapshot (Gemfile.lock):
logstash-input-fluentd fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-filter-unique fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-output-elasticsearch_java fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-codec-csv fails due to Bundler::VersionConflict: gem \"logstash-core-plugin-api\":\n In snapsh
logstash-output-rados fails due to Bundler::VersionConflict: gem \"logstash-core\":\n In snapshot (Gemfile
logstash-filter-lookup fails due to Bundler::VersionConflict: gem \"logstash-core\":\n In snapshot (Gemfile
logstash-output-zookeeper fails due to Bundler::VersionConflict: gem \"logstash-core\":\n In snapshot (Gemfile
logstash-output-monasca_log_api fails due to Bundler::VersionConflict: gem \"logstash-codec-json\":\n In snapshot (G
logstash-filter-cloudfoundry fails due to Bundler::VersionConflict: gem \"logstash-core\":\n In snapshot (Gemfile

jordansissel commented 7 years ago

The following plugins are marked skip in v5.6.0:

% cat rakelib/plugins-metadata.json| jq 'to_entries[] | select(.value["skip-list"]) | .key' "logstash-input-jdbc" "logstash-output-webhdfs" "logstash-filter-yaml" "logstash-input-example" "logstash-codec-example" "logstash-filter-example" "logstash-output-example" "logstash-input-drupal_dblog" "logstash-output-logentries" "logstash-output-newrelic" "logstash-output-slack" "logstash-input-neo4j" "logstash-output-neo4j" "logstash-input-perfmon" "logstash-input-rackspace" "logstash-output-rackspace" "logstash-input-dynamodb" "logstash-filter-language" "logstash-input-heroku" "logstash-output-google_cloud_storage" "logstash-input-journald" "logstash-input-log4j2" "logstash-codec-cloudtrail"

josephlewis42 commented 6 years ago

Hey @jordansissel, any idea why a plugin might be marked as skip?

I noticed the published docs for logstash-output-google_cloud_storage are pretty out of date and I'd like to get them out again. Maybe it's a quality issue more related to #6693?

jordansissel commented 6 years ago

It was added to the skip list in 97352c7960 (April 2016) with a commit message indicating that it failed to build (https://github.com/elastic/logstash/pull/5054). This was back when we were producing an artifact called "all plugins" which would contain all plugins, and we were rejecting plugins for this all-plugins artifact which had problems (test failures, dependency problems, etc). We no longer produce an all-plugins artifact, though.

@josephlewis42 It's possible that it's safe to remove this plugin from the skip list. Can you open a separate issue for this?

elastic / logstash

Remove broken plugins from documentation #8266

MASTER

5.6 failures

MASTER