[meta] defaults plugins migration to new Event interface and plugin-api 2.0

colinsurprenant commented 8 years ago

This is a meta issue to track progress for refactoring the defaults plugins. Core plugins migration has been completed per #5170. Related #5140.

This migration recipe goals are:

avoid publishing any core plugins for the refactor
be able to test refactored plugins within their refactor branch in Travis
be able to test/build logstash using the refactored plugins within their branches
Plugin Migration Recipe
create branch feature/plugin-api-2_0
this branch should be published upstream in the logstash-plugins repo.

.travis.yml should be updated to

sudo: false
jdk:
- oraclejdk8
language: ruby
cache: bundler
rvm:
- jruby-1.7.25
before_install:
- git clone -b feature/event_interface https://github.com/elastic/logstash
script:
- bundle exec rspec spec

minimally these gems need to be added to the plugin Gemfile

# this is temporary for the feature/plugin-api-2_0 branch and is meant for travis testing
gem "logstash-core", :path => "./logstash/logstash-core"
gem "logstash-core-plugin-api", :path => "./logstash/logstash-core-plugin-api"
gem "logstash-core-event-java", :path => "./logstash/logstash-core-event-java"
gem "logstash-devutils", :github => "elastic/logstash-devutils", :branch => "feature/plugin-api-2_0"

add any other plugins in the gemspec required by this plugin, for example, if logstash-codec-plain is a required plugin add this:
```
gem "logstash-codec-plain", :github => "logstash-plugins/logstash-codec-plain", :branch => "feature/plugin-api-2_0"
```
modify the gemspec to bump the plugin-api version constrain to 2.0:
```
s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
```
and finally, refactor code and specs for new Event interface
Running local specs
if you want to test locally you will have to create a symbolic link to the logstash code base with the feature/event_interface shared branch
```
ln -s /you/path/to/logstash .
bundle
bundle exec rspec spec
```
Helpful Scripts

@ph has added this ruby script to automate much of the initial work: https://gist.github.com/e28dddccfbe32c3985ab51d6f9b9ec41

Plugins for core specs

[x] https://github.com/elastic/logstash-devutils/pull/46
[x] https://github.com/logstash-plugins/logstash-filter-clone/pull/9 |
[x] https://github.com/logstash-plugins/logstash-patterns-core/pull/149 |
[x] https://github.com/logstash-plugins/logstash-filter-grok/pull/81 |
[x] https://github.com/logstash-plugins/logstash-filter-mutate/pull/72 |
[x] https://github.com/logstash-plugins/logstash-filter-multiline/pull/27 |
[x] https://github.com/logstash-plugins/logstash-codec-plain/pull/2 |
[x] https://github.com/logstash-plugins/logstash-input-generator/pull/13 |
[x] https://github.com/logstash-plugins/logstash-codec-line/pull/9 |
[x] https://github.com/logstash-plugins/logstash-codec-json/pull/27 |
[x] https://github.com/logstash-plugins/logstash-codec-json_lines/pull/26 |
[x] https://github.com/logstash-plugins/logstash-input-stdin/pull/10 |
[x] https://github.com/logstash-plugins/logstash-input-tcp/pull/52 |
[x] https://github.com/logstash-plugins/logstash-output-stdout/pull/8 |
[x] logstash-input-heartbeat | @suyograo
[x] logstash-output-zeromq | @suyograo
[x] logstash-codec-collectd | @suyograo
[x] logstash-output-xmpp | @suyograo
[x] logstash-codec-dots | @suyograo
[x] logstash-codec-edn | @suyograo
[x] logstash-codec-edn_lines | @suyograo
[x] logstash-codec-fluent | @suyograo
[x] logstash-codec-es_bulk | @suyograo
[x] logstash-codec-graphite | @suyograo
[x] logstash-codec-msgpack | @suyograo
[x] logstash-codec-multiline | @suyograo
[x] logstash-codec-netflow | @suyograo
[x] logstash-codec-oldlogstashjson | @suyograo
[x] logstash-codec-rubydebug | @suyograo
[x] logstash-filter-anonymize | @suyograo
[x] logstash-filter-checksum | @suyograo
[x] logstash-filter-clone | @suyograo
[x] logstash-filter-csv | @suyograo
[x] logstash-filter-date | @suyograo
[x] logstash-filter-dns | @suyograo
[x] logstash-filter-drop | @suyograo
[x] logstash-filter-fingerprint | @suyograo
[x] logstash-filter-geoip | @suyograo
[x] logstash-filter-json | @suyograo
[x] logstash-filter-kv | @suyograo
[x] logstash-filter-metrics | @suyograo
[x] logstash-filter-ruby | @suyograo
[x] logstash-filter-sleep | @suyograo
[x] logstash-filter-split | @suyograo
[x] logstash-filter-syslog_pri | @suyograo
[x] https://github.com/logstash-plugins/logstash-filter-throttle/pull/11 | @ph
[x] https://github.com/logstash-plugins/logstash-filter-urldecode/pull/7 | @ph
[x] https://github.com/logstash-plugins/logstash-filter-useragent/pull/29 | @ph
[x] https://github.com/logstash-plugins/logstash-filter-uuid/pull/5 | @ph
[x] https://github.com/logstash-plugins/logstash-filter-xml/pull/29 | @ph
[x] logstash-input-couchdb_changes | @ph
[x] logstash-input-elasticsearch | @ph
[x] logstash-input-eventlog | @ph
[x] logstash-input-exec | @ph
[x] logstash-input-file | @ph
[x] logstash-input-ganglia | @ph
[x] logstash-input-gelf | @ph
[x] logstash-input-graphite | @ph
[x] logstash-input-http | @ph
[x] logstash-input-http_poller | @ph
[x] logstash-input-imap | @ph
[x] logstash-input-irc | @ph
[x] logstash-input-jdbc | @ph
[x] logstash-input-log4j | @ph
[x] logstash-input-lumberjack | @ph
[x] logstash-input-pipe | @ph
[x] logstash-input-rabbitmq | @ph
[x] logstash-input-redis | @ph
[x] logstash-input-s3 | @ph
[x] logstash-input-snmptrap | @ph
[x] logstash-input-sqs | @ph
[x] logstash-input-syslog | @ph
[x] logstash-input-twitter | @ph
[x] logstash-input-udp | @ph
[x] logstash-input-unix | @ph
[x] logstash-input-xmpp | @guyboertje
[x] logstash-input-zeromq | @talevy
[x] logstash-input-kafka | @talevy
[x] logstash-input-beats | @talevy
[x] logstash-output-cloudwatch | @talevy
[x] logstash-output-csv | @talevy
[x] logstash-output-elasticsearch | @talevy
[x] logstash-output-email | @talevy
[x] logstash-output-exec | @talevy
[x] logstash-output-file | @talevy
[x] logstash-output-ganglia | @talevy
[x] logstash-output-gelf | @talevy
[x] logstash-output-graphite | @talevy
[x] logstash-output-hipchat | @talevy
[x] logstash-output-http | @talevy
[x] logstash-output-irc | @talevy
[x] logstash-output-juggernaut | @talevy
[x] logstash-output-lumberjack | @talevy
[x] logstash-output-nagios | @talevy
[x] logstash-output-nagios_nsca | @talevy
[x] logstash-output-null | @talevy
[x] logstash-output-opentsdb | @talevy
[x] logstash-output-pagerduty | @talevy
[x] logstash-output-pipe | @talevy
[x] logstash-output-rabbitmq | @talevy
[x] logstash-output-redis | @talevy
[x] logstash-output-s3 | @talevy
[x] logstash-output-sns | @talevy
[x] logstash-output-sqs | @talevy
[x] logstash-output-statsd | @talevy
[x] logstash-output-tcp | @talevy
[x] logstash-output-udp | @talevy
[x] logstash-output-kafka | @talevy
[x] logstash-mixin-aws @talevy
[x] logstash-mixin-http_client | @ph
[x] https://github.com/logstash-plugins/logstash-mixin-rabbitmq_connection/pull/21 | @ph

ph commented 8 years ago

We can automate all of the heavy work minus the actual code/spec? Are we fine doing that?

suyograo commented 8 years ago

@ph +1 for automating..

colinsurprenant commented 8 years ago

it might be tricky to automate the addition of dependent plugins from the gemspec but the basic block

# this is temporary for the feature/plugin-api-2_0 branch and is meant for travis testing
gem "logstash-core", :path => "./logstash/logstash-core"
gem "logstash-core-plugin-api", :path => "./logstash/logstash-core-plugin-api"
gem "logstash-core-event-java", :path => "./logstash/logstash-core-event-java"
gem "logstash-devutils", :github => "elastic/logstash-devutils", :branch => "feature/plugin-api-2_0"

can be easily automated

colinsurprenant commented 8 years ago

also I've seen a bit different .travis.yml, with the integration tests flag?

suyograo commented 8 years ago

Btw, all plugins should have the integration flag set if its not already there..

colinsurprenant commented 8 years ago

I added the list of plugins done so far for passing core specs.

ph commented 8 years ago

@suyograo logstash-codec-oldlogstashjson is marked as completed but I can't find the PR.

suyograo commented 8 years ago

Here's the merge plan:

[x] Review PRs for all plugins (@tal, @ph, @suyograo)
[ ] merge https://github.com/elastic/logstash/pull/5170 to master and 5.0 after cleaning up temp stuff in there (@colinsurprenant)
[ ] release new 5.0.0-beta1.snapshot1 core plugins (@suyograo)
[x] Create new branches (plugin-api-v1) off master in all plugins repo for 2.x development (@ph)
[x] Delete core-api-v2 (@ph)
[ ] Script merge of feature/plugin-api-2_0 to master in all plugins (@ph)
[ ] Update plugins .travis.yml, Gemfile via mass update (@ph)
[ ] Bump major version for all default plugins, add CHANGELOG entry (@ph)
[ ] Make sure tests are green for all plugins (@ph)
[ ] and mass publish (@ph)
[ ] Generate new lock file for master & 5.0 (@colinsurprenant)
[ ] Generate new lock file for 5.0 with newly published plugins, and release new 5.0.0-beta1.snapshot1 packages for testing (@suyograo)

colinsurprenant commented 8 years ago

@suyograo

the plugins branches off master will be 2.x ?
we need to revert both .travis.yml and Gemfile in mass update
you mean new 5.0.0-beta1.snapshot2 ?

suyograo commented 8 years ago

@colinsurprenant updated.

colinsurprenant commented 8 years ago

@suyograo @ph also, .travis.yml should not be updated to old state, we certainly want to keep the Java 8 selection?

colinsurprenant commented 8 years ago

@suyograo @ph core-api-v1 (not v2) branch for LS 2.x development.

ph commented 8 years ago

@colinsurprenant We already have v1, it was before the shutdown change, and used to target LS 1.5?

colinsurprenant commented 8 years ago

@ph how is that possible? we only recently introduced the plugin-core-api gem recently ah yes, ok, I think I remember - it's core-api, we need to create plugin-api-v1

suyograo commented 8 years ago

@colinsurprenant core-api-v1 branch already exists and was created to continue development of plugins under v1 core API (pre shutdown changes, < LS 2.0). core-api-v2 once branched will match v2 of plugin APIs and continue development for them (pre Java Event, < LS 2.0).

Also, here is a .travis.yml we can mass update with: https://gist.github.com/suyograo/331f6384e143b147c09db51a854540c4

Includes Java 8 and --integration tags

ph commented 8 years ago

Also, here is a .travis.yml we can mass update with: https://gist.github.com/suyograo/331f6384e143b147c09db51a854540c4

I am not sure we should mass update with the bundle exec rspec --tag integration, since integration usually mean that we need to add some steps in the travis file to make sure the environment is ready to be used for integration testing. Like installing redis, rabbitmq or supporting libraries.

But this could be a good way to find which plugins are failling and change their travis.yml manually.

@colinsurprenant @suyograo core-api-v2 I agree is a weird name, another option would be to make it explicit and rename them logstash-1.5.x and logstash-2.x ;)

ph commented 8 years ago

@suyograo After discussing on slack ~~zoom~~ I am ok to move ahead with the core-api-v2, its only a branch name, if we change our mind in the future we can always rename it/them.

@colinsurprenant Can we move forward?

colinsurprenant commented 8 years ago

@ph @suyograo it is still confused.

core-api-v1 branch was introduced for supporting the pre-shutdown changes for the 1.5 series. At this point we had no notion of plugin-api, we came up with that ad-hoc branch name at the time.
we need a branch to be able to continue to support le LS versions that are using the plugin-api v1 which are all pre-5.0 releases. if we use the name "v2" in that branch name it will be extremely confusing.
since we now have a concept of plugin-api formal versioning I think we need to use that concept to name branches and use corresponding names and versions numbers.
in that respect, the problem here is the original "core-api-v1" branch name which does not make sense with the new plugin-api concept.

For this I suggest we simply leave the core-api-v1 branch name as-is, for legacy reasons, which anyway will not be needed anymore, and introduce branches with the plugin-api name and versions.

suyograo commented 8 years ago

For this I suggest we simply leave the core-api-v1 branch name as-is, for legacy reasons, which anyway will not be needed anymore, and introduce branches with the plugin-api name and versions.

@colinsurprenant @ph I am fine with naming the branch based on plugin API version. Good with plugin-api-v1

colinsurprenant commented 8 years ago

@suyograo, @ph and myself just reviewed the tasks sequence and Houston, all engines go!

elastic / logstash

[meta] defaults plugins migration to new Event interface and plugin-api 2.0 #5227

Plugin Migration Recipe

Running local specs

Helpful Scripts

Plugins for core specs