migrate `obs-infraobs-integrations` owned packages to `format_version: 3.0.0`

tommyers-elastic commented 11 months ago

This issue captures all the work done and additional work required to migrate obs-infraobs-integrations owned packages to format_version 3.0.0.

In order to get some packages passing the new package-spec validations, several validation exclude rules were added. For these packages, the underlying issues should be fixed as soon as possible and the exclude rules removed. The fixes required for this are also captured in this issue.

The packages being migrated as part of this issue are:

[x] activemq
[x] airflow
[x] ~~apache [BLOCKED]~~
[x] apache_spark
[x] apache_tomcat
[x] azure_app_service
[x] azure_functions
[x] cassandra
[x] ceph
[x] citrix_adc
[x] cockroachdb
[x] coredns
[x] couchbase
[x] couchdb
[x] etcd
[x] golang
[x] hadoop
[ ] haproxy [BLOCKED]
[ ] ibmmq [BLOCKED]
[x] iis
[x] influxdb
[x] jolokia_input
[x] kafka
[x] kafka_log
[x] memcached
[x] microsoft_sqlserver
[x] mongodb
[x] mysql
[ ] nagios_xi [BLOCKED]
[x] nginx
[x] oracle
[x] oracle_weblogic
[x] php_fpm
[x] postgresql
[x] prometheus_input
[x] rabbitmq
[x] redis
[x] redisenterprise
[x] salesforce
[x] spring_boot
[ ] sql_input
[ ] statsd_input
[x] tomcat
[x] traefik
[x] vsphere
[x] websphere_application_server
[x] zookeeper

PRs:

https://github.com/elastic/integrations/pull/8159
https://github.com/elastic/integrations/pull/8215
https://github.com/elastic/integrations/pull/8170 (activemq -> etcd)
https://github.com/elastic/integrations/pull/8260
https://github.com/elastic/integrations/pull/8171 (golang -> nginx)
https://github.com/elastic/integrations/pull/8203 (oracle -> zookeeper)
https://github.com/elastic/integrations/pull/8216 (leftovers - azure_app_service, citrix_adc, redis, tomcat)
https://github.com/elastic/integrations/pull/8234 (more leftovers - influxdb, mongodb)
https://github.com/elastic/integrations/pull/8291 (statsd_input, sql_input)
https://github.com/elastic/integrations/pull/8347
https://github.com/elastic/integrations/pull/8348

Issues blocking migration to format_version 3.0.0

[x] #8209
[x] [azure_app_service]: resolving external fields failed: can't resolve fields: field geo.continent_name cannot be reused at top level (see: #1)
[x] [citrix_adc]: resolving external fields failed: can't resolve fields: field interface.id cannot be reused at top level
[ ] #8210
[x] [redis]: resolving external fields failed: can't resolve fields: field os.full cannot be reused at top level
[x] [tomcat]: resolving external fields failed: can't resolve fields: field geo.city_name cannot be reused at top level
[ ] [nagios_xi]: multiple set processors setting event.dataset in this file. (see: #1). Probable fix could be a breaking change.

Issues that need fixing but are currently skipped in 'validation.yml'

Package name	Validation skipped	Progress
activemq	SVR00004,SVR00002	- [ ]
airflow	SVR00002	- [ ]
apache_spark	SVR00004,SVR00002	- [ ]
apache_tomcat	SVR00004	- [ ]
azure_functions	SVR00004,SVR00002	- [ ]
cassandra	SVR00004,SVR00002	- [ ]
ceph	SVR00002	- [ ]
cockroachdb	SVR00002	- [ ]
coredns	SVR00002	- [ ]
couchbase	SVR00002	- [ ]
couchdb	SVR00002	- [ ]
etcd	SVR00002	- [ ]
golang	SVR00002	- [ ]
hadoop	SVR00004,SVR00002	- [ ]
iis	SVR00002,SVR00004	- [ ]
influxdb	SVR00002	- [ ]
kafka	SVR00002,SVR00004	- [ ]
memcached	SVR00003,SVR00002	- [ ]
microsoft_sqlserver	SVR00004,SVR00002	- [ ]
mongodb	SVR00004,SVR00002	- [ ]
mysql	SVR00002,SVR00004	- [ ]
nginx	SVR00004,SVR00002	- [ ]
oracle	SVR00002	- [ ]
oracle_weblogic	SVR00002,SVR00004	- [ ]
postgresql	SVR00001,SVR00002,SVR00004	- [ ]
rabbitmq	SVR00002	- [ ]
redisenterprise	SVR00002	- [ ]
salesforce	SVR00002	- [ ]
spring_boot	SVR00004,SVR00002	- [ ]
system (upgrade by different team)	SVR00004,SVR00002	- [ ]
traefik	SVR00004,SVR00002	- [ ]
vsphere	SVR00002	- [ ]
websphere_application_server	SVR00004,SVR00002	- [ ]
zookeeper	SVR00002	- [ ]
citrix_adc	SVR00002	- [ ]
redis	SVR00002,SVR00004	- [ ]

Other non-blocking issues that should be fixed in the future

[ ] [etcd]: While migrating we noticed the type {backend_commit_duration,wal_fsync_duration}.ns.{count,sum} is set to long but it should be float instead. (ref)
[ ] [azure_app_service]: geo mappings are not allowed at root level in ECS (ref)
[ ] [citrix_adc]: interface mappings are not allowed at root level in ECS (ref). Also, interface be nested like observer.egress.interface.*. Please refer ECS docs.
[ ] [redis]: os mappings are not allowed at root level in ECS (ref)
[ ] [tomcat]: geo mappings are not allowed at root level in ECS (ref)

ishleenk17 commented 11 months ago

@shmsr : I know there are couple integrations where the TSDB fields are touched. Would recommend to get TSDB testing done for some of those integrations to ensure there is no loss of data.

shmsr commented 11 months ago

@shmsr : I know there are couple integrations where the TSDB fields are touched. Would recommend to get TSDB testing done for some of those integrations to ensure there is no loss of data.

mongodb is the only package where I had to move dimension: true (for multiple datastreams) to ecs.yml to dedup the fields. Then I'll hold the PR until mongodb is tested properly.

tommyers-elastic commented 11 months ago

can we pull the mongodb changes into it's own PR so it doesn't block anything?

shmsr commented 11 months ago

Migrated 4 more packages. See: https://github.com/elastic/integrations/pull/8216

shmsr commented 11 months ago

Here's a script that'd make reviewing these PRs easier. Because of elastic-package test pipeline -g the *expected.json are modified (mostly with timestamp changes which is making the PR harder to review). See: https://github.com/elastic/integrations/pull/8203#discussion_r1360428961

Here is something that might help:

Go to your browser's bookmark manager. Right click and create a new bookmark and paste the following script there and save it.

javascript:(function() {
const headers = document.querySelectorAll('div[class*="file-header"][data-path*="log-expected.json"]');

headers.forEach(header => {
const checkbox = header.querySelector('.js-reviewed-checkbox');

if (checkbox && !checkbox.checked) {
  checkbox.click();
}
});
})();

See the data-path in the above script? It contains the path of the file. *= means contains. You want to further modify the script; refer: https://api.jquery.com/category/selectors/attribute-selectors/
So in this script, I'm marking the files as viewed (when in GitHub PR Review) which contains the string log-expected.json in data-path attribute.
Now when doing PR review click on the bookmark, those files will get marked as viewed and it'd make it much easier for you to review the changes that matters.

Hope this helps.

cc: @ishleenk17 @tommyers-elastic

shmsr commented 11 months ago

Should I pull out mongodb and influxdb from https://github.com/elastic/integrations/pull/8171? As influxdb ci failure is blocking us; I think it better to pull it out from that PR.

cc: @tommyers-elastic

tommyers-elastic commented 11 months ago

yep sounds good - thanks!

shmsr commented 11 months ago

Should I pull out mongodb and influxdb from #8171? As influxdb ci failure is blocking us; I think it better to pull it out from that PR.

cc: @tommyers-elastic

Done. See: https://github.com/elastic/integrations/pull/8234

shmsr commented 11 months ago

@andresrc mentioned a tool that helps in transforming the by-reference visualizations into by-value panels. Here's what I found:

https://github.com/elastic/package-spec/issues/316 https://github.com/elastic/kibana/issues/129303 (related issue to the tool) https://github.com/elastic/elastic-package/issues/791 (related issue to the tool)

Repository that contains the tool: https://github.com/flash1293/legacy_vis_analyzer (need to explore; at first glance looks simple to use). With this tool, we'd hopefully be able to address SVR00004 (currently for 19 packages we have skipped this validation).

lalit-satapathy commented 11 months ago

Summarising below the beta in fields.yml for each of the packages as part of the v3 migration. This will help to discuss any package specific unique scenarios.	Package name	Data stream Name with Beta in fields.yml
hadoop	namenode	Since hadoop all data streams is made GA, we should just remove beta from fields.yml. @milan-elastic to re-confirm
influxdb	advstatus and status streams	Since the package is in beta, we dont need separate beta in fields.yml
mysql	performance and status	This one still beats data streams is showing beta. We can temporarily keep data streams aligned as to beats and track a backlog separately. Some further analysis is needed.
oracle	memory, performance, system_statistics and tablespace	@agithomas can you confirm which of the data streams of oracle need to be kept beta.
redisenterprise	node and proxy	Since the package is in beta, we can remove beta from fields.yml

agithomas commented 11 months ago

@agithomas can you confirm which of the data streams of oracle need to be kept beta.

I don't find there is any reason to keep it beta anymore considering Oracle Integration package is used by a few customers and there exist no SDHs.

shmsr commented 11 months ago

@lalit-satapathy For mysql, I see:

https://github.com/elastic/beats/blob/main/metricbeat/module/mysql/status/_meta/fields.yml#L5 (status is ga) all other data streams are in beta

So to be consistent, marking them beta except status i.e., galera_status and performance

ali786XI commented 11 months ago

@shmsr Tested the TSDB related scenarios for MongoDB package in this PR and it seems to be working as expected.

milan-elastic commented 11 months ago

Since hadoop all data streams is made GA, we should just remove beta from fields.yml. @milan-elastic to re-confirm

Yes, I agree we should remove it since Hadoop is GA

drewdaemon commented 10 months ago

Repository that contains the tool: https://github.com/flash1293/legacy_vis_analyzer (need to explore; at first glance looks simple to use). With this tool, we'd hopefully be able to address SVR00004 (currently for 19 packages we have skipped this validation).

This is the old repository owned by a former Elastic employee. The maintained version is this fork.

We should be using that fork instead!

shmsr commented 10 months ago

As discussed in the last sprint, I am creating separate issues to track things not done yet as part of this meta issue.

Separate issue for "Issues blocking migration to format_version 3.0.0"
Separate issue for "Other non-blocking issues that should be fixed in the future"
One meta issue for all validations to address:
- Individual issue for each for validation: SVR0000{1,2,3,4,5} (optional; as per need)

After create the aforementioned issues I will close this issue. We will look at those specific issues for better trackability.

shmsr commented 10 months ago

As I have created separate issues for blocking and non-blocking issues, and skipped validations during package-spec upgrade; I am closing this issue. Please feel free to re-open the issue if any discussion is required.

agithomas commented 9 months ago

[etcd]: While migrating we noticed the type {backend_commit_duration,wal_fsync_duration}.ns.{count,sum} is set to long but it should be float instead. (ref)

@shmsr , the above mentioned fields will be changed to type histogram with this PR. I hope, this would solve the problem.

shmsr commented 9 months ago

[etcd]: While migrating we noticed the type {backend_commit_duration,wal_fsync_duration}.ns.{count,sum} is set to long but it should be float instead. (ref)

@shmsr , the above mentioned fields will be changed to type histogram with this PR. I hope, this would solve the problem.

Yep, looks good.

jsoriano commented 4 months ago

@tommyers-elastic please refresh my memory, why was a blocker for spec v3 to migrate dashboards to lens? I have tried to migrate the Apache package and it seems to be possible https://github.com/elastic/integrations/pull/9818

shmsr commented 4 months ago

@jsoriano While migrating packages to spec v3, we encountered issues with multiple packages, including Apache. Error example:

found legacy visualization "Uptime [Metrics Apache]" (metric, TSVB)

Find the actual error here. Noted down some errors while migrating to spec v3.

Later, this PR was opened to address the error above: https://github.com/elastic/integrations/pull/8173. After that, we did not try migrating to spec v3.

jsoriano commented 4 months ago

Later, this PR was opened to address the error above: https://github.com/elastic/integrations/pull/8173. After that, we did not try migrating to spec v3.

So https://github.com/elastic/integrations/issues/8209 could be already closed?

For the migration to v3, please take a look to https://github.com/elastic/integrations/pull/9818.

Thanks!

shmsr commented 4 months ago

Later, this PR was opened to address the error above: #8173. After that, we did not try migrating to spec v3.

So #8209 could be already closed?

For the migration to v3, please take a look to #9818.

Thanks!

Yes. Closed it. Thanks for the PR!

shmsr commented 3 months ago

The checklist in the issue description tracks what packages have been migrated or not from v1/v2 -> v3 format. There were some blockers earlier because of which the migration of some packages could not be done but most of them are hopefully unblocked now.

From the checklist, you see [BLOCKED] unticked boxes against package names that are yet to be migrated like haproxy, ibmmq, and nagios_xi. The input packages could not be migrated because we did not arrive at a conclusion then; see: https://github.com/elastic/integrations/pull/8291#discussion_r1371974310

elastic / integrations