elastic / connectors

Source code for all Elastic connectors, developed by the Search team at Elastic, and home of our Python connector development framework
https://www.elastic.co/guide/en/enterprise-search/master/index.html
Other
58 stars 116 forks source link

Add service.idling: 1 to all ftests + decrease size of ftest for gcs #2653

Closed artem-shelkovnikov closed 3 days ago

artem-shelkovnikov commented 3 days ago

Closes https://github.com/elastic/connectors/issues/2638

Functional tests became a bit slower after change in https://github.com/elastic/connectors/commit/b8b66aa31dec2d48396c7e79e604e9290695c82f#diff-5f4b320c98a5619e1197864adbca34517cf6ce4aafdb3d17ca5079dc859998a1: now cron does not respect seconds at all (which is impossible to change in Kibana anyway), so tests started to run a bit slower - full sync is scheduled 30/60 seconds later.

This PR slightly speeds up functional tests by making service.idling: 1 making service poll for jobs and try to schedule jobs every second.

Additionally this PR decreases the GCS functional test payload by around 10% so that it would always pass. Now test takes around 500-550 seconds and with scheduling taking up to 90 seconds it could exceed the threshold of 600 seconds that ftests has to run the first sync.

One small change is just standartisation of all ftest config.yml files: deleting unnecessary lines.

Checklists

Pre-Review Checklist

github-actions[bot] commented 3 days ago

šŸ’” Failed to create backport PR(s)

Status Branch Result
āŒ 8.13 Commit could not be cherrypicked due to conflicts
āœ… 8.14 https://github.com/elastic/connectors/pull/2654

Successful backport PRs will be merged automatically after passing CI.

To backport manually run: backport --pr 2653 --autoMerge --autoMergeMethod squash