search

elastic / elastic-package

elastic-package - Command line tool for developing Elastic Integrations
Other
49 stars 116 forks source link

[Serverless] Support testing against development serverless projects #1808

Open kpollich opened 6 months ago

kpollich commented 6 months ago

Ref draft PR https://github.com/elastic/elastic-package/pull/1766 Ref previous PR https://github.com/elastic/elastic-package/pull/1231

To support package development for serverless, elastic-package should support spinning up Elasticsearch and Kibana in serverless mode in its local stack, e.g. 

elastic-package stack up --serverless

We could also add serverless: true as a profile setting instead of using a CLI argument.

kpollich commented 6 months ago

cc @jsoriano @ruflin

I created this issue to capture discussion as it's not clear what the intended result is here based on the existing PR's. I believe elastic-package stack can already be used to manage serverless projects, they're just not local Docker containers. @jsoriano do we have any docs on that process?

jsoriano commented 6 months ago

@jsoriano do we have any docs on that process?

Umm, unfortunately it looks like we only have some documentation about the variables that can be used with the serverless provider (here), but not about this provider itself. @mrodm do you remember if we have some documentation about stack providers?

jsoriano commented 6 months ago

FTR, serverless support was added in https://github.com/elastic/elastic-package/pull/1374.

mrodm commented 6 months ago

@mrodm do you remember if we have some documentation about stack providers?

AFAIK I think there is no documentation about the stack providers available in elastic-package (e.g. serverless) :( Just the list of options that you already linked.

ruflin commented 6 months ago

elastic-package stack up is used for local development of integrations but also for testing latest snapshot builds quickly. Like it is done today for stateful, it should be as simple to setup a stateless setup (at least view, feature set) on your local machine to do the testing.

jsoriano commented 4 months ago

I gave another try to start the serverless containers locally, managed by elastic-package and docker-compose, with no success. It seems to have certain complexity, it is not very documented beyond the scripts in the Kibana developer environment, and failures found are quite dark. Given the complexity and the uncertainty of its maintenance costs, I am not sure if it is worth to maintain another way to deploy serverless, given also that elastic-package has support for actual serverless, and to reuse any other stack via environment variables.

So I am also considering other options:

From these options I think that the one that would provide more value is the last one. In my opinion package developers should use the existing serverless provider in most of the cases. If they, or some kibana developer, are working at the same time on some kibana feature, they will likely have a kibana developer environment, and it would be nice to make it easier for elastic-package to leverage this existing environment, and same thing with serverless development and devenv. Improvements in this area would also give us better foundations to support other environments that might appear in the future.

@ruflin @kpollich wdyt?

cc @mrodm

jsoriano commented 3 months ago

adding some feature to "import" stack configurations to profiles

Created issue about this, as we had yesterday another conversation about how to use elastic-package with the kibana developer environment: https://github.com/elastic/elastic-package/issues/1999