We want to register named custom server types. For instance clickhouse or superset or artifactory or vault. However, Server currently requires type to be one of the hard-coded existing types from the json schema.
Why?
Because it has been great to document in the data contract which infra applies to the data product it represents via the Server. And Servers seems like the natural place to have the cli/data platform tooling parse the infra and make things happen on said servers.
Implementation Alternatives
Allow registering custom servers in the cli
The mechanism of registering custom exporter/importer is great. Could we do the same but allow registering custom servers? Maybe as a Server factory in code?
Continuously expand the list of supported servers (Current)
Basically don't support custom, but define (and lower) barrier to entry for new Server types.
It would be naive to assume that the limited list of servers and functionality implemented in the CLI should also limit the list of servers for the data contract. Most platforms will have some need for customizing the list of servers.
If you would you rather see an expanding list of servers, how willing are you to have servers that do not have endpoints in the cli? Basically just existing as unimplemented but supported server types? I ask because building integrations for all the different server types would be an increasingly time consuming task. I presume you want to limit this to a set that you want to explicitly support and have seen from your customers. However, the contract itself then becomes hampered by this limitation.
Add more generic server types
Alternatively, would you be open to adding some more generic server types? I can think of:
Database
Web
Viz (BI/Data Viz portal/endpoint/app)
Catalog
Storage
Basically allow a custom generic server type more than local. (I guess it could be called custom too but that is really unclear during contract writing stage and for consumers of the contract).
Discussion
I think registering custom (but named) server types would be our preference for ease of implementation. However, upstreaming many more server types as long as the burden of adding them is not large is also fine, or adding more generic server types. It would be nice for DCS and the CLI to allow for more Server types.
Custom Server Types
We want to register named custom server types. For instance
clickhouseorsupersetorartifactoryorvault. However,Servercurrently requirestypeto be one of the hard-coded existing types from the json schema.Why?
Because it has been great to document in the data contract which infra applies to the data product it represents via the
Server. AndServersseems like the natural place to have the cli/data platform tooling parse the infra and make things happen on said servers.Implementation Alternatives
Allow registering custom servers in the cli
The mechanism of registering custom exporter/importer is great. Could we do the same but allow registering custom servers? Maybe as a Server factory in code?
Continuously expand the list of supported servers (Current)
Basically don't support custom, but define (and lower) barrier to entry for new
Servertypes.It would be naive to assume that the limited list of servers and functionality implemented in the CLI should also limit the list of servers for the data contract. Most platforms will have some need for customizing the list of servers.
If you would you rather see an expanding list of servers, how willing are you to have servers that do not have endpoints in the cli? Basically just existing as unimplemented but supported server types? I ask because building integrations for all the different server types would be an increasingly time consuming task. I presume you want to limit this to a set that you want to explicitly support and have seen from your customers. However, the contract itself then becomes hampered by this limitation.
Add more generic server types
Alternatively, would you be open to adding some more generic server types? I can think of:
Basically allow a custom generic server type more than
local. (I guess it could be calledcustomtoo but that is really unclear during contract writing stage and for consumers of the contract).Discussion
I think registering custom (but named) server types would be our preference for ease of implementation. However, upstreaming many more server types as long as the burden of adding them is not large is also fine, or adding more generic server types. It would be nice for DCS and the CLI to allow for more Server types.
Interested to hear your preference/thoughts.