Add endpoints for just updating specific portions of entityInfo

[alechenninger]

It's a bit stressful updating entityInfo because if anything is messed up it can be fatal, and often you just want to update one small thing, like hook configuration.

It'd be great if you could just hit an endpoint like PUT /metadata/user/hooks/notificationHook/ with notification hook configuration JSON.

[alechenninger]

@bserdar @jewzaam Would you be opposed to something like this? I'd be willing to work on a pull request this week if you don't see any issues. It wouldn't be comprehensive for all of entityInfo, but would set a pattern for other bits of entityInfo to follow.

Specifically I would like to implement something like PUT /metadata/{entityName]/entityInfo/hooks/{hookName} as this would help simplify our release process and reduce barrier to entry for folks still new to lightblue metadata management. We could also more simply automate hook config updates potentially (we already produce the config json on build, but actually getting this into lightblue is still a manual, copy+paste, error-prone endeavor).

In the future, you could imagine additions like PUT ../entityInfo/indexes or defaultVersion, etc.

It could be implemented purely in lightblue-rest as sugar for getEntityInfo -> updateEntityInfo where only relevant bit is replaced. Or, we could add additional API's to Metadata in core, implement them in mongo, and expose them in rest. This is more complicated, but could make the operations atomic, so that an update to >1 piece of entity info at the same time wouldn't risk losing one or more updates to unrelated pieces. That being said, I don't think that would be all that hard given the relative simplicity of metadata operations.

What do you guys think?

[bserdar]

+1 for implementing this in lightblue rest. Adding functionality to core would be difficult, as it does not specify how actually the metadata is stored and managed.

On Sat, Jun 11, 2016 at 8:24 AM, Alec Henninger notifications@github.com wrote:

@bserdar https://github.com/bserdar @jewzaam https://github.com/jewzaam Would you be opposed to something like this? I'd be willing to work on a pull request this week if you don't see any issues. It wouldn't be comprehensive for all of entityInfo, but would set a pattern for other bits of entityInfo to follow.

Specifically I would like to implement something like PUT /metadata/{entityName]/entityInfo/hooks/{hookName} as this would help simplify our release process and reduce barrier to entry for folks still new to lightblue metadata management. We could also more simply automate hook config updates potentially (we already produce the config json on build https://github.com/esbtools/lightblue-notification-hook/tree/master/config-generation, but actually getting this into lightblue is still a manual, copy+paste, error-prone endeavor).

In the future, you could imagine additions like PUT ../entityInfo/indexes or defaultVersion, etc.

It could be implemented purely in lightblue-rest as sugar for getEntityInfo -> updateEntityInfo where only relevant bit is replaced. Or, we could add additional API's to Metadata in core, implement them in mongo, and expose them in rest. This is more complicated, but could make the operations atomic, so that an update to >1 piece of entity info at the same time wouldn't risk losing one or more updates to unrelated pieces. That being said, I don't think that would be all that hard given the relative simplicity of metadata operations.

What do you guys think?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lightblue-platform/lightblue-rest/issues/235#issuecomment-225365434, or mute the thread https://github.com/notifications/unsubscribe/ADgDDcglHYwEgk8ivCWA1jmS1pLNdgO4ks5qKsU1gaJpZM4Igt6T .

[alechenninger]

as it does not specify how actually the metadata is stored and managed.

Naively, couldn't we add abstract methods such as updateHookConfig(String, HookConfiguration) and let lightblue-mongo figure out how to store it?

I guess core would have to be able to parse the hook configuration, but I think it can do this?

I'm mostly curious, I don't have an issue with implementing it only in lightblue-rest.

[bserdar]

It will be defined in core, but won't be used there. It will be like defining a function needed and used in one layer somewhere else just because there happens to be an interface where it fits.

Using the same logic., it doesn't make sense to have createMetadata, createSchema, etc. in core as well, and we should remove them from metadata interface. We can define a separate metadata management interface, and put these functions there. It could be defined in lightblue-mongo.

On Mon, Jun 13, 2016 at 6:23 AM, Alec Henninger notifications@github.com wrote:

as it does not specify how actually the metadata is stored and managed.

Naively, couldn't we add abstract methods such as updateHookConfig(String, HookConfiguration) and let lightblue-mongo figure out how to store it?

I guess core would have to be able to parse the hook configuration, but I think it can do this?

I'm mostly curious, I don't have an issue with implementing it only in lightblue-rest.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lightblue-platform/lightblue-rest/issues/235#issuecomment-225566062, or mute the thread https://github.com/notifications/unsubscribe/ADgDDSgJu5o1diDG-8UYmQ208J3NgMzDks5qLUvfgaJpZM4Igt6T .

[alechenninger]

But putting that in core decouples you from the backend right? I guess I understood the Metadata interface is like a Mediator but for crud on metadata instead of crud on entities. It abstracts the backend.

Again, just exploring this with you :). I'd like to understand the bits and pieces better.

On Mon, Jun 13, 2016, 8:52 AM Burak Serdar notifications@github.com wrote:

It will be defined in core, but won't be used there. It will be like defining a function needed and used in one layer somewhere else just because there happens to be an interface where it fits.

Using the same logic., it doesn't make sense to have createMetadata, createSchema, etc. in core as well, and we should remove them from metadata interface. We can define a separate metadata management interface, and put these functions there. It could be defined in lightblue-mongo.

On Mon, Jun 13, 2016 at 6:23 AM, Alec Henninger notifications@github.com wrote:

as it does not specify how actually the metadata is stored and managed.

Naively, couldn't we add abstract methods such as updateHookConfig(String, HookConfiguration) and let lightblue-mongo figure out how to store it?

I guess core would have to be able to parse the hook configuration, but I think it can do this?

I'm mostly curious, I don't have an issue with implementing it only in lightblue-rest.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/lightblue-platform/lightblue-rest/issues/235#issuecomment-225566062 , or mute the thread < https://github.com/notifications/unsubscribe/ADgDDSgJu5o1diDG-8UYmQ208J3NgMzDks5qLUvfgaJpZM4Igt6T

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lightblue-platform/lightblue-rest/issues/235#issuecomment-225571868, or mute the thread https://github.com/notifications/unsubscribe/ADVaby2XOhPSDtXoqfLqxMmXOZ4X3-y3ks5qLVJ4gaJpZM4Igt6T .

[bserdar]

I don't think putting that in core decouples front-end and back-end. I think it couples them to core unnecessarily.

core is at the bottom of the stack, and has a very limited view of the world. It mainly deals with crud orchestration. All it needs in terms of metadata is to retrieve it in a structure it can interpret.The actual metadata management is done in mongo, and that's only one of the possible metadata implementations. Rest layer has to know the capabilities of the implementation so it can expose the apis.

So ideally, all metadata management operations should be done in mongo, and exposed there, and then the rest layer can downcast that metadata implementation and expose its apis.

On Wed, Jun 15, 2016 at 5:23 AM, Alec Henninger notifications@github.com wrote:

But putting that in core decouples you from the backend right? I guess I understood the Metadata interface is like a Mediator but for crud on metadata instead of crud on entities. It abstracts the backend.

Again, just exploring this with you :). I'd like to understand the bits and pieces better.

On Mon, Jun 13, 2016, 8:52 AM Burak Serdar notifications@github.com wrote:

It will be defined in core, but won't be used there. It will be like defining a function needed and used in one layer somewhere else just because there happens to be an interface where it fits.

Using the same logic., it doesn't make sense to have createMetadata, createSchema, etc. in core as well, and we should remove them from metadata interface. We can define a separate metadata management interface, and put these functions there. It could be defined in lightblue-mongo.

On Mon, Jun 13, 2016 at 6:23 AM, Alec Henninger < notifications@github.com> wrote:

as it does not specify how actually the metadata is stored and managed.

Naively, couldn't we add abstract methods such as updateHookConfig(String, HookConfiguration) and let lightblue-mongo figure out how to store it?

I guess core would have to be able to parse the hook configuration, but I think it can do this?

I'm mostly curious, I don't have an issue with implementing it only in lightblue-rest.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/lightblue-platform/lightblue-rest/issues/235#issuecomment-225566062

, or mute the thread <

https://github.com/notifications/unsubscribe/ADgDDSgJu5o1diDG-8UYmQ208J3NgMzDks5qLUvfgaJpZM4Igt6T

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/lightblue-platform/lightblue-rest/issues/235#issuecomment-225571868 , or mute the thread < https://github.com/notifications/unsubscribe/ADVaby2XOhPSDtXoqfLqxMmXOZ4X3-y3ks5qLVJ4gaJpZM4Igt6T

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lightblue-platform/lightblue-rest/issues/235#issuecomment-226159719, or mute the thread https://github.com/notifications/unsubscribe/ADgDDZGmj3OyVOYc8SEYW47_WsMbwI7mks5qL-DLgaJpZM4Igt6T .

[alechenninger]

Interesting, makes more sense now, thank you. Do you think in the future there would be a good place other than mongo and core to add an interface to abstract working with metadata? (That is, capability we would expect of any metadata implementation, some of which may not be relevant to core)

[bserdar]

Can't say. It is possible to write a completely separate webapp that deals with metadata maintenance. That eliminates all the need to involve core or mongo in metadata maintenance. All we really care about is some way to get it.

On Wed, Jun 15, 2016 at 8:29 AM, Alec Henninger notifications@github.com wrote:

Interesting, makes more sense now, thank you. Do you think in the future there would be a good place other than mongo and core to add an interface to abstract working with metadata?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lightblue-platform/lightblue-rest/issues/235#issuecomment-226204659, or mute the thread https://github.com/notifications/unsubscribe/ADgDDX9NSN2K2VxgjM7SJq4fGOuiP68kks5qMAw3gaJpZM4Igt6T .

[bserdar]

Here's a proposal:

Entity info has these components:

• indexes[]
• hooks[]
• enums[]
• datastore

For the arrays, we can add a an add api that gets a json document: { indexes: [ ... ] } The api adds the index/hook/enum to metadata.

A remove api, something like DELETE .../indexes=[,...]

And a modify api that gets the same json doc as add function.

For datasource, we'd only have a modify api.

[jewzaam]

So the full entityInfo is in the request but use query param to indicate which subset to operate against?

[bserdar]

No, only the relevant part is in the request. Maybe something like a PATCH request containing an object with pieces of entityInfo, and it'll be patched to the stored entityInfo.

[jewzaam]

That makes sense. Could we have a quick spec on what is expected? Sample metadata and how to make a request to operate against an index, assuming it's the same for hooks and enums. How to delete, add, and update.

[bserdar]

Here's a first attempt:

PATCH rest/metadata/entityName

{ datastore: {...}, indexes: [ ... ], enums: [... ], hooks: [ ...], }

If a PATCHed object exists in entityInfo any field included in the patch replaces the existing value.

indexes/enums/hooks are matched by name. If an array element is not in the entityInfo, it is added. If it is there, it is PATCHed.

Deleting: Two options: 1) Separate delete API, or 2) { indexes: [ {name: indexName, delete:true} ] }

On Thu, Jun 23, 2016 at 11:40 AM, Naveen Malik notifications@github.com wrote:

That makes sense. Could we have a quick spec on what is expected? Sample metadata and how to make a request to operate against an index, assuming it's the same for hooks and enums. How to delete, add, and update.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lightblue-platform/lightblue-rest/issues/235#issuecomment-228125359, or mute the thread https://github.com/notifications/unsubscribe/ADgDDSEYUFJR2U6vTdxOFt5lMGKaFWCNks5qOsT3gaJpZM4Igt6T .