Object Discovery - How to find use cases, entities, etc in a Herbs project

[dalssoft]

Problem:

In order to use an use case or entity metadata or to know what the repositories of an application are, we first need to know where to find them.

The current solution are index.js files with reference to all objects. Ex: \srs\domain\usecases\index.js, \srs\domain\entities\index.js and \srs\infra\repositories\index.js

These are the problems we are currently facing:

• There is no standardization of the structure of these files
• There is no standard on how to find an object or a set (eg. how to find a particular UC in the list or filter by metadata)
• It is necessary to update the file for each addition or change of objects (UC, entity or repository)
• You must always know the path for index.js. This is a problem for glues.
• CLI update has difficulty keeping index.js files up to date

Also, some of these index.js files carry metadata, such as index.js for use cases (ex: { usecase: require('./user/createUser'), tags: { group: 'Users', type: 'create'} },)

Suggestion:

Ideally there would be an automatic discovery mechanism for the objects, using their own metadata. In addition, it would be possible to add other metadata to the objects.

As an initial solution I imagined a solution where each UC or entity could "register" in a centralized list.

Ex:

// src\domain\entities\user.js

const User =
        entity('User', {
        ...

discoveryservice.entities.add('User', User)

module.exports = User

// src\domain\usecases\user\createUser.js

const useCase = ({ userRepository }) => () =>
  usecase('Create User', {
  ...

discoveryservice.usecases.add('CreateUser', useCase).metadata.add({ tags: { group: 'Users', type: 'create', entity: 'User' } })

module.exports = useCase

// src\infra\data\repositories\userRepository.js

const repository = class UserRepository extends Repository {
    constructor(connection) {
        super({
            entity: User,
            table: "users",
            ids: ["id"],
            knex: connection
        })
    }
}

discoveryservice.repositories.add('UserRepository', repository).metadata.add({ entity: 'User' })

module.exports = repository

With that a Herbs glue could easly find all the objects in a project to perform its goal.

Ex:

discoveryservice.findAll() // list of all objects

discoveryservice.usecases.findAll() // list of all usecases

...

If this discovery mechanism works, it could be argued whether module.exports and require should be used for this objects at all. It would be easier to just call discoveryservice.usecases.find('CreateUser') because now it is not necessary to know the file path.

Regarding metadata, it would also be possible to add metadata for a object after its inital registration.

Ex:

discoveryservice.usecases.find('CreateUser').metadata.add({ rest: { httpVerb: 'get' } })

The suggested interface (method names, etc) for this solution needs refinement. But the general idea is here.

Given this, the nice thing would be to discuss all the premises presented here: if the problem is relevant, if this is the solution, etc. Please comment.

[jhomarolo]

I think it's a really cool feature to have, especially when we look at the goal of turning herbs into a tool that optimizes development time.

Some questions about your idea:

• Would the discovery service store this data in memory? How harmful can this list in memory be?
• Would the discovery service be a breaking change? How can it be implemented in libraries that already have the index.js file?

[dalssoft]

Would the discovery service store this data in memory? How harmful can this list in memory be?

I think it would be just a list of key/value with a dozens of objects on it. It should be negligible.

Would the discovery service be a breaking change?

Good point. A possible approach would be to keep both mechanisms for glues and start deprecating the index.js methods. However, I think the CLI should migrate and use just the discovery service.

[jhomarolo]

I think it would be just a list of key/value with dozens of objects on it. It should be negligible.

Let's keep in mind about persist those lists into the database through herbs2knex or herbs2mongo, in the future if is necessary.

I think the CLI should migrate and use just the discovery service.

It's a good point to start, change the cli and the todo-list with this new way to list the objects.

[dalssoft]

Let's keep in mind about persist those lists into the database through herbs2knex or herbs2mongo, in the future if is necessary.

Why save the list of objects (UCs, entities, repositories) in the DB? What's the use case?

[jhomarolo]

Why save the list of objects (UCs, entities, repositories) in the DB? What's the use case?

I put this point as an alternative for cases where the developer doesn't want to persist the discovery service in memory (for whatever reason) but in the database. Did you think this is a valid point?

[dalssoft]

Did you think this is a valid point?

not really. memory usage shouldn't be a problem

[dalssoft]

A proof of concept is ready to be evaluated. It was done using todo list on herbs repo.

The code is on this branch: https://github.com/herbsjs/todolist-on-herbs/tree/object-discovery-poc

A few takeaways:

• I created a herbarium list with all the metadata, including entities, use cases and repositories. This is a global repository with methods for including, finding and quering objects. I expect that this list can be the definitive structure for all glues. Diff: https://github.com/herbsjs/todolist-on-herbs/compare/object-discovery-poc?expand=1#diff-c5fa4d91f4dbfa8fe120d46e55f748e66f213052d5896d090a2a4a8241320d8c

• Both REST and GraphQL files are using metadata to find use cases instead of direct require. For me this is the biggest, short term gain. With that, there is no previous knowledge or any hardcoded use cases on this files. Diff: https://github.com/herbsjs/todolist-on-herbs/compare/object-discovery-poc?expand=1#diff-7019c3565f0befc24b91fb577e9a845e21ae73984bebb163a38bfac5427d94c1

• It was ok to remove _uclist.js (a kind of index.js) and create a specialized uc list for each glue (ex: Shelf and Repl) Ideally in the future the only necessary thing to pass to the glue is going to be the herbarium list. Glues can even improve their experience since now they can access all objects, not only UC or entities. Diff: https://github.com/herbsjs/todolist-on-herbs/compare/object-discovery-poc?expand=1#diff-57be294f3bf8f14fc38480414309ca5c18eb1d58a6f5a952ebbd0b34468c9e9bL8

• For applications already using Herbs there is a bit of work to catalog each object using the new method, but it shouldn't take more than a few hours.

The result showed that there is a huge value having a centralized and structured list of herbs object to be accessed anywhere by many parts of the system.

If there is no arguments against, the next steps would be:

• Polish (refactoring, tests, etc) and extract herbarium to its own glue

• Update CLI to reflect the necessary changes

• Enhance the glues (Shelf, mainly) to read direct from herbarium instead of a index.js file

Feedbacks are more than welcome.

[italojs]

I loved the idea, I think it could be useful for many things but I'm not sure if the herbraruim must to receive metadata. Metadata is the differential for the herbs libs, in a perfect world each lib must to have your own metadata, if we don't centralize it into the libs we could to create a strong coupled between libs and hebrarium or have a messy way to generate metadata.

e.g:

const useCase = ({ userRepository }) => () =>
  usecase('Create User', {
  ...

// this metadata must to be inside our usecase. like
// usecase('Create User', { request: {}, response: {}, metadata: { tags: { group: 'Users', type: 'create', entity: 'User' }
discoveryservice.usecases.add('CreateUser', useCase).metadata.add({ tags: { group: 'Users', type: 'create', entity: 'User' } })

module.exports = useCase

I have other point about he code, but I will wait the opened PR

[dalssoft]

but I'm not sure if the herbraruim must to receive metadata

I see your point and to be honest I don't think there are a easy and correct answer here.

Lets dive in:

First, I think the metadata can be "close" or "distant" from an object.

Follow as this... the "type" metadata (now "operation", like create, read, etc) I think it is very "close" to the use case object because it is probably independent of context.

But the "group" metadata is "far", since it might depend on the context to say which group a use case is. You could have many different groups for the same UC.

Metadata like "httpVerb", for instance, is very "far" since is very context-dependent (the REST context).

I would be ok to put "operation" on UC code metadata, but I'm not sure for "httpVerb" or even "group".

we could to create a strong coupled between libs and hebrarium

I'm ok with that. I see Herbarium (temporary name) could be as central as Gotu and Buchu, since it could be "THE WAY" to find objects in a Herbs project. The reason for that is that it brings a big improvement over previous method by using metadata to find objects, which is not possible to do it today. And that is key to the Herbs experience.

messy way to generate metadata

I see it and I don't know how to solve it. How to make it explicit how "far" a metadata is from the original object? How context-dependent a metadata is? That is a good point to explore.

[jhomarolo]

First of all, congrats! A really like the idea about the herbarium, a solution to "kill" all the "list files".

Overall, I believe the concept is ready to go.

Here I have some considerations about refactoring and/or improvement points that we can evaluate to do now in version 1.0 of this new library.

• Refactoring points

  • Remove require-all dependency and move internally to suma library or herbarium lib
  • In the function findUsecases we use a herbarium.crud property, but I don't see how can a user use his own convention (inject crud or something like that) to find usecases.

    const usecases = herbarium.usecases
    const getAll = usecases.findBy({ entity: entity, operation: herbarium.crud.read })[0].usecase  // herbarium.crud.readAll
    const getById = usecases.findBy({ entity: entity, operation: herbarium.crud.read })[0].usecase
    const post = usecases.findBy({ entity: entity, operation: herbarium.crud.create })[0].usecase
    const put = usecases.findBy({ entity: entity, operation: herbarium.crud.update })[0].usecase
    const del = usecases.findBy({ entity: entity, operation: herbarium.crud.delete })[0].usecase
    return { getAll, getById, post, put, del } } 

• Improvement points

  • For users that already use herbs, implementing herbarium by default, will be a difficult task to do, because all objects must be registered into the herbarium. Should be a way to "auto-register" (I don't know how to do it in a easy way) for objects that don't need to customize any metadata.

What do you think about it?

[dalssoft]

First of all, congrats! A really like the idea about the herbarium, a solution to "kill" all the "list files".

tks!

Overall, I believe the concept is ready to go.

great!

Remove require-all dependency and move internally to suma library or herbarium lib

Yeah. Initially I tried to do it without this lib, but it was painful. Maybe for the v1 I could give another try.

but I don't see how can a user use his own convention

Well, herbarium.crud is just a string enum. A dev could just create his own enum. HOWEVER, some glues or CLI might be expecting the herbarium.crud convention to be able to run properly. For instance, the REST example you brought, there is this a pre-process before send to the herbs2rest glue. But if the herbarium.crud convention get stronger, more consolidated, this pre-process could be inside the glue itself.

Should be a way to "auto-register" (I don't know how to do it in a easy way) for objects that don't need to customize any metadata

That would be great.

[dalssoft]

Herbarium is here. https://github.com/herbsjs/herbarium

closing this issue