SCDF Dashboard - Provide search/filter options in jobs, tasks, streams and apps view page

[majorisit]

The current version of scdf dashboard is not flexible enough to search/filter deployed/running components using its name, id, status, or time. Please make enhancement with the current scdf dashboard to provide search capability at the column header level. Currently, Its not flexible enough to search/filter scdf deployed components due to pagination.

[siddhantsorann]

@sabbyanandan Hey, I'd like to give this a shot, as we would love to have this feature for our use case as well. Can you let me know how to go about it?

[sabbyanandan]

@siddhantsorann: Excellent! @oodamien can help guide by showing you around the codebase, which hopefully can get you going.

[sabbyanandan]

The backend is probably already available. If so, we may have to move this card to the UI repo. @ilayaperumalg/@jvalkeal: could one of you confirm?

[siddhantsorann]

Hey, @sabbyanandan any updates on this? You can also let me know if there is some other ticket I can pick up which might be a good starting point.

[sabbyanandan]

@ilayaperumalg @jvalkeal @oodamien: Could one of you help guide us here?

[sabbyanandan]

I just also realize https://github.com/spring-cloud/spring-cloud-dataflow/issues/3564 could also benefit from this capability.

[siddhantsorann]

Yeah, that also seems to have the same context. We, ourselves would love to have this filter as well. Hence eager to pick this up.

[ilayaperumalg]

@siddhantsorann Thanks for the interest in this and sorry about the delay in getting back to you on this.

Here are the changes you would need to consider at the backend before you start designing at the UI (especially those queries related to searching batch job executions from and to etc.,) :

• Introduce the batch job retrieval methods that are necessary on these filtering cases in this JobService type.
• SCDF uses the SimpleJobService implementation of the JobService. Hence, you need to implement those new methods in here.
• You also need to add new contracts and implement the necessary DAO methods in the DAOs in here. For instance, the JdbcSearchableJobExecutionDao has some of the similar implementations.

[siddhantsorann]

Hey, @ilayaperumalg Thanks for the information. I'll get started upon understanding the code base and changes required.

[siddhantsorann]

Hey, @ilayaperumalg I'm planning on starting with a basic search filter on the job executions page. For that, I'll have to implement it in JobService right? And add a new DAO method?

[ilayaperumalg]

Yes, You need to start adding the specific filtering query at the DAO and the corresponding retrieval methods in SimpleJobService

[siddhantsorann]

Hey @ilayaperumalg, with regards to the API for fetch job executions defined here, should I continue using the same API and add an optional parameter "q" which if not empty, will call the DAO which searches with jobName Like.

[ilayaperumalg]

If adding a parameter provides reusability, you can use it.

I suggest you implement a first cut and submit a draft PR. I think it would be good to discuss from there.

[siddhantsorann]

Sure. That makes sense. Will do.

[siddhantsorann]

@ilayaperumalg I have raised the above Draft PR. Please go through it once and let me know if we can follow the same for other filters.

[siddhantsorann]

Hey, @ilayaperumalg any updates on this? Would really help to get an ETA on this. I can plan to pick up some other tickets accordingly. Thanks.

[actepotentiel]

Hello gentlemen, first of all I would like to congratulate you for your excellent work. Filtering job executions is the last feature my team is waiting for to migrate to SCDF. Would it also be possible to include a filter on the job parameters? If this is not a priority for you I would be happy to contribute to the project by developing this feature. Regards.

[siddhantsorann]

Hey @actepotentiel ! This was a similar case with my team, we are currently using SCDF and wanted the filters for jobs, which is why I decided to pick this ticket up and contribute. Currently, as part of the PR I have only added a filter to search with job name. Was hoping this would get me a deeper insight into the codebase to pick up various other filters. Although, depending on the priority defined by SCDF maintainers we could divide this further into multiple filters and issues.

[siddhantsorann]

Hey @cppwfs @ilayaperumalg For this ticket should I raise a PR for other filters? Like job/task execution ID, date range, status etc?

[ilayaperumalg]

Hi @siddhantsorann ,

Sure, please go ahead and submit PRs. Thanks for your interest in contributing!

[siddhantsorann]

@cppwfs @ilayaperumalg Please have a look at the above PR when possible.

[siddhantsorann]

@sabbyanandan Does it also make sense to include a filter for dslText and/or definition for tasks as part of this issue? Currently, filter is available only by name.

[sabbyanandan]

@siddhantsorann: Sure, why not. It is a String literal similar to the Task definition name anyway. Let's add it.

[siddhantsorann]

@cppwfs @ilayaperumalg If we add more filters to job executions, do we need to combine them as well? For example date filter + name filter. Currently, you can only filter by either. Also, if we do that should we consider using queryDSL and predicates? As the repo call becomes dynamic based on all parameters provided by the user in the API request. Otherwise, we might have too many if-else based on all parameters to figure out which repo call to use.

[cppwfs]

Good idea. But for now let us get an MVP where we handle each filter independently.

[siddhantsorann]

So for the next filter, I'll be working on filtering out job executions by task-execution id. Should this be a separate API call in which only task execution id is sent as a parameter?

[siddhantsorann]

@cppwfs @ilayaperumalg Are there any other filters that we need to create? For tasks or batch jobs?

[cppwfs]

@siddhantsorann Absolutely :-) Here they are: https://github.com/spring-cloud/spring-cloud-dataflow/issues/4165 https://github.com/spring-cloud/spring-cloud-dataflow/issues/4166

[cppwfs]

Re Opening because these solutions are mutually exclusive to each other. You can search for exec_id or instance_id or start / end dates. But they can't be combined.