A fast, easy-to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models.
In this PR I ship extension to core block in form of universal query / operation language.
The idea is the following:
we define JSON-based language to express operations on different types of data that we support (for example, for sv.Detections we support filtering, offseting and getting detections metadata - like size or location) and evaluation of unary and binary operations that are useful for conditional logic
definition is extensible - to add new operation we only need to create pydantic entity to host its definition, implementation of that operation and register the operation in UQL execution engine
definitions are self-describing - entities defining ops expose JSON manifests including custom fields expressing types of operations - that together should allow creating relatively generic UI for different kinds of blocks using UQL and execute simple type inference procedure on UI side to very effectively narrow down allowed operations that users can apply in specific contexts - hopefully making the whole thing quite intuitive
we do not make changes into workflows EE, apart from quality of life improvements - selectors embedded in blocks manifests dicts (so far we supported only singular references and lists of references, now we would like to be able to provide Dict[str, reference] as input - as this is very useful for providing parametrisation of UQL execution engine - let's say that u want to filter detections, but threshold of confidence is provided in runtime, not hardcoded in definition)
I do not imply that all blocks should use UQL, this is optional breadth-wise extension for workflows, but I think it will be beneficial, as:
we would be able to have quite generic builders for tons of operations in UI
we would be able to quickly extend UQL
we would eliminate need for multiple custom glue code blocks and python-code blocks (to some extent)
Downsides of the idea:
there needs to be some intelligence in the UI to create builders for operations and to know how to behave in context of specific blocks (for instance - condition would provide all parameters of UQL EE in the manifest definition vs detections filter would implicitly iterate over predictions in batch that is provided as predictions parameter)
the JSON definition is not quite easy to read by human - but it is quite easy to programatically construct and parse, which is more important in that case. I do recommend not pushing into "verbose" UQL, as we will add significant amount of work on the parsing side and on the query construction side, as both sides would need to implement "parser" of some kind
Description
In this PR I ship extension to core block in form of universal query / operation language.
The idea is the following:
sv.Detectionswe support filtering, offseting and getting detections metadata - like size or location) and evaluation of unary and binary operations that are useful for conditional logicworkflowsEE, apart from quality of life improvements - selectors embedded in blocks manifests dicts (so far we supported only singular references and lists of references, now we would like to be able to provide Dict[str, reference] as input - as this is very useful for providing parametrisation of UQL execution engine - let's say that u want to filter detections, but threshold of confidence is provided in runtime, not hardcoded in definition)Downsides of the idea:
predictionsparameter)Here are the examples:
As u see, they all operate on the very similar definitions and in general this extension bring a lot of expression power.
Type of change
Please delete options that are not relevant.
How has this change been tested, please provide a testcase or example of how you tested the change?
Any specific deployment considerations
sv.Detectionsas main data format to proceed with this PR (requires merging https://github.com/roboflow/inference/pull/392)Docs