Unable to view two group slices with sort&filtering applied to active slice [BUG]

sergiev commented 7 months ago

Describe the problem

Our current use case of 51 is to look for the samples with FPs of most confidence. I have a dataset view for it, code of view below:

Recently it become clear that each sample should have auxiliary image for better navigation. I've added it, but in this state I have unpleasant dichotomy:

either all of my previous filter&sorting view are either disabled (e.g. label thresholding by confidence)
or samples are shown without access to auxiliary image, like at first screenshot in this issue.

Code to reproduce issue

Second dataset d2 is simply dataset samples + corresponding auxiliary samples. The exact code is here:

orientir_root = "/path/to/auxiliary/root"
def find_orientir(sample, orientir_root=orientir_root):
    stem = Path(sample.filepath).name
    x = glob(orientir_root+"/**/"+stem, recursive=True)
    if len(x)>0:
        return x[0]
    return False

d2 = fo.Dataset(NAME + "_orientir", overwrite=True)
d2.add_group_field("group", default="tile")
d2_samples = []
for sample in tqdm(dataset[:10], desc="adding orientir"):
    group = fo.Group()
    new_sample = sample.copy()
    new_sample["group"]=group.element("tile")
    orientir_path = find_orientir(sample)
    d2_samples.append(new_sample)
    if orientir_path:
        orientir_sample = fo.Sample(filepath=orientir_path, group=group.element("orientir"))
        d2_samples.append(orientir_sample)
d2.add_samples(d2_samples)
print([len(i) for i in [d2, d2_samples]])

Here's the code I expect to do the desired filtering, evaluation & sorting WITH access to aux image, but it gives nothing more than at first screenshot in this issue:

c2 = F("eval").switch(mapping={F() == "fp": F("confidence")}, default=0)
h2 = d2.filter_labels("predictions", F("confidence") > CONF_THD, only_matches=False)
he2 = h2.evaluate_detections("predictions", gt_field="ground_truth", eval_key="eval", iou=0.25)
v2 = h2.set_field("predictions.confp", F("detections").map(c2).max())
s2 = fo.launch_app(dataset=h2, color_scheme=COLORS, port=1553, auto=False)

System information

OS Platform and Distribution Linux Ubuntu 22.04.3
Python version 3.10.9
FiftyOne version 0.23.6
FiftyOne installed from pip

Other info/logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Please do not use screenshots for sharing text. Code snippets should be used instead when providing tracebacks, logs, etc.

Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the FiftyOne codebase?

[ ] Yes. I can contribute a fix for this bug independently
[X] Yes. I would be willing to contribute a fix for this bug with guidance from the FiftyOne community
[ ] No. I cannot contribute a bug fix at this time

benjaminpkane commented 7 months ago

Hi @sergiev. Thanks for the report. In your first sceenshot I believe there is a SelectGroupSlices stage in your view bar? I believe that is the issue. Can you share how you are creating that view?

sergiev commented 7 months ago

Hi @benjaminpkane Nope, that's the view of original dataset without such stage But when I try to apply same filter&sort on d2 (here I refer to c2 / h2 code block above), such slice stage appears. In other words, my question is how to de-select that slice when I look for sorted samples in GUI.

benjaminpkane commented 7 months ago

I believe I understand your issue. I've just undrafted https://github.com/voxel51/fiftyone/pull/4097, which should patch it

If you want to test the change yourself, we have source installation directions here! 🤗

sergiev commented 7 months ago

@benjaminpkane well the view is good now at every aspect... except essential one - the auxiliary images are not being displayed: If i click on it, there'll be just blank gray app background.

d2 = fo.Dataset(NAME + "_orientir", overwrite=True)
d2.add_group_field("group", default="tile")
d2_samples = []
for sample in tqdm(dataset[:50], desc="adding orientir"):
    group = fo.Group()
    new_sample = sample.copy()
    new_sample["group"]=group.element("tile")
    d2_samples.append(new_sample)
    orientir_path = find_orientir(sample)
    if orientir_path:
        orientir_sample = fo.Sample(filepath=orientir_path, group=group.element("orientir"))
        d2_samples.append(orientir_sample)
d2.add_samples(d2_samples)
c2 = F("eval").switch(mapping={F() == "fp": F("confidence")}, default=0)
hcv2 = d2.filter_labels("predictions", F("confidence") > CONF_THD, only_matches=False)
_ = d2.select_group_slices("tile").evaluate_detections("predictions", gt_field="ground_truth", eval_key="eval", iou=0.1)
v2 = hcv2.set_field("predictions.confp", F("detections").map(c2).max())
session = fo.launch_app(dataset=d2, color_scheme=COLORS, port=1703, auto=False)
session.view = v2.sort_by(("predictions.confp"), reverse=True)

I've checked that the orientir_path is achievable from the same ipython process, and it's the classic JPG, but it's still unavailable. I'll be extremely grateful to be told how to make it right...

sergiev commented 7 months ago

A couple of observations. If I add select_group_slices() as final stage view:

the aux images are being displayed in list, but by click it still opens blank background. Still not sure which part of the system the issue relates to:
Label filtering disappears (labels should look sparse like at screenshot below that's made when no slice selection stage added):

benjaminpkane commented 7 months ago

Hi @sergiev.

Can you share any browser errors for the blank images? Or a dataset that reproduces the issue? The sample could be malformed.

Regarding SelectGroupSlices should probably be avoided for your use case. It will "reload" the unfiltered samples (see below demonstration)

import fiftyone as fo
import fiftyone.zoo as foz

F = fo.ViewField

dataset = foz.load_zoo_dataset("quickstart-groups")
view = dataset.filter_labels("ground_truth", F("label") == "Car")
fo.pprint(view._pipeline())

Out

[
    {'$match': {'$expr': {'$eq': ['$group.name', 'left']}}},
    {
        '$addFields': {
            'ground_truth.detections': {
                '$filter': {
                    'input': '$ground_truth.detections',
                    'cond': {'$eq': ['$$this.label', 'Car']},
                },
            },
        },
    },
    {
        '$match': {
            '$expr': {
                '$gt': [
                    {'$size': {'$ifNull': ['$ground_truth.detections', []]}},
                    0,
                ],
            },
        },
    },
]

With SelectGroupSlices

selection = view.select_group_slices("left")
fo.pprint(selection._pipeline())

Out (note the lookup stage reloads the samples into the view)

[
    {'$match': {'$expr': {'$eq': ['$group.name', 'left']}}},
    {
        '$addFields': {
            'ground_truth.detections': {
                '$filter': {
                    'input': '$ground_truth.detections',
                    'cond': {'$eq': ['$$this.label', 'Car']},
                },
            },
        },
    },
    {
        '$match': {
            '$expr': {
                '$gt': [
                    {'$size': {'$ifNull': ['$ground_truth.detections', []]}},
                    0,
                ],
            },
        },
    },
    {'$project': {'group': True}},
    {
        '$lookup': {
            'from': 'samples.66198ca8773bd547b725727a',
            'let': {'group_id': '$group._id'},
            'pipeline': [
                {
                    '$match': {
                        '$expr': {
                            '$and': [
                                {'$eq': ['$group._id', '$$group_id']},
                                {'$eq': ['$group.name', 'left']},
                            ],
                        },
                    },
                },
            ],
            'as': 'groups',
        },
    },
    {'$unwind': '$groups'},
    {'$replaceRoot': {'newRoot': '$groups'}},
]

sergiev commented 6 months ago

Hi @benjaminpkane

Can you share any browser errors for the blank images?

Checked at latest stable versions of Chrome & Edge. Here's what i've noticed:

Also, in class names of canvases that are not being rendered in desired manner appears word invisible. The one in top-left corner has not such property.

Or a dataset that reproduces the issue? The sample could be malformed.

Here's 10 pairs from my dataset, none of them displayed properly, but 20 corresponding images has been saved via v2[:10].export(export_dir="kexport", dataset_type=fo.types.FiftyOneDataset: orientir10.zip

Regarding SelectGroupSlices should probably be avoided for your use case.

Here's the couple of clarifications about the view where screenshots above were made

sergiev commented 6 months ago

What I suppose now is that I should address not bare $predictions but corresponding field of each group's tile sample. Any ideas on doing it without excluding orientir slice from the view?

sergiev commented 6 months ago

Hi @benjaminpkane @brimoor @ehofesmann! I really hope to get a hint on it.

sergiev commented 6 months ago

Hiiiiiii @benjaminpkane @brimoor @ehofesmann! I'm still really hoping to get a hint on it.

sergiev commented 6 months ago

@benjaminpkane any news on it?

benjaminpkane commented 6 months ago

Hi @sergiev. Pardon the delay. It appears you are encountering an issue fixed by https://github.com/voxel51/fiftyone/pull/4345.

Adding the below to your view may resolve the issue

view = view.exists("ground_truth.detections", bool=False).set_field(
    "ground_truth", None
)

voxel51 / fiftyone