voxel51 / fiftyone

The open-source tool for building high-quality datasets and computer vision models
https://fiftyone.ai
Apache License 2.0
8.08k stars 537 forks source link

[BUG] Performing compute_visualization while on patch view throws error (app) #4619

Open ZoeHermansVintecc opened 1 month ago

ZoeHermansVintecc commented 1 month ago

Describe the problem

An error occurs when using the compute_visualization method from the FiftyOne brain plugin in the app. This issue arises specifically when the view in the app is set to "patch view." The objective is to view images at the patch level, create embeddings for the patches, and make a subset of these patches for reclustering to perform more in-depth data curation. However, the compute_visualization method fails when computing the visualization while on "patch view".

Code to reproduce issue

The issue arises when using the compute_visualization plugin in the app.

Steps to reproduce

  1. Create FiftyOne detection dataset from an image dataset and a COCO JSON file with annotations.
  2. Launch the app from a python script.
  3. Navigate to the "patch view"
  4. Run the compute_visualization plugin on patch level.

The following error will occur after a few seconds.

▼Error occurred during operator execution

Traceback (most recent call last):
  File "/home/vintecc/.local/lib/python3.10/site-packages/fiftyone/operators/executor.py", line 277, in execute_or_delegate_operator
    result = await do_execute_operator(operator, ctx, exhaust=exhaust)
  File "/home/vintecc/.local/lib/python3.10/site-packages/fiftyone/operators/executor.py", line 319, in do_execute_operator
    result = await (
  File "/home/vintecc/.local/lib/python3.10/site-packages/fiftyone/core/utils.py", line 2343, in run_sync_task
    return await loop.run_in_executor(_get_sync_task_executor(), func, *args)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/vintecc/fiftyone/__plugins__/@voxel51/brain/__init__.py", line 77, in execute
    fob.compute_visualization(
  File "/home/vintecc/.local/lib/python3.10/site-packages/fiftyone/brain/__init__.py", line 405, in compute_visualization
    return fbv.compute_visualization(
  File "/home/vintecc/.local/lib/python3.10/site-packages/fiftyone/brain/visualization.py", line 108, in compute_visualization
    embeddings, sample_ids, label_ids = fbu.get_embeddings(
  File "/home/vintecc/.local/lib/python3.10/site-packages/fiftyone/brain/internal/core/utils.py", line 759, in get_embeddings
    embeddings = samples.compute_patch_embeddings(
  File "/home/vintecc/.local/lib/python3.10/site-packages/fiftyone/core/collections.py", line 3188, in compute_patch_embeddings
    return fomo.compute_patch_embeddings(
  File "/home/vintecc/.local/lib/python3.10/site-packages/fiftyone/core/models.py", line 1470, in compute_patch_embeddings
    return _embed_patches_data_loader(
  File "/home/vintecc/.local/lib/python3.10/site-packages/fiftyone/core/models.py", line 1657, in _embed_patches_data_loader
    ctx.save(sample)
  File "/home/vintecc/.local/lib/python3.10/site-packages/fiftyone/core/collections.py", line 152, in save
    sample_ops, frame_ops = sample._save(deferred=False)
TypeError: cannot unpack non-iterable NoneType object

Extra information

I wanted to note that when computing visualization via the plugin, the following arguments are passed to the compute_visualization function:

fob.compute_visualization(
    target_view,
    patches_field=patches_field,
    embeddings=embeddings,
    brain_key=brain_key,
    model=model,
    method=method,
    batch_size=batch_size,
    num_workers=num_workers,
    skip_failures=skip_failures,
)

When calling the function when the app is in "image-view", the target_view variable is of the class 'fiftyone.core.view.DatasetView'. However, when calling the function from "patch-view", target_view is of the class 'fiftyone.core.patches.PatchesView'.

The code fails when running sample_ops, frame_ops = sample._save(deferred=False). I believe this is because sample is None, which makes sense as the code tries to iterate over the samples of the dataset, but the dataset consists of patches.

System information

Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the FiftyOne codebase?

brimoor commented 4 weeks ago

Hi @ZoeHermansVintecc 👋

I believe this is fixed by https://github.com/voxel51/fiftyone/pull/4636, which is available on develop now and will be released as part of fiftyone==0.25.0 which is slated for release on Monday, August 19!