[RFC] Frontend 3D Model Support

[zackkrida]

Note: this proposal will be preceded by RFCs in the API and Catalog repositories. That work should be started, or at least outlined, prior to this work.

3D Models in the Openverse Frontend

Openverse will add first-class support for 3D models, alongside audio and images. This means 3D models will have their own table in the Catalog, and their own API endpoints. Adding these to the frontend will require some new components and utilities, but should be more straightforward than the implementation of audio because we are not conducting a complete redesign of the site in parallel.

This outline of work is predicated on the idea that we've identified sources of 3D models, and determined the list of metadata fields we need for each model. Basically, this plan talks about the literal, technical frontend implementation of this feature once we've figured out all of the more product-focused details.

Some potential risks

• 3D models are often heavy files. We should lazy-load and defer loading of these resources as much as possible.
• We should probably use thumbnails rather than actually render 3D models in the search results. We could consider something like SketchFab's search results, where they render a 3D model on hover of the search results, when on desktop.
• We'll need to find a standardized 3d model viewer that will work with various formats. We really, really don't want to write our own code here. We will also need to decide if we want to use this universally, or use SketchFab's viewer for SketchFab models, which is an excellent viewer.

Prior Art

• We already show 3D models as part of images, which is great! Fewer scripts to write on the catalog side.
• We also already show Sketchfab's 3D model viewer for their single results: https://search-staging.openverse.engineering/image/00a6a7a4-6605-4cba-bacf-a08b0da546b9

Some code considerations

• When a unique identifier is necessary, we should use the string constant const MODEL_3D = '3d model' to refer to 3D models. Compare this to AUDIO for audio files, for example.
• 3D model support should be enabled via a feature flag, and this work should be able to be committed directly to main without a base feature branch needed. Meta search support for 3D models does not need to be feature flagged.
• e2e tests should be added for any new routes; and updated for existing routes that are modified. PRs will be blocked in the abcense of these tests.
• New content types should always launch in beta until we reach a certain number of results and/or are confident in our UI components (this metric(s) need to be determined).
• Many of the todos identified here can be broken down into even smaller pieces in the future, particularly around the user interface.

Technical implementation

1. Add 3D Models to Meta Search

Prerequisites

• Need a list of 3D model providers to use in meta search. This could be a mix of sources we intend to use in the Catalog but also some that don't have APIs and will stay in meta search after launching 3D model support.

Todos

• [ ] Add 3D models to the content switcher components:
  • [ ] https://github.com/WordPress/openverse-frontend/blob/bbccc36ed49186b369cc4a69693fbc6d1411f29a/src/components/VContentSwitcher/VContentSwitcherButton.vue
  • [ ] https://github.com/WordPress/openverse-frontend/blob/7d74f033ddc4d35889d7c43756c36150beee4192/src/components/VContentSwitcher/VContentTypes.vue
• [ ] Create a model/index.vue endpoint for 3D models in the pages directory
• [ ] Add 3D model sources to the legacy sources lookup
• [ ] Add 3D models to the media constants file, making sure it is set to the ADDITIONAL status.

2. Setup the data for 3D models

• [ ] Create a feature flag for 3D model support. All 3D model features will be behind this.
• [ ] Generate mock/sample api response data for 3D models.
• [ ] Create a new media service for Models in the data directory
• [ ] Update the media constants file with new constants for models, and under the BETA flag.
• [ ] Update the media store with 3D model support.

3. Implement new 3D model UI

Prerequisites

Design work for the new components must be completed, or at least in a good enough place that we decide work can begin.

Todos

• [ ] Add 3D models to the All Results view
  • [ ] Create a VModelCell.vue component in the VAllResultsGrid directory
  • [ ] Add the logic to display models in the VAllResultsGrid.vue component.
• [ ] Create a new single model route pages/model/_id.vue for viewing a 3D model in detail.
  • [ ] Render the model
  • [ ] Render the model info
  • [ ] Render the attribution information
  • [ ] Render the related models
• [ ] Create a new VThreeDModel.vue component that supports displaying 3D models of various file formats

Open Questions

• What model formats do we want/need to support?
• What features do people need in a 3D model search engine? What functionality does our model viewer need to have?
• Should we use SketchFab's viewer for SketchFab models?
• Some models are meant to be 3D printed; some are meant to be used as assets in digital art and games; others are more like standalone pieces of artwork. There are probably different user stories for each of these types. How can we optimize our UI for all of these different experiences?

[AetherUnbound]

When a unique identifier is necessary, we should use the string constant 'MODELS' to refer to 3D models. Compare this to AUDIO for audio files, for example.

Ha, the plural versions of these words are throwing me off :sweat_smile: That means we'll have IMAGES, MODELS, and AUDIO, correct?

[zackkrida]

@AetherUnbound that's such helpful feedback, I was sloppy about pluralization. I've changed the constant to THREE_D_MODEL to be singular and more explicit, and fixed other pluralization inconsistencies in the proposal.

[AetherUnbound]

Awesome, thanks! Maybe we could do MODEL_3D? 😁

[zackkrida]

...that's also much better haha!

[sarayourfriend]

Thanks for writing this up! It's exciting to consider all the new media types. My go-to "birds" search is going to get a lot more diverse and interesting :slightly_smiling_face:

Basically, this plan talks about the literal, technical frontend implementation of this feature once we've figured out all of the more product-focused details

Do we need to discuss the catalog and API side of things as well? Identifying the metadata fields we'd like to try to obtain for each 3D model, how to structure the API, and whether we'd want to somehow integrate the 3D models into the thumbnail cache all jump out to me as things that would be good to discuss up front along with the frontend implementation (or potentially before/alongside as a separate RFC that would inform the frontend implementation).

This mostly comes from an inclination (from experience) that frontend implementation plans are not very good drivers for backend API design; that is, designing a general API with optional specific enhancements required by the frontend, might be a better place to start and could uncover specific hurdles we'd need to cross in the frontend that won't be obvious until we've discussed the API. Audio was this way for example with the long discussions about the alt_files being informed by frontend data needs but also having to happen all the way at the catalog level (i.e., early in the implementation process).

We'll need to find a standardized 3d model viewer that will work with various formats. We really, really don't want to write our own code here. We will also need to decide if we want to use this universally, or use SketchFab's viewer for SketchFab models, which is an excellent viewer.

I would really really like to avoid SketchFab's viewer if possible.

1. It is not free software
2. It has much looser/worse privacy guarantees than we've tried to achieve with Openverse so far

On the other hand, their viewer is extremely convenient and works very well.

In a brief search I found this js-3d-model-viewer library that is MIT licensed. It's not great, but I wonder if we could help improve it and add some of the features we might want (like UI buttons for controlling zoom when a mousewheel isn't available, recentering, snapping to specific axis, etc).

Currently it only support OBJ files though and has poor texture support (the latter of which is something we could probably spend some time contributing against). These might be unacceptable limitations though.

I guess it depends on how much upfront work we're willing to do for this or if it's something we'd rather just get out the door and be able to move on to video.

3D model support should be enabled via a feature flag

Do we have a good way of manipulating feature flags? Do we need to add something to Django for this or is it already configured? If not, it could be nice to add something like waffle to manage these. At minimum I'd say we should be able to enable the feature per-host (i.e., turn it on in staging but off in production) all via the API rather than a deployment or some other thing like that.

Alternatively we could just rely on deployments and use nuxt-feature-toggle. Worst case we could just use a query param that toggles it on.

My vote is for the query param, but it'd be nice to have a recommendation in the RFC for this.

[zackkrida]

Do we need to discuss the catalog and API side of things as well?

Definitely, @AetherUnbound signed up to do the full 3D model feature RFC so I focused solely on the frontend. A lot of this has been floating in my brain for a bit so I wanted to get it out. I do think it makes sense to do the Catalog and API implementations a bit before the frontend.

Regarding viewers, I chatted a bit with @antpb today who works on wp core media, but also around open source projects tackling standardizing 3D content performance, and might have some recommendations for us here 😄

The suggestion to think about feature flagging is really welcome, and probably warrants its own RFC. I like the idea of a query param approach and maybe writing some helpers:

• a useFeature(FEATURE_NAME) composable
• a <VFeature name="FEATURE_NAME"> wrapper component

Historically we've done it with env vars and deploys.

[sarayourfriend]

Definitely, @AetherUnbound signed up to do the full 3D model feature RFC so I focused solely on the frontend. A lot of this has been floating in my brain for a bit so I wanted to get it out. I do think it makes sense to do the Catalog and API implementations a bit before the frontend.

Oh! Sorry, I totally missed this piece of information. That sounds great then :+1:

Regarding viewers, I chatted a bit with @antpb today who works on wp core media, but also around open source projects tackling standardizing 3D content performance, and might have some recommendations for us here smile

Yes! That's awesome. I'd love it if we could help promote those projects and also contribute new features and bug reports :nerd_face:

Feature flags ... and probably warrants its own RFC

For sure, it could be a nice small one I think :slightly_smiling_face:

[AetherUnbound]

Another question, do we need to distinguish between 3D models specifically, or just models in general? For instance, this resource has both 2D and 3D models that are CC licensed: https://kenney.nl/assets?q=2d

[sarayourfriend]

@AetherUnbound I think the 2d stuff there would be classified as textures, not models, right? Though I guess some of those are sprite sheets which aren't exactly textures (that'd be more of an implementation detail).

Maybe "model" by itself is the right way to go.

I was thinking about textures as well, those would probably just fall under our regular image content type (they have all the same metadata like resolution) but potentially as a distinct category. Sprite sheets wouldn't make sense in the images content type; sprite sheet dimensions aren't really relevant, they probably have a whole different set of interesting criteria people would want to filter by.

From a UI perspective it'd be nice to at least be able to filter by them... and I think from a viewer perspective it'd be nice for sprite sheets to have something to zoom and pan around the sheet.

[AetherUnbound]

That totally makes sense @sarayourfriend! I think having them fall under image with a different category works great.

[AetherUnbound]

Closing in favor of WordPress/openverse#171