media types

[crzypatchwork]

• audio
• text
• 3D (unity/webassembly)
• pdf?

[fraguada]

3d format could start with glTF, and perhaps others. Making a viewer should be straightforward with three.js. I guess it would be somewhere around here. Would be good to make a reusable media component. I'm more experienced with vue, but curious to try it out for react.

[andrevenancio]

I'm happy to create a component that renders .gltf's, .obj's and other common 3d file types. However, they normally just define a 3D model (vertices, faces, etc) and textures. They don't define light, camera (fov, distance, near and far planes etc.)

Another potential problem with allowing this media type is that not all 3d artist have the knowledge to produce 3d models that can render in real-time. Normally a 3d artist working on movie VFX would have no concerns over the number of polygons as they rely on render farms to render their models/animations offline. However, to display them in the browser, they would have to be optimised. Allowing these files without any checks for max file size, a max number of vertices etc can potentially crash the browser.

Its an original and interesting idea, but just requires a bit of thought before implementing.

[fraguada]

All good points. We can extract the appropriate information from glTF and other file types which aren't purely binary (or where binary data can be easily peeked).

[fraguada]

Not sure if model viewer includes some sort of limit parameters on verts, etc. That should be easy enough to extract from json, and if it passes, it can be loaded with https://github.com/google/model-viewer

Also suggested here: https://twitter.com/thespite/status/1368510216636145666?s=20

[spite]

glb would be the format that is self contained and binary https://en.wikipedia.org/wiki/GlTF#GLB, and model-viewer supports it.

[bentstamnes]

I’d very much appreciate audio, and since an NFT should have some real value, supporting WAV is essential. FLAC is an alternative for non-lossy audio. And then I would also provide an MP3 for ease of use.

Supporting private albums on BandCamp can also be an option, but for raw file formats, the above would be a good start.

Edit: WAV, MP3 and FLAC are all natively supported by most browsers through the

[mrdoob]

SuperRare uses https://github.com/google/model-viewer to display glb files 👍

Some examples: https://superrare.co/artwork-v2/you-are-here-16036 https://superrare.co/artwork-v2/starry-night-sky-18966 https://superrare.co/artwork-v2/star-20020

[fraguada]

FYI on file size limits on SuperRare: http://help.superrare.co/en/articles/4199365-faq-tokenizing-3d-art-on-superrare

What are the file size limits? Artwork file: 50MB Thumbnail: 10MB

[andrevenancio]

Interesting. I'm helping doing a bit of cleanup across the site, once things are more stable I'll look into this. Thanks for all the resources!

[mrdoob]

viv3 also uses modelviewer 🤓

https://viv3.com/products/1084

[andrevenancio]

I'm all in for this. but how do you guys think they handle:

1. Different scale models
2. Potentially large geometries from a 3d artist that never exported for real time graphics
3. Light setup?

We can use that "old" library (three.js) (you guys might have heard of it) and use the box bounding box to calculate a distance to the camera or maybe to change the scale of it? How would you implement something so generic @mrdoob @spite ?

[mrdoob]

We can use that "old" library (three.js) (you guys might have heard of it)

I'm starting to forget what that is... 😁

1. modelviewer takes care of that
2. They'll have to learn to optimize for realtime. They can use this tester.
3. No custom lights support. It uses a single environment image (Performance is predictable. Apple does the same in their USDZ viewer).

SuperRare instructions: http://help.superrare.co/en/articles/4199365-faq-tokenizing-3d-art-on-superrare

If anything I think it's a good start.

Disclosure: I've worked on modelviewer. We did it because making a viewer is actually not super easy.

[andrevenancio]

Is your advice to use the modelviewer instead of building a custom one @mrdoob ?

@crzypatchwork what do you think? should we start aiming to introduce this 3d media type?

[mrdoob]

Yes, I recommend starting with modelviewer and build a custom one further down the road if required.

Integrating modelviewer should be pretty straighforward:

If the NFT is a glb file we need include this script (can be hosted in hicetnunc too):

<script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>

And then use the custom element, just as if it was a <img> tag:

<model-viewer src="file.glb" auto-rotate camera-controls></model-viewer>

For the gallery sections I recommend this configuration instead:

<a href><model-viewer src="file1.glb" auto-rotate></model-viewer></a>
<a href><model-viewer src="file2.glb" auto-rotate"></model-viewer></a>
<a href><model-viewer src="file3.glb" auto-rotate></model-viewer></a>
<a href><model-viewer src="file4.glb" auto-rotate></model-viewer></a>
...

This will make sure complex 3d files don't affect the page's responsiveness.

The dimensions can be set via css like any other element:

model-viewer {
  width: 383px;
  height: 383px;
}

[andrevenancio]

Thanks for this. I've just done a quick test and everything seems to be working fine out of the box. I was looking into more detail about the tester you shared earlier and it got me thinking...

Right now the "magic" of the platform is that you drag a file (let's say a .glb), you mint it and it will be automatically uploaded.

Here's where I'm not sure and all feedback is welcome. When we're on the feed page with multiple assets being loaded and displayed at the same time, should we:

1. Display a poster image, instead of loading the 3D model and once the user clicks it, it takes you to the detail page, and there you have the interactive 3d model?

Or should we

2. Display the component on the feed page, allowing the user to interact with it before choosing to open it in detail.

The PRO and CONS for option 1 PRO

• Improves user experience by loading a smaller asset instead of a potentially large 3D model

CONS

• It requires us to update the Mint page and require a poster image if the user uploads a .glb or .gltf file

The PRO and CONS for option 2 PRO

• The user is able to interact with the 3d model on the feed page, making it easy for them to preview the object before leaving the feed page.
• "We" don't have to implement an over-engineered mint form where we need to upload an extra poster image for the 3d model.

CONS

• It can potencially degrade the user experience if the feed has a lot of large 3d models.

Any thoughts?

[elalish]

<model-viewer> is designed for efficient page load; it lazy loads by default and has a lot of interaction customizability. It sounds like you might be interested in reveal="interaction", which only loads the model and dismisses the poster when the user interacts: https://modelviewer.dev/docs/#entrydocs-loading-attributes-reveal

Our posters offer seamless transition from 2D to 3D, even with an unknown aspect ratio of the <model-viewer> element, so I'd recommend keeping the poster with the element to take advantage of that.

[mrdoob]

@andrevenancio

Yes, I vote for 2 as it treats glb files just like the site currently treats images and videos. Plus no need to change the submission form.

However, I proposed using interaction-prompt="none" for performance reasons, but I forgot to mention that I do not think 3d models should be interactive in gallery pages and that parameters solves that too. I would make them interactive only in the artwork page.

[andrevenancio]

Just published the very first one https://www.hicetnunc.xyz/objkt/4132 It should be available at some point today after @crzypatchwork pushes to production

[spite]

seems to be working already? very cool!

[mrdoob]

Nice! I want to collect it! 😁

[mrdoob]

@andrevenancio

Would it be possible to remove the camera-controls parameter when not in the item's page? Right now the UX is a bit confusing plus it steals mousewheel events and breaks the scrolling navigation.

[andrevenancio]

Yeah, I'm all in for advice on that component. I'm updating it now, but I don't have deployment permissions. so it might be a few hours until it gets merged

[andrevenancio]

Nice! I want to collect it! 😁

I think mattdesl was quicker than you :)

[mrdoob]

Grrrr!

[andrevenancio]

@mrdoob I might have missed something... Basically on the objkt detail page we're allowing it to be interactive, and on the feed we're not. but I can still interact with it on the feed. Am I setting the wrong prop?

import React from 'react'
import styles from './index.module.scss'

/* DOCS: https://modelviewer.dev/ */
export const GLBComponent = ({ src, interactive = false }) => {
  console.log('is interactive', interactive)
  return (
    <div className={styles.container}>
      <model-viewer
        className={styles.glb}
        src={src}
        auto-rotate
        camera-controls
        interaction-prompt={interactive ? 'when-focused' : 'none'}
      ></model-viewer>
    </div>
  )
}

[mrdoob]

Sorry, I was wrong.

I think it should be like this:

import React from 'react'
import styles from './index.module.scss'

/* DOCS: https://modelviewer.dev/ */
export const GLBComponent = ({ src, interactive = false }) => {
  console.log('is interactive', interactive)
  return (
    <div className={styles.container}>
      <model-viewer
        className={styles.glb}
        src={src}
        auto-rotate
        {interactive ? 'camera-controls' : ''}
      ></model-viewer>
    </div>
  )
}

[mrdoob]

Also, data-js-focus-visible removes the blue outline:

import React from 'react'
import styles from './index.module.scss'

/* DOCS: https://modelviewer.dev/ */
export const GLBComponent = ({ src, interactive = false }) => {
  console.log('is interactive', interactive)
  return (
    <div className={styles.container}>
      <model-viewer
        className={styles.glb}
        src={src}
        auto-rotate
        data-js-focus-visible
        {interactive ? 'camera-controls' : ''}
      ></model-viewer>
    </div>
  )
}

[andrevenancio]

Can I mint a screenshot of you saying you're wrong? 😹 (joke).

Thanks appreciate it! will commit the changes!

[andrevenancio]

You can now upload .glb and .gltf! Go crazy 🥳

[SableRaf]

Could we possibly add html canvas to the list? The possibilities would be out of this world! Especially with p5.js https://p5js.org/ 👀

[crzypatchwork]

I've seen some interesting things on itch.io as well but I'm not sure how they embed games

[pichiste]

Adding website (an html/js/css/etc bundle?) as a media type in general would be amazing. Thinking of AR/VR, interactive, generative works, etc. It would cover the p5.js and Unity scenarios mentioned above!

[mmetaa]

i would like to mint a pdf of a very large & complex vector illustration i have made. this is a complex piece with a lot of layers, transparencies, etc and trying to view it in a browser brings everything to a halt for a minute or so while it renders. (you can see some images of it here: http://meta.am/cr4trev2/)

ideally, the pdf would be minted but not viewable or downloadable from the artwork page. instead there would be multiple jpg previews to give a sense of the overall piece & the detail involved, and then only the buyer would get access to the actual vector pdf file. could this be made possible?

[mrdoob]

Uploading a zip file that contains a website could be interesting indeed.

The first thing that comes to mind are bad actors though (trackers, ransomware, etc).

I've not tested this but I think <iframe src="index.html" sandbox="allow-scripts"></iframe> should be able to solve that by only allowing running the scripts within the iframe and disallow making connections to other websites.

[pichiste]

Yeah it seem like iframe with sandbox="allow-scripts" would be the way to go. It would keep the iframe from having any access to the containing page, which is maybe the main security concern?

Don't think there's a way of keeping the iframe from requesting arbitrary web resources, but maybe that's ok, and even desirable/necessary for some artworks.

I could imagine a version of this where listings of type website are hidden behind an image/video thumbnail with a "Click to launch" overlay, so the iframe is never embedded until clicked on.

[halvves]

@pichiste @mrdoob If you really wanted to keep things safe and require self contained zips (no external files), you could inject a CSP meta tag on upload.

[andrevenancio]

I've implemented a similar solution for interactive SVGs here for the code and here for the iframe

I imagine a similar solution can be done for that.

[mrdoob]

@andrevenancio I have made a game with SVG. Seems like the site is not using <iframe>s yet?

[andrevenancio]

Yeah... I tag @crzypatchwork every 5minutes but he might have a lot on his plate. Only him is able to make deployments. I can only help with the code and merge and review stuff. But there's a "manual" continuous integration 😂 process in place

[mrdoob]

No rush! 😁

[mrdoob]

Seems like the resizing code has some issues on desktop... 😕

https://www.hicetnunc.xyz/objkt/7577

[andrevenancio]

I was just coming here to let you know that it got deployed! But I guess you sleep less than me. I can see it, but I can't interact with it. what are we missing? Click events? Keyboard events?

[pichiste]

@andrevenancio @mrdoob Hey I've been experimenting with html support via zip archive and have something working. Basically it lets you load a zip file that contains an index.html file and all related resources (js, css, images, etc), and displays it in a sandboxed iframe.

As suggested it's based on the component/sandbox pattern from your svg implementation (thanks!). The trickiest bit is that all resources are extracted as Blobs from the zip archive, so paths referenced in html/js/css point to nowhere. The best solution I could come up with was to use a service worker to intercept all http requests and respond with the Blob data for the given resource.

I know it's a convoluted approach but couldn't think of any other way to do it if we have to stick to uploading a single zip file to ipfs. Maybe there's an easier way?

I should note I haven't gotten as far as actually uploading anything to ipfs – just displaying in the preview window on the mint page, but imagine that should just work.

If you're open to contributions I'm happy to help on this, let me know!

[spite]

Speaking of iframes and potential html content... what do we think about permissions like camera or microphone? i've made https://www.hicetnunc.xyz/objkt/7770 using getUserMEdia but it fails due to permissions.

[elalish]

A glTF (same as many file formats) is just a zip archive with internal URLs pointing to included files. So it seems like there must be an easier way. Three.js is loading these and various other zip-based formats, so perhaps a pattern from there would be useful?

[mrdoob]

@pichiste Nice exploration!

I think we can start by supporting single html files with everything embedded in it? (like demoscene 4k and js13k games do it)

[pichiste]

@mrdoob Makes a lot of sense!

I guess ultimately something like this would be ideal: https://web.dev/web-bundles/

[pichiste]

Submitted a PR for single page HTML support: https://github.com/hicetnunc2000/hicetnunc/pull/189

[veqtor]

@andrevenancio @mrdoob Hey I've been experimenting with html support via zip archive and have something working. Basically it lets you load a zip file that contains an index.html file and all related resources (js, css, images, etc), and displays it in a sandboxed iframe.

As suggested it's based on the component/sandbox pattern from your svg implementation (thanks!). The trickiest bit is that all resources are extracted as Blobs from the zip archive, so paths referenced in html/js/css point to nowhere. The best solution I could come up with was to use a service worker to intercept all http requests and respond with the Blob data for the given resource.

I know it's a convoluted approach but couldn't think of any other way to do it if we have to stick to uploading a single zip file to ipfs. Maybe there's an easier way?

I should note I haven't gotten as far as actually uploading anything to ipfs – just displaying in the preview window on the mint page, but imagine that should just work.

If you're open to contributions I'm happy to help on this, let me know!

I think the proper way to do this is to upload a "folder" to IPFS which is a feature they already support. This is the recommended way to upload a complete "website" to IPFS with multiple resources.

https://tarunbatra.com/blog/decentralization/Deploy-your-website-on-IPFS-Why-and-How/

[pichiste]

(Sorry to duplicate from discord, but just in case some people aren't there)

Good to know! And completely in favor of letting people upload a zip file without having to deal with any bundling friction.

The problem still remains when displaying the preview on the mint page, since the files aren't on IPFS yet. But the service worker solution above seems to work well for that context. Thoughts?

Re: cover image, what about having a convention where the user can just include a cover.jpg image in the zip file and it will automatically get picked up as the cover? It could also be a separate upload field, in case we want this applied to other media types.